Eye smarter than scientists believed: neural computations in circuits of the retina

Artificial intelligence techniques for retinal prostheses: a comprehensive review and future direction

Objective. Retinal prostheses are promising devices to restore vision for patients with severe age-related macular degeneration or retinitis pigmentosa disease. The visual processing mechanism embodied in retinal prostheses play an important role in the restoration effect. Its performance depends on our understanding of the retina's working mechanism and the evolvement of computer vision models. Recently, remarkable progress has been made in the field of processing algorithm for retinal prostheses where the new discovery of the retina's working principle and state-of-the-arts computer vision models are combined together.Approach. We investigated the related research on artificial intelligence techniques for retinal prostheses. The processing algorithm in these studies could be attributed to three types: computer vision-related methods, biophysical models, and deep learning models.Main results. In this review, we first illustrate the structure and function of the normal and degenerated retina, then demonstrate the vision rehabilitation mechanism of three representative retinal prostheses. It is necessary to summarize the computational frameworks abstracted from the normal retina. In addition, the development and feature of three types of different processing algorithms are summarized. Finally, we analyze the bottleneck in existing algorithms and propose our prospect about the future directions to improve the restoration effect.Significance. This review systematically summarizes existing processing models for predicting the response of the retina to external stimuli. What's more, the suggestions for future direction may inspire researchers in this field to design better algorithms for retinal prostheses.

Graphene/MoS2-xOx/graphene photomemristor with tunable non-volatile responsivities for neuromorphic vision processing

Conventional artificial intelligence (AI) machine vision technology, based on the von Neumann architecture, uses separate sensing, computing, and storage units to process huge amounts of vision data generated in sensory terminals. The frequent movement of redundant data between sensors, processors and memory, however, results in high-power consumption and latency. A more efficient approach is to offload some of the memory and computational tasks to sensor elements that can perceive and process the optical signal simultaneously. Here, we proposed a non-volatile photomemristor, in which the reconfigurable responsivity can be modulated by the charge and/or photon flux through it and further stored in the device. The non-volatile photomemristor has a simple two-terminal architecture, in which photoexcited carriers and oxygen-related ions are coupled, leading to a displaced and pinched hysteresis in the current-voltage characteristics. For the first time, non-volatile photomemristors implement computationally complete logic with photoresponse-stateful operations, for which the same photomemristor serves as both a logic gate and memory, using photoresponse as a physical state variable instead of light, voltage and memresistance. The polarity reversal of photomemristors shows great potential for in-memory sensing and computing with feature extraction and image recognition for neuromorphic vision.

Role of Oxidative Stress in Ocular Diseases: A Balancing Act

Redox homeostasis is a delicate balancing act of maintaining appropriate levels of antioxidant defense mechanisms and reactive oxidizing oxygen and nitrogen species. Any disruption of this balance leads to oxidative stress, which is a key pathogenic factor in several ocular diseases. In this review, we present the current evidence for oxidative stress and mitochondrial dysfunction in conditions affecting both the anterior segment (e.g., dry eye disease, keratoconus, cataract) and posterior segment (age-related macular degeneration, proliferative vitreoretinopathy, diabetic retinopathy, glaucoma) of the human eye. We posit that further development of therapeutic interventions to promote pro-regenerative responses and maintenance of the redox balance may delay or prevent the progression of these major ocular pathologies. Continued efforts in this field will not only yield a better understanding of the molecular mechanisms underlying the pathogenesis of ocular diseases but also enable the identification of novel druggable redox targets and antioxidant therapies.

Nine insights from internet engineering that help us understand brain network communication

Philosophers have long recognized the value of metaphor as a tool that opens new avenues of investigation. By seeing brains as having the goal of representation, the computer metaphor in its various guises has helped systems neuroscience approach a wide array of neuronal behaviors at small and large scales. Here I advocate a complementary metaphor, the internet. Adopting this metaphor shifts our focus from computing to communication, and from seeing neuronal signals as localized representational elements to seeing neuronal signals as traveling messages. In doing so, we can take advantage of a comparison with the internet's robust and efficient routing strategies to understand how the brain might meet the challenges of network communication. I lay out nine engineering strategies that help the internet solve routing challenges similar to those faced by brain networks. The internet metaphor helps us by reframing neuronal activity across the brain as, in part, a manifestation of routing, which may, in different parts of the system, resemble the internet more, less, or not at all. I describe suggestive evidence consistent with the brain's use of internet-like routing strategies and conclude that, even if empirical data do not directly implicate internet-like routing, the metaphor is valuable as a reference point for those investigating the difficult problem of network communication in the brain and in particular the problem of routing.

Emerging memristive neurons for neuromorphic computing and sensing

Inspired by the principles of the biological nervous system, neuromorphic engineering has brought a promising alternative approach to intelligence computing with high energy efficiency and low consumption. As pivotal components of neuromorphic system, artificial spiking neurons are powerful information processing units and can achieve highly complex nonlinear computations. By leveraging the switching dynamic characteristics of memristive device, memristive neurons show rich spiking behaviors with simple circuit. This report reviews the memristive neurons and their applications in neuromorphic sensing and computing systems. The switching mechanisms that endow memristive devices with rich dynamics and nonlinearity are highlighted, and subsequently various nonlinear spiking neuron behaviors emulated in these memristive devices are reviewed. Then, recent development is introduced on neuromorphic system with memristive neurons for sensing and computing. Finally, we discuss challenges and outlooks of the memristive neurons toward high-performance neuromorphic hardware systems and provide an insightful perspective for the development of interactive neuromorphic electronic systems.

The Impact of Multivesicular Release on the Transmission of Sensory Information by Ribbon Synapses

The statistics of vesicle release determine how synapses transfer information, but the classical Poisson model of independent release does not always hold at the first stages of vision and hearing. There, ribbon synapses also encode sensory signals as events comprising two or more vesicles released simultaneously. The implications of such coordinated multivesicular release (MVR) for spike generation are not known. Here we investigate how MVR alters the transmission of sensory information compared with Poisson synapses using a pure rate-code. We used leaky integrate-and-fire models incorporating the statistics of release measured experimentally from glutamatergic synapses of retinal bipolar cells in zebrafish (both sexes) and compared these with models assuming Poisson inputs constrained to operate at the same average rates. We find that MVR can increase the number of spikes generated per vesicle while reducing interspike intervals and latency to first spike. The combined effect was to increase the efficiency of information transfer (bits per vesicle) over a range of conditions mimicking target neurons of different size. MVR was most advantageous in neurons with short time constants and reliable synaptic inputs, when less convergence was required to trigger spikes. In the special case of a single input driving a neuron, as occurs in the auditory system of mammals, MVR increased information transfer whenever spike generation required more than one vesicle. This study demonstrates how presynaptic integration of vesicles by MVR can increase the efficiency with which sensory information is transmitted compared with a rate-code described by Poisson statistics.SIGNIFICANCE STATEMENT Neurons communicate by the stochastic release of vesicles at the synapse and the statistics of this process will determine how information is represented by spikes. The classical model is that vesicles are released independently by a Poisson process, but this does not hold at ribbon-type synapses specialized to transmit the first electrical signals in vision and hearing, where two or more vesicles can fuse in a single event by a process termed coordinated multivesicular release. This study shows that multivesicular release can increase the number of spikes generated per vesicle and the efficiency of information transfer (bits per vesicle) over a range of conditions found in the retina and peripheral auditory system.

Efficient processing of natural scenes in visual cortex

Neural circuits in the periphery of the visual, auditory, and olfactory systems are believed to use limited resources efficiently to represent sensory information by adapting to the statistical structure of the natural environment. This "efficient coding" principle has been used to explain many aspects of early visual circuits including the distribution of photoreceptors, the mosaic geometry and center-surround structure of retinal receptive fields, the excess OFF pathways relative to ON pathways, saccade statistics, and the structure of simple cell receptive fields in V1. We know less about the extent to which such adaptations may occur in deeper areas of cortex beyond V1. We thus review recent developments showing that the perception of visual textures, which depends on processing in V2 and beyond in mammals, is adapted in rats and humans to the multi-point statistics of luminance in natural scenes. These results suggest that central circuits in the visual brain are adapted for seeing key aspects of natural scenes. We conclude by discussing how adaptation to natural temporal statistics may aid in learning and representing visual objects, and propose two challenges for the future: (1) explaining the distribution of shape sensitivity in the ventral visual stream from the statistics of object shape in natural images, and (2) explaining cell types of the vertebrate retina in terms of feature detectors that are adapted to the spatio-temporal structures of natural stimuli. We also discuss how new methods based on machine learning may complement the normative, principles-based approach to theoretical neuroscience.

Calibrating Vision: Concepts and Questions

The idea that visual coding and perception are shaped by experience and adjust to changes in the environment or the observer is universally recognized as a cornerstone of visual processing, yet the functions and processes mediating these calibrations remain in many ways poorly understood. In this article we review a number of facets and issues surrounding the general notion of calibration, with a focus on plasticity within the encoding and representational stages of visual processing. These include how many types of calibrations there are - and how we decide; how plasticity for encoding is intertwined with other principles of sensory coding; how it is instantiated at the level of the dynamic networks mediating vision; how it varies with development or between individuals; and the factors that may limit the form or degree of the adjustments. Our goal is to give a small glimpse of an enormous and fundamental dimension of vision, and to point to some of the unresolved questions in our understanding of how and why ongoing calibrations are a pervasive and essential element of vision.

A minimal-complexity light-sheet microscope maps network activity in 3D neuronal systems

In vitro systems mimicking brain regions, brain organoids, are revolutionizing the neuroscience field. However, characterization of their electrical activity has remained a challenge as it requires readout at millisecond timescale in 3D at single-neuron resolution. While custom-built microscopes used with genetically encoded sensors are now opening this door, a full 3D characterization of organoid neural activity has not been performed yet, limited by the combined complexity of the optical and the biological system. Here, we introduce an accessible minimalistic light-sheet microscope to the neuroscience community. Designed as an add-on to a standard inverted microscope it can be assembled within one day. In contrast to existing simplistic setups, our platform is suited to record volumetric calcium traces. We successfully extracted 4D calcium traces at high temporal resolution by using a lightweight piezo stage to allow for 5 Hz volumetric scanning combined with a processing pipeline for true 3D neuronal trace segmentation. As a proof of principle, we created a 3D connectivity map of a stem cell derived neuron spheroid by imaging its activity. Our fast, low complexity setup empowers researchers to study the formation of neuronal networks in vitro for fundamental and neurodegeneration research.

Bio-mimetic high-speed target localization with fused frame and event vision for edge application

Evolution has honed predatory skills in the natural world where localizing and intercepting fast-moving prey is required. The current generation of robotic systems mimics these biological systems using deep learning. High-speed processing of the camera frames using convolutional neural networks (CNN) (frame pipeline) on such constrained aerial edge-robots gets resource-limited. Adding more compute resources also eventually limits the throughput at the frame rate of the camera as frame-only traditional systems fail to capture the detailed temporal dynamics of the environment. Bio-inspired event cameras and spiking neural networks (SNN) provide an asynchronous sensor-processor pair (event pipeline) capturing the continuous temporal details of the scene for high-speed but lag in terms of accuracy. In this work, we propose a target localization system combining event-camera and SNN-based high-speed target estimation and frame-based camera and CNN-driven reliable object detection by fusing complementary spatio-temporal prowess of event and frame pipelines. One of our main contributions involves the design of an SNN filter that borrows from the neural mechanism for ego-motion cancelation in houseflies. It fuses the vestibular sensors with the vision to cancel the activity corresponding to the predator's self-motion. We also integrate the neuro-inspired multi-pipeline processing with task-optimized multi-neuronal pathway structure in primates and insects. The system is validated to outperform CNN-only processing using prey-predator drone simulations in realistic 3D virtual environments. The system is then demonstrated in a real-world multi-drone set-up with emulated event data. Subsequently, we use recorded actual sensory data from multi-camera and inertial measurement unit (IMU) assembly to show desired working while tolerating the realistic noise in vision and IMU sensors. We analyze the design space to identify optimal parameters for spiking neurons, CNN models, and for checking their effect on the performance metrics of the fused system. Finally, we map the throughput controlling SNN and fusion network on edge-compatible Zynq-7000 FPGA to show a potential 264 outputs per second even at constrained resource availability. This work may open new research directions by coupling multiple sensing and processing modalities inspired by discoveries in neuroscience to break fundamental trade-offs in frame-based computer vision.

Systematic reduction of the dimensionality of natural scenes allows accurate predictions of retinal ganglion cell spike outputs

The mammalian retina engages a broad array of linear and nonlinear circuit mechanisms to convert natural scenes into retinal ganglion cell (RGC) spike outputs. Although many individual integration mechanisms are well understood, we know less about how multiple mechanisms interact to encode the complex spatial features present in natural inputs. Here, we identified key spatial features in natural scenes that shape encoding by primate parasol RGCs. Our approach identified simplifications in the spatial structure of natural scenes that minimally altered RGC spike responses. We observed that reducing natural movies into 16 linearly integrated regions described ∼80% of the structure of parasol RGC spike responses; this performance depended on the number of regions but not their precise spatial locations. We used simplified stimuli to design high-dimensional metamers that recapitulated responses to naturalistic movies. Finally, we modeled the retinal computations that convert flashed natural images into one-dimensional spike counts.

Derivative-Free Observability Analysis for Sensor Placement Optimization of Bioinspired Flexible Flapping Wing System

Observability analysis of a bioinspired flexible flapping wing system provides a measure of how well the states of flexible flapping wing micro-aerial vehicles can be estimated from real-time measurements during high-speed flight. However, the traditional observability analysis approaches have trouble in terms of lack of quantitative analysis index, high computational complexity, low accuracy, and unavailability in stochastic systems with memory, including bioinspired flexible flapping wing systems. Therefore, a novel derivative-free observability analysis method is proposed here based on the generalized polynomial chaos expansion. By formulating a surrogate model to represent the relationship between the cumulative measurement and the random initial state, the observability coefficient matrix is calculated and the observability rank condition is stated. Consequently, several observability indices are proposed to quantity the observability of the system. Altogether, the proposed method avoids the disadvantages of the traditional approaches, especially in assessing the observability degree of each state and the effect of stochastic noise on observability. The validation of the proposed method is first provided by demonstrating the equivalence between the traditional and proposed methods and subsequently by comparing the observability of the Lorenz system calculated via three different approaches. Finally, the proposed method is applied on a bioinspired flexible wing system to optimize the placement of sensors, which is consistent with the natural configuration of campaniform sensilla on the wing of the hawkmoth.

Homeostatic plasticity in the retina

Vision begins in the retina, whose intricate neural circuits extract salient features of the environment from the light entering our eyes. Neurodegenerative diseases of the retina (e.g., inherited retinal degenerations, age-related macular degeneration, and glaucoma) impair vision and cause blindness in a growing number of people worldwide. Increasing evidence indicates that homeostatic plasticity (i.e., the drive of a neural system to stabilize its function) can, in principle, preserve retinal function in the face of major perturbations, including neurodegeneration. Here, we review the circumstances and events that trigger homeostatic plasticity in the retina during development, sensory experience, and disease. We discuss the diverse mechanisms that cooperate to compensate and the set points and outcomes that homeostatic retinal plasticity stabilizes. Finally, we summarize the opportunities and challenges for unlocking the therapeutic potential of homeostatic plasticity. Homeostatic plasticity is fundamental to understanding retinal development and function and could be an important tool in the fight to preserve and restore vision.

Layer-Specific Developmentally Precise Axon Targeting of Transient Suppressed-by-Contrast Retinal Ganglion Cells

The mouse retina encodes diverse visual features in the spike trains of >40 retinal ganglion cell (RGC) types. Each RGC type innervates a specific subset of the >50 retinorecipient brain areas. Our catalog of RGC types and feature representations is nearing completion. Yet, we know little about where specific RGC types send their information. Furthermore, the developmental strategies by which RGC axons choose their targets and pattern their terminal arbors remain obscure. Here, we identify a genetic intersection (Cck-Cre and Brn3cCKOAP ) that selectively labels transient Suppressed-by-Contrast (tSbC) RGCs, a member of an evolutionarily conserved functionally mysterious RGC subclass. We find that tSbC RGCs selectively innervate the dorsolateral geniculate nucleus (dLGN) and ventrolateral geniculate nucleus (vLGN) of the thalamus, the superior colliculus (SC), and the nucleus of the optic tract (NOT) in mice of either sex. They binocularly innervate dLGN and vLGN but project only contralaterally to SC and NOT. In each target, tSbC RGC axons occupy a specific sublayer, suggesting that they restrict their input to specific circuits. The tSbC RGC axons span the length of the optic tract by birth and remain poised there until they simultaneously innervate their four targets around postnatal day 3. The tSbC RGC axons choose the right targets and establish mature stratification patterns from the outset. This precision is maintained in the absence of Brn3c. Our results provide the first map of SbC inputs to the brain, revealing a narrow target set, unexpected laminar organization, target-specific binocularity, and developmental precision.SIGNIFICANCE STATEMENT In recent years, we have learned a lot about the visual features encoded by RGCs, the output neurons of the eye. In contrast, we know little about where RGCs send their information and how RGC axons, which carry this information, target specific brain areas during development. Here, we develop an intersectional strategy to label a unique RGC type, the tSbC RGC, and map its projections. We find that tSbC RGC axons are highly selective. They innervate few retinal targets and restrict their arbors to specific sublayers within these targets. The selective tSbC RGC projection patterns develop synchronously and without trial and error, suggesting molecular determinism and coordination.

Sensor-level computer vision with pixel processor arrays for agile robots

Vision processing for control of agile autonomous robots requires low-latency computation, within a limited power and space budget. This is challenging for conventional computing hardware. Parallel processor arrays (PPAs) are a new class of vision sensor devices that exploit advances in semiconductor technology, embedding a processor within each pixel of the image sensor array. Sensed pixel data are processed on the focal plane, and only a small amount of relevant information is transmitted out of the vision sensor. This tight integration of sensing, processing, and memory within a massively parallel computing architecture leads to an interesting trade-off between high performance, low latency, low power, low cost, and versatility in a machine vision system. Here, we review the history of image sensing and processing hardware from the perspective of in-pixel computing and outline the key features of a state-of-the-art smart camera system based on a PPA device, through the description of the SCAMP-5 system. We describe several robotic applications for agile ground and aerial vehicles, demonstrating PPA sensing functionalities including high-speed odometry, target tracking, obstacle detection, and avoidance. In the conclusions, we provide some insight and perspective on the future development of PPA devices, including their application and benefits within agile, robust, adaptable, and lightweight robotics.

Neural System Identification With Spike-Triggered Non-Negative Matrix Factorization

Neuronal circuits formed in the brain are complex with intricate connection patterns. Such complexity is also observed in the retina with a relatively simple neuronal circuit. A retinal ganglion cell (GC) receives excitatory inputs from neurons in previous layers as driving forces to fire spikes. Analytical methods are required to decipher these components in a systematic manner. Recently a method called spike-triggered non-negative matrix factorization (STNMF) has been proposed for this purpose. In this study, we extend the scope of the STNMF method. By using retinal GCs as a model system, we show that STNMF can detect various computational properties of upstream bipolar cells (BCs), including spatial receptive field, temporal filter, and transfer nonlinearity. In addition, we recover synaptic connection strengths from the weight matrix of STNMF. Furthermore, we show that STNMF can separate spikes of a GC into a few subsets of spikes, where each subset is contributed by one presynaptic BC. Taken together, these results corroborate that STNMF is a useful method for deciphering the structure of neuronal circuits.

Learning, fast and slow

Animals can learn efficiently from a single experience and change their future behavior in response. However, in other instances, animals learn very slowly, requiring thousands of experiences. Here, I survey tasks involving fast and slow learning and consider some hypotheses for what differentiates the underlying neural mechanisms. It has been proposed that fast learning relies on neural representations that favor efficient Hebbian modification of synapses. These efficient representations may be encoded in the genome, resulting in a repertoire of fast learning that differs across species. Alternatively, the required neural representations may be acquired from experience through a slow process of unsupervised learning from the environment.

Neuroscience: Reliable and refined motion computations in the retina

We can distinguish between the direction and speed of a moving object effortlessly, but this is actually a very challenging computational task. A new study demonstrates that this process begins at the first stages of visual processing in the retina.

Conserved circuits for direction selectivity in the primate retina

The detection of motion direction is a fundamental visual function and a classic model for neural computation. In the non-primate retina, direction selectivity arises in starburst amacrine cell (SAC) dendrites, which provide selective inhibition to direction-selective retinal ganglion cells (dsRGCs). Although SACs are present in primates, their connectivity and the existence of dsRGCs remain open questions. Here, we present a connectomic reconstruction of the primate ON SAC circuit from a serial electron microscopy volume of the macaque central retina. We show that the structural basis for the SACs' ability to confer directional selectivity on postsynaptic neurons is conserved. SACs selectively target a candidate homolog to the mammalian ON-sustained dsRGCs that project to the accessory optic system (AOS) and contribute to gaze-stabilizing reflexes. These results indicate that the capacity to compute motion direction is present in the retina, which is earlier in the primate visual system than classically thought.

[Optogenetics and cell replacement in retinology : Regenerative ophthalmology-What we can do!]

For many degenerative retinal diseases that progressively lead to blindness, no treatment options are available so far. In recent years, several innovative therapies have been experimentally explored, which are promising because they are independent of the genetic cause of the degenerative disease. One of these is optogenetics, which involves light-sensitive proteins that selectively act as ion channels or ion pumps to control the potential of the treated cell. Thus, these cells can be stimulated or inhibited by light, quasi functionally remote controlled. In this way artificial photoreceptors are induced from the remaining cells, which has already been successfully employed in animal experiments. This type of treatment is already being tested on patients and leads to an improvement in vision, but so far only data from one patient are available. The use of optogenetics additionally requires special eyeglasses to adapt the light impulses in adequate strength and wavelength for the respective optogenes. Another exciting approach is cell replacement therapy of retinal pigment epithelium (RPE) and photoreceptor cells to exchange degenerated cell material. This appears to be very successful for RPE cells in clinical trials. Obtaining human photoreceptors from stem cells is technically possible, but very laborious. The integration of the transplanted photoreceptors into the host retinal tissue also needs further optimization for broader clinical applications; however, both cell replacement and optogenetics approaches are promising, so that the translation from basic research into clinical application will be successful.

Tools and Biomarkers for the Study of Retinal Ganglion Cell Degeneration

The retina is part of the central nervous system, its analysis may provide an idea of the health and functionality, not only of the retina, but also of the entire central nervous system, as has been shown in Alzheimer’s or Parkinson’s diseases. Within the retina, the ganglion cells (RGC) are the neurons in charge of processing and sending light information to higher brain centers. Diverse insults and pathological states cause degeneration of RGC, leading to irreversible blindness or impaired vision. RGCs are the measurable endpoints in current research into experimental therapies and diagnosis in multiple ocular pathologies, like glaucoma. RGC subtype classifications are based on morphological, functional, genetical, and immunohistochemical aspects. Although great efforts are being made, there is still no classification accepted by consensus. Moreover, it has been observed that each RGC subtype has a different susceptibility to injury. Characterizing these subtypes together with cell death pathway identification will help to understand the degenerative process in the different injury and pathological models, and therefore prevent it. Here we review the known RGC subtypes, as well as the diagnostic techniques, probes, and biomarkers for programmed and unprogrammed cell death in RGC.

Is our retina really upside down?

In the vertebrate eye, photoreceptors are covered beneath a thick sheet of neural retina and face away from the light. This seemingly awkward arrangement has led to the popular notion that our retinas are upside down, implying a deep design flaw. Baden and Nilsson argue that, from an evolutionary perspective, an inverted design actually offers many notable benefits that might have never been exploited if things had started off the other way round.

Donut-like organization of inhibition underlies categorical neural responses in the midbrain

Categorical neural responses underlie various forms of selection and decision-making. Such binary-like responses promote robust signaling of the winner in the presence of input ambiguity and neural noise. Here, we show that a 'donut-like' inhibitory mechanism in which each competing option suppresses all options except itself, is highly effective at generating categorical neural responses. It surpasses motifs of feedback inhibition, recurrent excitation, and divisive normalization invoked frequently in decision-making models. We demonstrate experimentally not only that this mechanism operates in the midbrain spatial selection network in barn owls, but also that it is necessary for categorical signaling by it. The functional pattern of neural inhibition in the midbrain forms an exquisitely structured 'multi-holed' donut consistent with this network's combinatorial inhibitory function for stimulus selection. Additionally, modeling reveals a generalizable neural implementation of the donut-like motif for categorical selection. Self-sparing inhibition may, therefore, be a powerful circuit module central to categorization.

Adaptation in cone photoreceptors contributes to an unexpected insensitivity of primate On parasol retinal ganglion cells to spatial structure in natural images

Neural circuits are constructed from nonlinear building blocks, and not surprisingly overall circuit behavior is often strongly nonlinear. But neural circuits can also behave near linearly, and some circuits shift from linear to nonlinear behavior depending on stimulus conditions. Such control of nonlinear circuit behavior is fundamental to neural computation. Here, we study a surprising stimulus dependence of the responses of macaque On (but not Off) parasol retinal ganglion cells: these cells respond nonlinearly to spatial structure in some stimuli but near linearly to spatial structure in others, including natural inputs. We show that these differences in the linearity of the integration of spatial inputs can be explained by a shift in the balance of excitatory and inhibitory synaptic inputs that originates at least partially from adaptation in the cone photoreceptors. More generally, this highlights how subtle asymmetries in signaling - here in the cone signals - can qualitatively alter circuit computation.

Emergence of Direction-Selective Retinal Cell Types in Task-Optimized Deep Learning Models

Convolutional neural networks (CNNs), a class of deep learning models, have experienced recent success in modeling sensory cortices and retinal circuits through optimizing performance on machine learning tasks, otherwise known as task optimization. Previous research has shown task-optimized CNNs to be capable of providing explanations as to why the retina efficiently encodes natural stimuli and how certain retinal cell types are involved in efficient encoding. In our work, we sought to use task-optimized CNNs as a means of explaining computational mechanisms responsible for motion-selective retinal circuits. We designed a biologically constrained CNN and optimized its performance on a motion-classification task. We drew inspiration from psychophysics, deep learning, and systems neuroscience literature to develop a toolbox of methods to reverse engineer the computational mechanisms learned in our model. Through reverse engineering our model, we proposed a computational mechanism in which direction-selective ganglion cells and starburst amacrine cells, both experimentally observed retinal cell types, emerge in our model to discriminate among moving stimuli. This emergence suggests that direction-selective circuits in the retina are ecologically designed to robustly discriminate among moving stimuli. Our results and methods also provide a framework for how to build more interpretable deep learning models and how to understand them.

Retina organoids: Window into the biophysics of neuronal systems

With a kind of magnetism, the human retina draws the eye of neuroscientist and physicist alike. It is attractive as a self-organizing system, which forms as a part of the central nervous system via biochemical and mechanical cues. The retina is also intriguing as an electro-optical device, converting photons into voltages to perform on-the-fly filtering before the signals are sent to our brain. Here, we consider how the advent of stem cell derived in vitro analogs of the retina, termed retina organoids, opens up an exploration of the interplay between optics, electrics, and mechanics in a complex neuronal network, all in a Petri dish. This review presents state-of-the-art retina organoid protocols by emphasizing links to the biochemical and mechanical signals of in vivo retinogenesis. Electrophysiological recording of active signal processing becomes possible as retina organoids generate light sensitive and synaptically connected photoreceptors. Experimental biophysical tools provide data to steer the development of mathematical models operating at different levels of coarse-graining. In concert, they provide a means to study how mechanical factors guide retina self-assembly. In turn, this understanding informs the engineering of mechanical signals required to tailor the growth of neuronal network morphology. Tackling the complex developmental and computational processes in the retina requires an interdisciplinary endeavor combining experiment and theory, physics, and biology. The reward is enticing: in the next few years, retina organoids could offer a glimpse inside the machinery of simultaneous cellular self-assembly and signal processing, all in an in vitro setting.

Individual variations of visual information

In this issue of Neuron, Shah et al. reveal that coding of visual space by the primate retina varies across individuals and sexes. A computational model provides insight into themes and variations of coding. Implications for sight restoration are explored.

Mnemonic-opto-synaptic transistor for in-sensor vision system

A mnemonic-opto-synaptic transistor (MOST) that has triple functions is demonstrated for an in-sensor vision system. It memorizes a photoresponsivity that corresponds to a synaptic weight as a memory cell, senses light as a photodetector, and performs weight updates as a synapse for machine vision with an artificial neural network (ANN). Herein the memory function added to a previous photodetecting device combined with a photodetector and a synapse provides a technical breakthrough for realizing in-sensor processing that is able to perform image sensing and signal processing in a sensor. A charge trap layer (CTL) was intercalated to gate dielectrics of a vertical pillar-shaped transistor for the memory function. Weight memorized in the CTL makes photoresponsivity tunable for real-time multiplication of the image with a memorized photoresponsivity matrix. Therefore, these multi-faceted features can allow in-sensor processing without external memory for the in-sensor vision system. In particular, the in-sensor vision system can enhance speed and energy efficiency compared to a conventional vision system due to the simultaneous preprocessing of massive data at sensor nodes prior to ANN nodes. Recognition of a simple pattern was demonstrated with full sets of the fabricated MOSTs. Furthermore, recognition of complex hand-written digits in the MNIST database was also demonstrated with software simulations.

Synchronous inhibitory pathways create both efficiency and diversity in the retina

Complex connections in neural circuits make it difficult to quantitatively assign even the most basic neural computations to the actions of specific neurons. Retinal ganglion cells are most sensitive to changes in intensity across space and over time. This property, caused by a region known as the receptive field surround, improves information transmission about natural scenes. We dynamically manipulated individual interneurons to directly measure their effect on retinal receptive fields, finding that two inhibitory neuron types, horizontal cells and amacrine cells, synchronously create the same contribution to the receptive field surround at different spatial scales. By analyzing large populations of ganglion cells, we show that this arrangement increases diversity in retinal signaling while preserving maximal information transmission about natural scenes.

Sensory receptive fields combine features that originate in different neural pathways. Retinal ganglion cell receptive fields compute intensity changes across space and time using a peripheral region known as the surround, a property that improves information transmission about natural scenes. The visual features that construct this fundamental property have not been quantitatively assigned to specific interneurons. Here, we describe a generalizable approach using simultaneous intracellular and multielectrode recording to directly measure and manipulate the sensory feature conveyed by a neural pathway to a downstream neuron. By directly controlling the gain of individual interneurons in the circuit, we show that rather than transmitting different temporal features, inhibitory horizontal cells and linear amacrine cells synchronously create the linear surround at different spatial scales and that these two components fully account for the surround. By analyzing a large population of ganglion cells, we observe substantial diversity in the relative contribution of amacrine and horizontal cell visual features while still allowing individual cells to increase information transmission under the statistics of natural scenes. Established theories of efficient coding have shown that optimal information transmission under natural scenes allows a diverse set of receptive fields. Our results give a mechanism for this theory, showing how distinct neural pathways synthesize a sensory computation and how this architecture both generates computational diversity and achieves the objective of high information transmission.

Retinomorphic optoelectronic devices for intelligent machine vision

Biological visual system can efficiently handle optical information within the retina and visual cortex of the brain, which suggests an alternative approach for the upgrading of the current low-intelligence, large energy consumption, and complex circuitry of the artificial vision system for high-performance edge computing applications. In recent years, retinomorphic machine vision based on the integration of optoelectronic image sensors and processors has been regarded as a promising candidate to improve this phenomenon. This novel intelligent machine vision technology can perform information preprocessing near or even within the sensor in the front end, thereby reducing the transmission of redundant raw data and improving the efficiency of the back-end processor for high-level computing tasks. In this contribution, we try to present a comprehensive review on the recent progress achieved in this emergent field.

Natural image statistics for mouse vision

The mouse has dichromatic color vision based on two different types of opsins: short (S)- and middle (M)-wavelength-sensitive opsins with peak sensitivity to ultraviolet (UV; 360 nm) and green light (508 nm), respectively. In the mouse retina, cone photoreceptors that predominantly express the S-opsin are more sensitive to contrasts and denser towards the ventral retina, preferentially sampling the upper part of the visual field. In contrast, the expression of the M-opsin gradually increases towards the dorsal retina that encodes the lower visual field. Such a distinctive retinal organization is assumed to arise from a selective pressure in evolution to efficiently encode the natural scenes. However, natural image statistics of UV light remain largely unexplored. Here we developed a multi-spectral camera to acquire high-quality UV and green images of the same natural scenes, and examined the optimality of the mouse retina to the image statistics. We found that the local contrast and the spatial correlation were both higher in UV than in green for images above the horizon, but lower in UV than in green for those below the horizon. This suggests that the dorsoventral functional division of the mouse retina is not optimal for maximizing the bandwidth of information transmission. Factors besides the coding efficiency, such as visual behavioral requirements, will thus need to be considered to fully explain the characteristic organization of the mouse retina.

Retinal Processing: Insights from Mathematical Modelling

The retina is the entrance of the visual system. Although based on common biophysical principles, the dynamics of retinal neurons are quite different from their cortical counterparts, raising interesting problems for modellers. In this paper, I address some mathematically stated questions in this spirit, discussing, in particular: (1) How could lateral amacrine cell connectivity shape the spatio-temporal spike response of retinal ganglion cells? (2) How could spatio-temporal stimuli correlations and retinal network dynamics shape the spike train correlations at the output of the retina? These questions are addressed, first, introducing a mathematically tractable model of the layered retina, integrating amacrine cells' lateral connectivity and piecewise linear rectification, allowing for computing the retinal ganglion cells receptive field together with the voltage and spike correlations of retinal ganglion cells resulting from the amacrine cells networks. Then, I review some recent results showing how the concept of spatio-temporal Gibbs distributions and linear response theory can be used to characterize the collective spike response to a spatio-temporal stimulus of a set of retinal ganglion cells, coupled via effective interactions corresponding to the amacrine cells network. On these bases, I briefly discuss several potential consequences of these results at the cortical level.

Mouse Lines with Cre-Mediated Recombination in Retinal Amacrine Cells

Amacrine cells (ACs) are the most diverse neuronal cell type in the vertebrate retina. Yet little is known about the contribution of ACs to visual processing and retinal disease. A major challenge in evaluating AC function is genetic accessibility. A classic tool of mouse genetics, Cre-mediated recombination, can provide such access. We have screened existing genetically-modified mouse strains and identified multiple candidates that express Cre-recombinase in subsets of retinal ACs. The Cre-expressing mice were crossed to fluorescent-reporter mice to assay Cre expression. In addition, a Cre-dependent fluorescent reporter plasmid was electroporated into the subretinal space of Cre strains. Herein, we report three mouse lines (Tac1::IRES-cre, Camk2a-cre, and Scx-cre) that express Cre recombinase in sub-populations of ACs. In two of these lines, recombination occurred in multiple AC types and a small number of other retinal cell types, while recombination in the Camk2a-cre line appears specific to a morphologically distinct AC. We anticipate that these characterized mouse lines will be valuable tools to the community of researchers who study retinal biology and disease.

Development of Functional Properties in the Early Visual System: New Appreciations of the Roles of Lateral Geniculate Nucleus

In the years following Hubel and Wiesel's first reports on ocular dominance plasticity and amblyopia, much attention has been focused on understanding the role of cortical circuits in developmental and experience-dependent plasticity. Initial studies found few differences between retinal ganglion cells and neurons in the lateral geniculate nucleus and uncovered little evidence for an impact of altered visual experience on the functional properties of lateral geniculate nucleus neurons. In the last two decades, however, studies have revealed that the connectivity between the retina and lateral geniculate nucleus is much richer than was previously appreciated, even revealing visual plasticity - including ocular dominance plasticity - in lateral geniculate nucleus neurons. Here we review the development of the early visual system and the impact of experience with a distinct focus on recent discoveries about lateral geniculate nucleus, its connectivity, and evidence for its plasticity and rigidity during development.

All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition

With the advent of the Internet of Things era, the detection and recognition of moving objects is becoming increasingly important¹. The current motion detection and recognition (MDR) technology based on the complementary metal oxide semiconductor (CMOS) image sensors (CIS) platform contains redundant sensing, transmission conversion, processing and memory modules, rendering the existing systems bulky and inefficient in comparison to the human retina. Until now, non-memory capable vision sensors have only been used for static targets, rather than MDR. Here, we present a retina-inspired two-dimensional (2D) heterostructure based retinomorphic hardware device with all-in-one perception, memory and computing capabilities for the detection and recognition of moving trolleys. The proposed 2D retinomorphic device senses an optical stimulus to generate progressively tuneable positive/negative photoresponses and memorizes it, combined with interframe differencing computations, to achieve 100% separation detection of moving trichromatic trolleys without ghosting. The detected motion images are fed into a conductance mapped neural network to achieve fast trolley recognition in as few as four training epochs at 10% noise level, outperforming previous results from similar customized datasets. The prototype demonstration of a 2D retinomorphic device with integrated perceptual memory and computation provides the possibility of building compact, efficient MDR hardware.

Retinal Protein O-GlcNAcylation and the Ocular Renin-angiotensin System: Signaling Cross-roads in Diabetic Retinopathy

It is well established that diabetes and its associated hyperglycemia negatively impact retinal function, yet we know little about the role played by augmented flux through the Hexosamine Biosynthetic Pathway (HBP). This offshoot of the glycolytic pathway produces UDP-Nacetyl- glucosamine, which serves as the substrate for post-translational O-linked modification of proteins in a process referred to as O-GlcNAcylation. HBP flux and subsequent protein O-GlcNAcylation serve as nutrient sensors, enabling cells to integrate metabolic information to appropriately modulate fundamental cellular processes including gene expression. Here we summarize the impact of diabetes on retinal physiology, highlighting recent studies that explore the role of O-GlcNAcylation- induced variation in mRNA translation in retinal dysfunction and the pathogenesis of Diabetic Retinopathy (DR). Augmented O-GlcNAcylation results in wide variation in the selection of mRNAs for translation, in part, due to O-GlcNAcylation of the translational repressor 4E-BP1. Recent studies demonstrate that 4E-BP1 plays a critical role in regulating O-GlcNAcylation-induced changes in the translation of the mRNAs encoding Vascular Endothelial Growth Factor (VEGF), a number of important mitochondrial proteins, and CD40, a key costimulatory molecule involved in diabetes-induced retinal inflammation. Remarkably, 4E-BP1/2 ablation delays the onset of diabetes- induced visual dysfunction in mice. Thus, pharmacological interventions to prevent the impact of O-GlcNAcylation on 4E-BP1 may represent promising therapeutics to address the development and progression of DR. In this regard, we discuss the potential interplay between retinal O-GlcNAcylation and the ocular renin-angiotensin system as a potential therapeutic target of future interventions.

Shaping the Microglia in Retinal Degenerative Diseases Using Stem Cell Therapy: Practice and Prospects

Retinal degenerative disease (RDD) refers to a group of diseases with retinal degeneration that cause vision loss and affect people's daily lives. Various therapies have been proposed, among which stem cell therapy (SCT) holds great promise for the treatment of RDDs. Microglia are immune cells in the retina that have two activation phenotypes, namely, pro-inflammatory M1 and anti-inflammatory M2 phenotypes. These cells play an important role in the pathological progression of RDDs, especially in terms of retinal inflammation. Recent studies have extensively investigated the therapeutic potential of stem cell therapy in treating RDDs, including the immunomodulatory effects targeting microglia. In this review, we substantially summarized the characteristics of RDDs and microglia, discussed the microglial changes and phenotypic transformation of M1 microglia to M2 microglia after SCT, and proposed future directions for SCT in treating RDDs.

Realistic retinal modeling unravels the differential role of excitation and inhibition to starburst amacrine cells in direction selectivity

Retinal direction-selectivity originates in starburst amacrine cells (SACs), which display a centrifugal preference, responding with greater depolarization to a stimulus expanding from soma to dendrites than to a collapsing stimulus. Various mechanisms were hypothesized to underlie SAC centrifugal preference, but dissociating them is experimentally challenging and the mechanisms remain debatable. To address this issue, we developed the Retinal Stimulation Modeling Environment (RSME), a multifaceted data-driven retinal model that encompasses detailed neuronal morphology and biophysical properties, retina-tailored connectivity scheme and visual input. Using a genetic algorithm, we demonstrated that spatiotemporally diverse excitatory inputs-sustained in the proximal and transient in the distal processes-are sufficient to generate experimentally validated centrifugal preference in a single SAC. Reversing these input kinetics did not produce any centrifugal-preferring SAC. We then explored the contribution of SAC-SAC inhibitory connections in establishing the centrifugal preference. SAC inhibitory network enhanced the centrifugal preference, but failed to generate it in its absence. Embedding a direction selective ganglion cell (DSGC) in a SAC network showed that the known SAC-DSGC asymmetric connectivity by itself produces direction selectivity. Still, this selectivity is sharpened in a centrifugal-preferring SAC network. Finally, we use RSME to demonstrate the contribution of SAC-SAC inhibitory connections in mediating direction selectivity and recapitulate recent experimental findings. Thus, using RSME, we obtained a mechanistic understanding of SACs' centrifugal preference and its contribution to direction selectivity.

An uncommon neuronal class conveys visual signals from rods and cones to retinal ganglion cells

Neurons in the central nervous system (CNS) are distinguished by the neurotransmitter types they release, their synaptic connections, morphology, and genetic profiles. To fully understand how the CNS works, it is critical to identify all neuronal classes and reveal their synaptic connections. The retina has been extensively used to study neuronal development and circuit formation. Here, we describe a previously unidentified interneuron in mammalian retina. This interneuron shares some morphological, physiological, and molecular features with retinal bipolar cells, such as receiving input from photoreceptors and relaying visual signals to retinal ganglion cells. It also shares some features with amacrine cells (ACs), particularly Aii-ACs, such as their neurite morphology in the inner plexiform layer, the expression of some AC-specific markers, and possibly the release of the inhibitory neurotransmitter glycine. Thus, we unveil an uncommon interneuron, which may play an atypical role in vision.

Unraveling neural coding of dynamic natural visual scenes via convolutional recurrent neural networks

Traditional models of retinal system identification analyze the neural response to artificial stimuli using models consisting of predefined components. The model design is limited to prior knowledge, and the artificial stimuli are too simple to be compared with stimuli processed by the retina. To fill in this gap with an explainable model that reveals how a population of neurons work together to encode the larger field of natural scenes, here we used a deep-learning model for identifying the computational elements of the retinal circuit that contribute to learning the dynamics of natural scenes. Experimental results verify that the recurrent connection plays a key role in encoding complex dynamic visual scenes while learning biological computational underpinnings of the retinal circuit. In addition, the proposed models reveal both the shapes and the locations of the spatiotemporal receptive fields of ganglion cells.

The cell adhesion molecule Sdk1 shapes assembly of a retinal circuit that detects localized edges

Nearly 50 different mouse retinal ganglion cell (RGC) types sample the visual scene for distinct features. RGC feature selectivity arises from their synapses with a specific subset of amacrine (AC) and bipolar cell (BC) types, but how RGC dendrites arborize and collect input from these specific subsets remains poorly understood. Here we examine the hypothesis that RGCs employ molecular recognition systems to meet this challenge. By combining calcium imaging and type-specific histological stains, we define a family of circuits that express the recognition molecule Sidekick-1 (Sdk1), which include a novel RGC type (S1-RGC) that responds to local edges. Genetic and physiological studies revealed that Sdk1 loss selectively disrupts S1-RGC visual responses, which result from a loss of excitatory and inhibitory inputs and selective dendritic deficits on this neuron. We conclude that Sdk1 shapes dendrite growth and wiring to help S1-RGCs become feature selective.

Inducible Pluripotent Stem Cells to Model and Treat Inherited Degenerative Diseases of the Outer Retina: 3D-Organoids Limitations and Bioengineering Solutions

Inherited retinal degenerations (IRD) affecting either photoreceptors or pigment epithelial cells cause progressive visual loss and severe disability, up to complete blindness. Retinal organoids (ROs) technologies opened up the development of human inducible pluripotent stem cells (hiPSC) for disease modeling and replacement therapies. However, hiPSC-derived ROs applications to IRD presently display limited maturation and functionality, with most photoreceptors lacking well-developed outer segments (OS) and light responsiveness comparable to their adult retinal counterparts. In this review, we address for the first time the microenvironment where OS mature, i.e., the subretinal space (SRS), and discuss SRS role in photoreceptors metabolic reprogramming required for OS generation. We also address bioengineering issues to improve culture systems proficiency to promote OS maturation in hiPSC-derived ROs. This issue is crucial, as satisfying the demanding metabolic needs of photoreceptors may unleash hiPSC-derived ROs full potential for disease modeling, drug development, and replacement therapies.

Temporal vision: measures, mechanisms and meaning

Time is largely a hidden variable in vision. It is the condition for seeing interesting things such as spatial forms and patterns, colours and movements in the external world, and yet is not meant to be noticed in itself. Temporal aspects of visual processing have received comparatively little attention in research. Temporal properties have been made explicit mainly in measurements of resolution and integration in simple tasks such as detection of spatially homogeneous flicker or light pulses of varying duration. Only through a mechanistic understanding of their basis in retinal photoreceptors and circuits can such measures guide modelling of natural vision in different species and illuminate functional and evolutionary trade-offs. Temporal vision research would benefit from bridging traditions that speak different languages. Towards that goal, I here review studies from the fields of human psychophysics, retinal physiology and neuroethology, with a focus on fundamental constraints set by early vision.

Summary: Simple measures of temporal vision such as the critical flicker frequency can be useful for modelling natural vision only if their relationship to photoreceptor responses and retinal processing is understood.

Scalable and accurate method for neuronal ensemble detection in spiking neural networks

We propose a novel, scalable, and accurate method for detecting neuronal ensembles from a population of spiking neurons. Our approach offers a simple yet powerful tool to study ensemble activity. It relies on clustering synchronous population activity (population vectors), allows the participation of neurons in different ensembles, has few parameters to tune and is computationally efficient. To validate the performance and generality of our method, we generated synthetic data, where we found that our method accurately detects neuronal ensembles for a wide range of simulation parameters. We found that our method outperforms current alternative methodologies. We used spike trains of retinal ganglion cells obtained from multi-electrode array recordings under a simple ON-OFF light stimulus to test our method. We found a consistent stimuli-evoked ensemble activity intermingled with spontaneously active ensembles and irregular activity. Our results suggest that the early visual system activity could be organized in distinguishable functional ensembles. We provide a Graphic User Interface, which facilitates the use of our method by the scientific community.

Feedback from retinal ganglion cells to the inner retina

Retinal ganglion cells (RGCs) are thought to be strictly postsynaptic within the retina. They carry visual signals from the eye to the brain, but do not make chemical synapses onto other retinal neurons. Nevertheless, they form gap junctions with other RGCs and amacrine cells, providing possibilities for RGC signals to feed back into the inner retina. Here we identified such feedback circuitry in the salamander and mouse retinas. First, using biologically inspired circuit models, we found mutual inhibition among RGCs of the same type. We then experimentally determined that this effect is mediated by gap junctions with amacrine cells. Finally, we found that this negative feedback lowers RGC visual response gain without affecting feature selectivity. The principal neurons of the retina therefore participate in a recurrent circuit much as those in other brain areas, not being a mere collector of retinal signals, but are actively involved in visual computations.