A computational-observer model of spatial contrast sensitivity: Effects of wave-front-based optics, cone-mosaic structure, and inference engine

High refresh rate display for natural monocular viewing in AOSLO psychophysics experiments

By combining an external display operating at 360 frames per second with an Adaptive Optics Scanning Laser Ophthalmoscope (AOSLO) for human foveal imaging, we demonstrate color stimulus delivery at high spatial and temporal resolution in AOSLO psychophysics experiments. A custom pupil relay enables viewing of the stimulus through a 3-mm effective pupil diameter and provides refractive error correction from -8 to +4 diopters. Performance of the assembled and aligned pupil relay was validated by measuring the wavefront error across the field of view and correction range, and the as-built Strehl ratio was 0.64 or better. High-acuity stimuli were rendered on the external display and imaged through the pupil relay to demonstrate that spatial frequencies up to 54 cycles per degree, corresponding to 20/11 visual acuity, are resolved. The completed external display was then used to render fixation markers across the field of view of the monitor, and a continuous retinal montage spanning 9.4 by 5.4 degrees of visual angle was acquired with the AOSLO. We conducted eye-tracking experiments during free-viewing and high-acuity tasks with polychromatic images presented on the external display. Sub-arcminute eye position uncertainty was achieved, enabling precise localization of the line of sight on the monitor while simultaneously imaging the fine structure of the human central fovea. This high refresh rate display overcomes the temporal, spectral, and field of view limitations of AOSLO-based stimulus presentation, enabling natural monocular viewing of stimuli in psychophysics experiments conducted with AOSLO.

Theoretical impact of chromatic aberration correction on visual acuity

It has been known for more than 220 years that the image quality of the human eye is significantly degraded by chromatic aberrations. Recently, it was shown experimentally that correcting chromatic aberrations results in a 0.2- to 0.8-line improvement in visual acuity. Here we ask, is this expected? We developed tools that enable simulations of the optical impact of physiologically relevant amounts of chromatic aberration in real human eyes and combined these with tools that compute the visual acuity of an ideal observer. This allows us to characterize the theoretical impact of chromatic aberration correction on visual acuity. Results indicate a substantive improvement of 0.4- to 2-lines in ideal observer visual acuity with chromatic aberration correction. Ideal observer thresholds benefit significantly more from correction of longitudinal than correction of transverse chromatic aberration. Finally, improvements in ideal observer visual acuity are greater for subjects with less monochromatic aberration, such that subjects with better baseline optical quality benefit most from correction of chromatic aberrations.

Eccentricity strongly modulates visual processing delays

The temporal dynamics of visual information processing varies with the stimulus being processed and with the retinal location that initiates the processing. Here, we present psychophysical data with sub-millisecond precision showing that increasing eccentricity decreases the delay with which stimuli are processed. We show that, even within the central +/-6° of the visual field, processing delays change by a factor of up to three times. A simple model, grounded in retinal physiology, provides a good account of the data. The relative delays are on the order of only milliseconds. But if later processing leaves the delays unresolved, they can cause dramatic misperceptions of motion and 3D layout. We discuss the implications for how the human visual system solves the temporal binding problem across eccentricity. The results highlight the severe computational challenge of obtaining accurate, temporally-unified percepts of the environment with spatiotemporally-staggered processing across the visual field.

Effect of fixational eye movements in corneal topography measurements with optical coherence tomography

There is an increasing interest in applying optical coherence tomography (OCT) to quantify the topography of ocular structures. However, in its most usual configuration, OCT data is acquired sequentially while a beam is scanned through the region of interest, and the presence of fixational eye movements can affect the accuracy of the technique. Several scan patterns and motion correction algorithms have been proposed to minimize this effect, but there is no consensus on the ideal parameters to obtain a correct topography. We have acquired corneal OCT images with raster and radial patterns, and modeled the data acquisition in the presence of eye movements. The simulations replicate the experimental variability in shape (radius of curvature and Zernike polynomials), corneal power, astigmatism, and calculated wavefront aberrations. The variability of the Zernike modes is highly dependent on the scan pattern, with higher variability in the direction of the slow scan axis. The model can be a useful tool to design motion correction algorithms and to determine the variability with different scan patterns.

Texture statistics involved in specular highlight exclusion for object lightness perception

The human visual system estimates the physical properties of objects, such as their lightness. Previous studies on the lightness perception of glossy three-dimensional objects have suggested that specular highlights are detected and excluded in lightness perception. However, only a few studies have attempted to elucidate the mechanisms underlying this exclusion. This study aimed to elucidate the image features that contribute to the highlight exclusion of lightness perception. We used Portilla-Simoncelli texture statistics (PS statistics), an image feature set similar to the representation in the early visual cortex, to explore their relationships with highlight exclusion for lightness perception. In experiment 1, computer graphics images of bumpy plastic plates with various physical parameters were used as stimuli, and the lightness perception on them was measured using a lightness matching task. We then calculated the highlight exclusion index, which represented the degree of highlight exclusion. Finally, we evaluated the correlation between the highlight exclusion index and the four PS statistic subsets. In experiment 2, an image synthesis algorithm was used to create images in which either the PS statistic subset was manipulated. The highlight exclusion indexes of the synthesized images were then measured. The results revealed that the PS statistic subset consisting of lowest-order image features, such as moment statistics of luminance, acts as a necessary condition for highlight exclusion, whereas the other three subsets consisting of higher order features are not crucial. These results suggest that the low-order image features are the most important among the features in PS statistics for highlight exclusion, even though image features higher order than those in PS statistics must be directly involved.

In vivo chromatic and spatial tuning of foveolar retinal ganglion cells in Macaca fascicularis

The primate fovea is specialized for high acuity chromatic vision, with the highest density of cone photoreceptors and a disproportionately large representation in visual cortex. The unique visual properties conferred by the fovea are conveyed to the brain by retinal ganglion cells, the somas of which lie at the margin of the foveal pit. Microelectrode recordings of these centermost retinal ganglion cells have been challenging due to the fragility of the fovea in the excised retina. Here we overcome this challenge by combining high resolution fluorescence adaptive optics ophthalmoscopy with calcium imaging to optically record functional responses of foveal retinal ganglion cells in the living eye. We use this approach to study the chromatic responses and spatial transfer functions of retinal ganglion cells using spatially uniform fields modulated in different directions in color space and monochromatic drifting gratings. We recorded from over 350 cells across three Macaca fascicularis primates over a time period of weeks to months. We find that the majority of the L vs. M cone opponent cells serving the most central foveolar cones have spatial transfer functions that peak at high spatial frequencies (20-40 c/deg), reflecting strong surround inhibition that sacrifices sensitivity at low spatial frequencies but preserves the transmission of fine detail in the retinal image. In addition, we fit to the drifting grating data a detailed model of how ganglion cell responses draw on the cone mosaic to derive receptive field properties of L vs. M cone opponent cells at the very center of the foveola. The fits are consistent with the hypothesis that foveal midget ganglion cells are specialized to preserve information at the resolution of the cone mosaic. By characterizing the functional properties of retinal ganglion cells in vivo through adaptive optics, we characterize the response characteristics of these cells in situ.

Optical modelling of an accommodative light field display system and prediction of human eye responses

The spatio-angular resolution of a light field (LF) display is a crucial factor for delivering adequate spatial image quality and eliciting an accommodation response. Previous studies have modelled retinal image formation with an LF display and evaluated whether accommodation would be evoked correctly. The models were mostly based on ray-tracing and a schematic eye model, which pose computational complexity and inaccurately represent the human eye population's behaviour. We propose an efficient wave-optics-based framework to model the human eye and a general LF display. With the model, we simulated the retinal point spread function (PSF) of a point rendered by an LF display at various depths to characterise the retinal image quality. Additionally, accommodation responses to the rendered point were estimated by computing the visual Strehl ratio based on the optical transfer function (VSOTF) from the PSFs. We assumed an ideal LF display that had an infinite spatial resolution and was free from optical aberrations in the simulation. We tested points rendered at 0-4 dioptres of depths having angular resolutions of up to 4x4 viewpoints within a pupil. The simulation predicted small and constant accommodation errors, which contradict the findings of previous studies. An evaluation of the optical resolution on the retina suggested a trade-off between the maximum achievable resolution and the depth range of a rendered point where in-focus resolution is kept high. The proposed framework can be used to evaluate the upper bound of the optical performance of an LF display for realistically aberrated eyes, which may help to find an optimal spatio-angular resolution required to render a high quality 3D scene.

Correlation between higher-order aberration and photophobia after cataract surgery

Cataract surgery impinges on the spatial properties and wavelength distribution of retinal images, which changes the degree of light-induced visual discomfort/photophobia. However, no study has analyzed the alteration in photophobia before and after cataract surgery or the association between retinal spatial property and photophobia. Here, we measured the higher-order aberrations (HOAs) of the entire eye and the subjective photophobia score. This study investigated 71 eyes in 71 patients who received conventional cataract surgery. Scaling of photophobia was based on the following grading system: when the patient is outdoor on a sunny day, score of 0 and 10 points were assigned to the absence of photophobia and the presence of severe photophobia prevents eye-opening, respectively. We decomposed wavefront errors using Zernike polynomials for a 3-mm pupil diameter and analyzed the association between photophobia scores and HOAs with Spearman’s rank sum correlation (rs). We classified patients into two groups: photophobia (PP) unconcerned included patients who selected 0 both preoperatively or postoperatively and PP concerned included the remaining patients. After cataract surgery, photophobia scores increased, remained unchanged (stable), and decreased in 3, 41, and 27 cases, respectively. In the stable group, 35 of 41 cases belonged to PP unconcerned. In PP concerned, there were significant correlations between photophobia score and postoperative root-mean-square values of total HOAs (rs = 0.52, p = 0.002), total coma (rs = 0.52, p = 0.002), total trefoil (rs = 0.47, p = 0.006), and third-order group (rs = 0.53, p = 0.002). In contrast, there was no significant correlation between photophobia scores and preoperative HOAs. Our results suggest that the spatial properties of retinal image modified by HOAs may affect the degree of photophobia. Scattering light due to cataracts could contribute to photophobia more than HOAs, which may mask the effect of HOAs for photophobia preoperatively.

Contrast sensitivity functions in autoencoders

Three decades ago, Atick et al. suggested that human frequency sensitivity may emerge from the enhancement required for a more efficient analysis of retinal images. Here we reassess the relevance of low-level vision tasks in the explanation of the contrast sensitivity functions (CSFs) in light of 1) the current trend of using artificial neural networks for studying vision, and 2) the current knowledge of retinal image representations. As a first contribution, we show that a very popular type of convolutional neural networks (CNNs), called autoencoders, may develop human-like CSFs in the spatiotemporal and chromatic dimensions when trained to perform some basic low-level vision tasks (like retinal noise and optical blur removal), but not others (like chromatic) adaptation or pure reconstruction after simple bottlenecks). As an illustrative example, the best CNN (in the considered set of simple architectures for enhancement of the retinal signal) reproduces the CSFs with a root mean square error of 11% of the maximum sensitivity. As a second contribution, we provide experimental evidence of the fact that, for some functional goals (at low abstraction level), deeper CNNs that are better in reaching the quantitative goal are actually worse in replicating human-like phenomena (such as the CSFs). This low-level result (for the explored networks) is not necessarily in contradiction with other works that report advantages of deeper nets in modeling higher level vision goals. However, in line with a growing body of literature, our results suggests another word of caution about CNNs in vision science because the use of simplified units or unrealistic architectures in goal optimization may be a limitation for the modeling and understanding of human vision.

Equivalent noise characterization of human lightness constancy

A goal of visual perception is to provide stable representations of task-relevant scene properties (e.g. object reflectance) despite variation in task-irrelevant scene properties (e.g. illumination and reflectance of other nearby objects). To study such stability in the context of the perceptual representation of lightness, we introduce a threshold-based psychophysical paradigm. We measure how thresholds for discriminating the achromatic reflectance of a target object (task-relevant property) in rendered naturalistic scenes are impacted by variation in the reflectance functions of background objects (task-irrelevant property), using a two-alternative forced-choice paradigm in which the reflectance of the background objects is randomized across the two intervals of each trial. We control the amount of background reflectance variation by manipulating a statistical model of naturally occurring surface reflectances. For low background object reflectance variation, discrimination thresholds were nearly constant, indicating that observers’ internal noise determines threshold in this regime. As background object reflectance variation increases, its effects start to dominate performance. A model based on signal detection theory allows us to express the effects of task-irrelevant variation in terms of the equivalent noise, that is relative to the intrinsic precision of the task-relevant perceptual representation. The results indicate that although naturally occurring background object reflectance variation does intrude on the perceptual representation of target object lightness, the effect is modest – within a factor of two of the equivalent noise level set by internal noise.

An image reconstruction framework for characterizing initial visual encoding

We developed an image-computable observer model of the initial visual encoding that operates on natural image input, based on the framework of Bayesian image reconstruction from the excitations of the retinal cone mosaic. Our model extends previous work on ideal observer analysis and evaluation of performance beyond psychophysical discrimination, takes into account the statistical regularities of the visual environment, and provides a unifying framework for answering a wide range of questions regarding the visual front end. Using the error in the reconstructions as a metric, we analyzed variations of the number of different photoreceptor types on human retina as an optimal design problem. In addition, the reconstructions allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling, and how these vary with eccentricity. Furthermore, in simulations of color deficiencies and interferometric experiments, we found that the reconstructed images provide a reasonable proxy for modeling subjects' percepts. Lastly, we used the reconstruction-based observer for the analysis of psychophysical threshold, and found notable interactions between spatial frequency and chromatic direction in the resulting spatial contrast sensitivity function. Our method is widely applicable to experiments and applications in which the initial visual encoding plays an important role.

Asymmetries around the visual field: From retina to cortex to behavior

Visual performance varies around the visual field. It is best near the fovea compared to the periphery, and at iso-eccentric locations it is best on the horizontal, intermediate on the lower, and poorest on the upper meridian. The fovea-to-periphery performance decline is linked to the decreases in cone density, retinal ganglion cell (RGC) density, and V1 cortical magnification factor (CMF) as eccentricity increases. The origins of polar angle asymmetries are not well understood. Optical quality and cone density vary across the retina, but recent computational modeling has shown that these factors can only account for a small percentage of behavior. Here, we investigate how visual processing beyond the cone photon absorptions contributes to polar angle asymmetries in performance. First, we quantify the extent of asymmetries in cone density, midget RGC density, and V1 CMF. We find that both polar angle asymmetries and eccentricity gradients increase from cones to mRGCs, and from mRGCs to cortex. Second, we extend our previously published computational observer model to quantify the contribution of phototransduction by the cones and spatial filtering by mRGCs to behavioral asymmetries. Starting with photons emitted by a visual display, the model simulates the effect of human optics, cone isomerizations, phototransduction, and mRGC spatial filtering. The model performs a forced choice orientation discrimination task on mRGC responses using a linear support vector machine classifier. The model shows that asymmetries in a decision maker's performance across polar angle are greater when assessing the photocurrents than when assessing isomerizations and are greater still when assessing mRGC signals. Nonetheless, the polar angle asymmetries of the mRGC outputs are still considerably smaller than those observed from human performance. We conclude that cone isomerizations, phototransduction, and the spatial filtering properties of mRGCs contribute to polar angle performance differences, but that a full account of these differences will entail additional contribution from cortical representations.

Cortical magnification in human visual cortex parallels task performance around the visual field

Human vision has striking radial asymmetries, with performance on many tasks varying sharply with stimulus polar angle. Performance is generally better on the horizontal than vertical meridian, and on the lower than upper vertical meridian, and these asymmetries decrease gradually with deviation from the vertical meridian. Here, we report cortical magnification at a fine angular resolution around the visual field. This precision enables comparisons between cortical magnification and behavior, between cortical magnification and retinal cell densities, and between cortical magnification in twin pairs. We show that cortical magnification in the human primary visual cortex, measured in 163 subjects, varies substantially around the visual field, with a pattern similar to behavior. These radial asymmetries in the cortex are larger than those found in the retina, and they are correlated between monozygotic twin pairs. These findings indicate a tight link between cortical topography and behavior, and suggest that visual field asymmetries are partly heritable.

A quadratic model captures the human V1 response to variations in chromatic direction and contrast

An important goal for vision science is to develop quantitative models of the representation of visual signals at post-receptoral sites. To this end, we develop the quadratic color model (QCM) and examine its ability to account for the BOLD fMRI response in human V1 to spatially-uniform, temporal chromatic modulations that systematically vary in chromatic direction and contrast. We find that the QCM explains the same, cross-validated variance as a conventional general linear model, with far fewer free parameters. The QCM generalizes to allow prediction of V1 responses to a large range of modulations. We replicate the results for each subject and find good agreement across both replications and subjects. We find that within the LM cone contrast plane, V1 is most sensitive to L-M contrast modulations and least sensitive to L+M contrast modulations. Within V1, we observe little to no change in chromatic sensitivity as a function of eccentricity.

The Relationship Between Visual Sensitivity and Eccentricity, Cone Density and Outer Segment Length in the Human Foveola

Purpose

The cellular topography of the human foveola, the central 1° diameter of the fovea, is strikingly non-uniform, with a steep increase of cone photoreceptor density and outer segment (OS) length toward its center. Here, we assessed to what extent the specific cellular organization of the foveola of an individual is reflected in visual sensitivity and if sensitivity peaks at the preferred retinal locus of fixation (PRL).

Methods

Increment sensitivity to small-spot, cone-targeted visual stimuli (1 × 1 arcmin, 543-nm light) was recorded psychophysically in four human participants at 17 locations concentric within a 0.2° diameter on and around the PRL with adaptive optics scanning laser ophthalmoscopy-based microstimulation. Sensitivity test spots were aligned with cell-resolved maps of cone density and cone OS length.

Results

Peak sensitivity was at neither the PRL nor the topographical center of the cone mosaic. Within the central 0.1° diameter, a plateau-like sensitivity profile was observed. Cone density and maximal OS length differed significantly across participants, correlating with their peak sensitivity. Based on these results, biophysical simulation allowed to develop a model of visual sensitivity in the foveola, with distance from the PRL (eccentricity), cone density, and OS length as parameters.

Conclusions

Small-spot sensitivity thresholds in healthy retinas will help to establish the range of normal foveolar function in cell-targeted vision testing. Because of the high reproducibility in replicate testing, threshold variability not explained by our model is assumed to be caused by individual cone and bipolar cell weighting at the specific target locations.

Population Models, Not Analyses, of Human Neuroscience Measurements

Selectivity for many basic properties of visual stimuli, such as orientation, is thought to be organized at the scale of cortical columns, making it difficult or impossible to measure directly with noninvasive human neuroscience measurement. However, computational analyses of neuroimaging data have shown that selectivity for orientation can be recovered by considering the pattern of response across a region of cortex. This suggests that computational analyses can reveal representation encoded at a finer spatial scale than is implied by the spatial resolution limits of measurement techniques. This potentially opens up the possibility to study a much wider range of neural phenomena that are otherwise inaccessible through noninvasive measurement. However, as we review in this article, a large body of evidence suggests an alternative hypothesis to this superresolution account: that orientation information is available at the spatial scale of cortical maps and thus easily measurable at the spatial resolution of standard techniques. In fact, a population model shows that this orientation information need not even come from single-unit selectivity for orientation tuning, but instead can result from population selectivity for spatial frequency. Thus, a categorical error of interpretation can result whereby orientation selectivity can be confused with spatial frequency selectivity. This is similarly problematic for the interpretation of results from numerous studies of more complex representations and cognitive functions that have built upon the computational techniques used to reveal stimulus orientation. We suggest in this review that these interpretational ambiguities can be avoided by treating computational analyses as models of the neural processes that give rise to measurement. Building upon the modeling tradition in vision science using considerations of whether population models meet a set of core criteria is important for creating the foundation for a cumulative and replicable approach to making valid inferences from human neuroscience measurements. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Time-Varying Light Exposure in Chronobiology and Sleep Research Experiments

Light exposure profoundly affects human physiology and behavior through circadian and neuroendocrine photoreception primarily through the melanopsin-containing intrinsically photosensitive retinal ganglion cells. Recent research has explored the possibility of using temporally patterned stimuli to manipulate circadian and neuroendocrine responses to light. This mini-review, geared to chronobiologists, sleep researchers, and scientists in adjacent disciplines, has two objectives: (1) introduce basic concepts in time-varying stimuli and (2) provide a checklist-based set of recommendations for documenting time-varying light exposures based on current best practices and standards.

Emulated retinal image capture (ERICA) to test, train and validate processing of retinal images

High resolution retinal imaging systems, such as adaptive optics scanning laser ophthalmoscopes (AOSLO), are increasingly being used for clinical research and fundamental studies in neuroscience. These systems offer unprecedented spatial and temporal resolution of retinal structures in vivo. However, a major challenge is the development of robust and automated methods for processing and analysing these images. We present ERICA (Emulated Retinal Image CApture), a simulation tool that generates realistic synthetic images of the human cone mosaic, mimicking images that would be captured by an AOSLO, with specified image quality and with corresponding ground-truth data. The simulation includes a self-organising mosaic of photoreceptors, the eye movements an observer might make during image capture, and data capture through a real system incorporating diffraction, residual optical aberrations and noise. The retinal photoreceptor mosaics generated by ERICA have a similar packing geometry to human retina, as determined by expert labelling of AOSLO images of real eyes. In the current implementation ERICA outputs convincingly realistic en face images of the cone photoreceptor mosaic but extensions to other imaging modalities and structures are also discussed. These images and associated ground-truth data can be used to develop, test and validate image processing and analysis algorithms or to train and validate machine learning approaches. The use of synthetic images has the advantage that neither access to an imaging system, nor to human participants is necessary for development.

A computational observer model of spatial contrast sensitivity: Effects of photocurrent encoding, fixational eye movements, and inference engine

We have recently shown that the relative spatial contrast sensitivity function (CSF) of a computational observer operating on the cone mosaic photopigment excitations of a stationary retina has the same shape as human subjects. Absolute human sensitivity, however, is 5- to 10-fold lower than the computational observer. Here we model how additional known features of early vision affect the CSF: fixational eye movements and the conversion of cone photopigment excitations to cone photocurrents (phototransduction). For a computational observer that uses a linear classifier applied to the responses of a stimulus-matched linear filter, fixational eye movements substantially change the shape of the CSF by reducing sensitivity above 10 c/deg. For a translation-invariant computational observer that operates on the squared response of a quadrature-pair of linear filters, the CSF shape is little changed by eye movements, but there is a two fold reduction in sensitivity. Phototransduction dynamics introduce an additional two fold sensitivity decrease. Hence, the combined effects of fixational eye movements and phototransduction bring the absolute CSF of the translation-invariant computational observer to within a factor of 1 to 2 of the human CSF. We note that the human CSF depends on processing of the retinal representation by many thalamo-cortical neurons, which are individually quite noisy. Our modeling suggests that the net effect of post-retinal noise on contrast-detection performance, when considered at the neural population and behavioral level, is quite small: The inference mechanisms that determine the CSF, presumably in cortex, make efficient use of the information carried by the cone photocurrents of the fixating eye.

Orientation-specific long-term neural adaptation of the visual system in keratoconus

Eyes with the corneal ectasia keratoconus have performed better than expected (e.g. visual acuity) given their elevated levels of higher-order aberrations that cause rotationally asymmetric retinal blur. Adapted neural processing has been suggested as an explanation but has not been measured across multiple meridional orientations. Using a custom Maxwellian-view laser interferometer to bypass ocular optics, sinusoidal grating neural contrast sensitivity was measured in six eyes (three subjects) with keratoconus and four typical eyes (two subjects) at six spatial frequencies and eight orientations using a two-interval forced-choice paradigm. Total measurement duration was 24 to 28 hours per subject. Neural contrast sensitivity functions of typical eyes agreed with literature and generally showed the oblique effect on a linear-scale and rotational symmetry on a log-scale (rotational symmetry was quantified as the ratio of the minor and major radii of an ellipse fit to all orientations within each spatial frequency; typical eye mean 0.93, median 0.93; where a circle = 1). Mean sensitivities of eyes with keratoconus were 20% to 60% lower (at lower and higher spatial frequencies respectively) than typical eyes. Orientation-specific neural contrast sensitivity functions in keratoconus showed substantial rotational asymmetry (ellipse radii ratio: mean 0.84; median 0.86) and large meridional reductions. The visual image quality metric VSX was used with a permutation test to combine the asymmetric optical aberrations of the eyes with keratoconus and their measured asymmetric neural functions, which illustrated how the neural sensitivities generally mitigated the detrimental effects of the optics.

Color, Pattern, and the Retinal Cone Mosaic

Textbook trichromacy accounts for human color vision in terms of spectral sampling by three classes of cone photoreceptors. This account neglects entangling of color and pattern information created by wavelength-dependent optical blur (chromatic aberrations) and interleaved spatial sampling of the retinal image by the three classes of cones. Recent experimental, computational, and neurophysiological work is now considering color and pattern vision at the elementary scale of daylight vison, that is at the scale of individual cones. The results provide insight about rich interactions between color and pattern vision as well as the role of the statistical structure of natural scenes in shaping visual processing.

Ray tracing 3D spectral scenes through human optics models

Scientists and engineers have created computations and made measurements that characterize the first steps of seeing. ISETBio software integrates such computations and data into an open-source software package. The initial ISETBio implementations modeled image formation (physiological optics) for planar or distant scenes. The ISET3d software described here extends that implementation, simulating image formation for three-dimensional scenes. The software system relies on a quantitative computer graphics program that ray traces the scene radiance through the physiological optics to the retinal irradiance. We describe and validate the implementation for several model eyes. Then, we use the software to quantify the impact of several physiological optics parameters on three-dimensional image formation. ISET3d is integrated with ISETBio, making it straightforward to convert the retinal irradiance into cone excitations. These methods help the user compute the predictions of optics models for a wide range of spatially rich three-dimensional scenes. They can also be used to evaluate the impact of nearby visual occlusion, the information available to binocular vision, or the retinal images expected from near-field and augmented reality displays.

Computational-observer analysis of illumination discrimination

The spectral properties of the ambient illumination provide useful information about time of day and weather. We study the perceptual representation of illumination by analyzing measurements of how well people discriminate between illuminations across scene configurations. More specifically, we compare human performance to a computational-observer analysis that evaluates the information available in the isomerizations of cone photopigment in a model human photoreceptor mosaic. The performance of such an observer is limited by the Poisson variability of the number of isomerizations in each cone. The overall level of Poisson-limited computational-observer sensitivity exceeded that of human observers. This was modeled by increasing the amount of noise in the number of isomerizations of each cone. The additional noise brought the overall level of performance of the computational observer into the same range as that of human observers, allowing us to compare the pattern of sensitivity across stimulus manipulations. Key patterns of human performance were not accounted for by the computational observer. In particular, neither the elevation of illumination-discrimination thresholds for illuminant changes in a blue color direction (when thresholds are expressed in CIELUV ΔE units), nor the effects of varying the ensemble of surfaces in the scenes being viewed, could be accounted for by variation in the information available in the cone isomerizations.

Modeling visual performance differences 'around' the visual field: A computational observer approach

Visual performance depends on polar angle, even when eccentricity is held constant; on many psychophysical tasks observers perform best when stimuli are presented on the horizontal meridian, worst on the upper vertical, and intermediate on the lower vertical meridian. This variation in performance 'around' the visual field can be as pronounced as that of doubling the stimulus eccentricity. The causes of these asymmetries in performance are largely unknown. Some factors in the eye, e.g. cone density, are positively correlated with the reported variations in visual performance with polar angle. However, the question remains whether these correlations can quantitatively explain the perceptual differences observed 'around' the visual field. To investigate the extent to which the earliest stages of vision-optical quality and cone density-contribute to performance differences with polar angle, we created a computational observer model. The model uses the open-source software package ISETBIO to simulate an orientation discrimination task for which visual performance differs with polar angle. The model starts from the photons emitted by a display, which pass through simulated human optics with fixational eye movements, followed by cone isomerizations in the retina. Finally, we classify stimulus orientation using a support vector machine to learn a linear classifier on the photon absorptions. To account for the 30% increase in contrast thresholds for upper vertical compared to horizontal meridian, as observed psychophysically on the same task, our computational observer model would require either an increase of ~7 diopters of defocus or a reduction of 500% in cone density. These values far exceed the actual variations as a function of polar angle observed in human eyes. Therefore, we conclude that these factors in the eye only account for a small fraction of differences in visual performance with polar angle. Substantial additional asymmetries must arise in later retinal and/or cortical processing.