Statistics of spatial cone-excitation ratios in natural scenes

What is white?

To shed light on the perceptual basis of the color white, we measured settings of unique white in a dark surround. We find that settings reliably show more variability in an oblique (blue-yellow) direction in color space than along the cardinal axes of the cone-opponent mechanisms. This is against the idea that white perception arises at the null point of the cone-opponent mechanisms, but one alternative possibility is that it occurs through calibration to the visual environment. We found that the locus of maximum variability in settings lies close to the locus of natural daylights, suggesting that variability may result from uncertainty about the color of the illuminant. We tested this by manipulating uncertainty. First, we altered the extent to which the task was absolute (requiring knowledge of the illumination) or relative. We found no clear effect of this factor on the reduction in sensitivity in the blue-yellow direction. Second, we provided a white surround as a cue to the illumination or left the surround dark. Sensitivity was selectively worse in the blue-yellow direction when the surround was black than when it was white. Our results can be functionally related to the statistics of natural images, where a greater blue-yellow dispersion is characteristic of both reflectances (where anisotropy is weak) and illuminants (where it is very pronounced). Mechanistically, the results could suggest a neural signal responsive to deviations from the blue-yellow locus or an adaptively matched range of contrast response functions for signals that encode different directions in color space.

Image-difference prediction: from color to spectral

We propose a new strategy to evaluate the quality of multi and hyperspectral images, from the perspective of human perception. We define the spectral image difference as the overall perceived difference between two spectral images under a set of specified viewing conditions (illuminants). First, we analyze the stability of seven image-difference features across illuminants, by means of an information-theoretic strategy. We demonstrate, in particular, that in the case of common spectral distortions (spectral gamut mapping, spectral compression, spectral reconstruction), chromatic features vary much more than achromatic ones despite considering chromatic adaptation. Then, we propose two computationally efficient spectral image difference metrics and compare them to the results of a subjective visual experiment. A significant improvement is shown over existing metrics such as the widely used root-mean square error.

Contributions of rhodopsin, cone opsins, and melanopsin to postreceptoral pathways inferred from natural image statistics

Visual neural representation is constrained by the statistical properties of the environment. Prior analysis of cone pigment excitations for natural images revealed three principal components corresponding to the major retinogeniculate pathways identified by anatomical and physiological studies in primates. Here, principal component analyses were conducted on the excitations of rhodopsin, cone opsins, and melanopsin for nine hyperspectral images under 21 natural illuminants. The results suggested that rhodopsin and melanopsin may contribute to the three major retinogeniculate pathways. Rhodopsin and melanopsin may provide additional constraints in natural scene statistics, leading to new components that cannot be revealed by analysis based on cone opsin excitations only.

Adaptive colour contrast coding in the salamander retina efficiently matches natural scene statistics

The visual system continually adjusts its sensitivity to the statistical properties of the environment through an adaptation process that starts in the retina. Colour perception and processing is commonly thought to occur mainly in high visual areas, and indeed most evidence for chromatic colour contrast adaptation comes from cortical studies. We show that colour contrast adaptation starts in the retina where ganglion cells adjust their responses to the spectral properties of the environment. We demonstrate that the ganglion cells match their responses to red-blue stimulus combinations according to the relative contrast of each of the input channels by rotating their functional response properties in colour space. Using measurements of the chromatic statistics of natural environments, we show that the retina balances inputs from the two (red and blue) stimulated colour channels, as would be expected from theoretical optimal behaviour. Our results suggest that colour is encoded in the retina based on the efficient processing of spectral information that matches spectral combinations in natural scenes on the colour processing level.

Processing of the S-cone signals in the early visual cortex of primates

The short-wavelength-sensitive (S) cones play an important role in color vision of primates, and may also contribute to the coding of other visual features, such as luminance and motion. The color signals carried by the S cones and other cone types are largely separated in the subcortical visual pathway. Studies on nonhuman primates or humans have suggested that these signals are combined in the striate cortex (V1) following a substantial amplification of the S-cone signals in the same area. In addition to reviewing these studies, this review describes the circuitry in V1 that may underlie the processing of the S-cone signals and the dynamics of this processing. It also relates the interaction between various cone signals in V1 to the results of some psychophysical and physiological studies on color perception, which leads to a discussion of a previous model, in which color perception is produced by a multistage processing of the cone signals. Finally, I discuss the processing of the S-cone signals in the extrastriate area V2.

Estimating information from image colors: an application to digital cameras and natural scenes

The colors present in an image of a scene provide information about its constituent elements. But the amount of information depends on the imaging conditions and on how information is calculated. This work had two aims. The first was to derive explicitly estimators of the information available and the information retrieved from the color values at each point in images of a scene under different illuminations. The second was to apply these estimators to simulations of images obtained with five sets of sensors used in digital cameras and with the cone photoreceptors of the human eye. Estimates were obtained for 50 hyperspectral images of natural scenes under daylight illuminants with correlated color temperatures 4,000, 6,500, and 25,000 K. Depending on the sensor set, the mean estimated information available across images with the largest illumination difference varied from 15.5 to 18.0 bits and the mean estimated information retrieved after optimal linear processing varied from 13.2 to 15.5 bits (each about 85 percent of the corresponding information available). With the best sensor set, 390 percent more points could be identified per scene than with the worst. Capturing scene information from image colors depends crucially on the choice of camera sensors.

Nonlinearities and adaptation of color vision from sequential principal curves analysis

Mechanisms of human color vision are characterized by two phenomenological aspects: the system is nonlinear and adaptive to changing environments. Conventional attempts to derive these features from statistics use separate arguments for each aspect. The few statistical explanations that do consider both phenomena simultaneously follow parametric formulations based on empirical models. Therefore, it may be argued that the behavior does not come directly from the color statistics but from the convenient functional form adopted. In addition, many times the whole statistical analysis is based on simplified databases that disregard relevant physical effects in the input signal, as, for instance, by assuming flat Lambertian surfaces. In this work, we address the simultaneous statistical explanation of the nonlinear behavior of achromatic and chromatic mechanisms in a fixed adaptation state and the change of such behavior (i.e., adaptation) under the change of observation conditions. Both phenomena emerge directly from the samples through a single data-driven method: the sequential principal curves analysis (SPCA) with local metric. SPCA is a new manifold learning technique to derive a set of sensors adapted to the manifold using different optimality criteria. Here sequential refers to the fact that sensors (curvilinear dimensions) are designed one after the other, and not to the particular (eventually iterative) method to draw a single principal curve. Moreover, in order to reproduce the empirical adaptation reported under D65 and A illuminations, a new database of colorimetrically calibrated images of natural objects under these illuminants was gathered, thus overcoming the limitations of available databases. The results obtained by applying SPCA show that the psychophysical behavior on color discrimination thresholds, discount of the illuminant, and corresponding pairs in asymmetric color matching emerge directly from realistic data regularities, assuming no a priori functional form. These results provide stronger evidence for the hypothesis of a statistically driven organization of color sensors. Moreover, the obtained results suggest that the nonuniform resolution of color sensors at this low abstraction level may be guided by an error-minimization strategy rather than by an information-maximization goal.

Digital camera simulation

We describe a simulation of the complete image processing pipeline of a digital camera, beginning with a radiometric description of the scene captured by the camera and ending with a radiometric description of the image rendered on a display. We show that there is a good correspondence between measured and simulated sensor performance. Through the use of simulation, we can quantify the effects of individual digital camera components on system performance and image quality. This computational approach can be helpful for both camera design and image quality assessment.

Contrast edge colors under different natural illuminations

Essential to sensory processing in the human visual system is natural illumination, which can vary considerably not only across space but also along the day depending on the atmospheric conditions and the sun's position in the sky. In this work, edges derived from the three postreceptoral Luminance, Red-Green, and Blue-Yellow signals were computed from hyperspectral images of natural scenes rendered with daylights of Correlated Color Temperatures (CCTs) from 2735 to 25,889 K; for low CCT, the same analysis was performed using Planckian illuminants up to 800 K. It was found that average luminance and chromatic edge contrasts were maximal for low correlated color temperatures and almost constants above 10,000 K. The magnitude of these contrast changes was, however, only about 2% across the tested daylights. Results suggest that the postreceptoral opponent and nonopponent color vision mechanisms produce almost constant responses for color edge detection under natural illumination.

Predicting frequency of metamerism in natural scenes by entropy of colors

Estimating the frequency of metameric surfaces in natural scenes usually requires many comparisons of surface colors to determine which are visually indistinguishable under one light but distinguishable-by a certain criterion degree-under another. The aim here was to test the predictive power of a simpler approach to estimation based on the entropy of colors. In simulations with 50 hyperspectral images of natural scenes, the logarithm of the observed relative frequency of metamerism in each scene under two successive daylights was regressed on combinations of the estimated Shannon differential entropies of the colors of the scene under the same two daylights. The regression was strong, and it remained so when restricted to the estimated differential entropy under just the first daylight, providing that the criterion degree of metamerism was limited. When the criterion degree was made more extreme, however, the restricted regression failed. A possible explanation of the predictive power of differential entropy is briefly considered.

Are Gaussian spectra a viable perceptual assumption in color appearance?

Natural illuminant and reflectance spectra can be roughly approximated by a linear model with as few as three basis functions, and this has suggested that the visual system might construct a linear representation of the spectra by estimating the weights of these functions. However, such models do not accommodate nonlinearities in color appearance, such as the Abney effect. Previously, we found that these nonlinearities are qualitatively consistent with a perceptual inference that stimulus spectra are instead roughly Gaussian, with the hue tied to the inferred centroid of the spectrum [J. Vision 6(9), 12 (2006)]. Here, we examined to what extent a Gaussian inference provides a sufficient approximation of natural color signals. Reflectance and illuminant spectra from a wide set of databases were analyzed to test how well the curves could be fit by either a simple Gaussian with three parameters (amplitude, peak wavelength, and standard deviation) versus the first three principal component analysis components of standard linear models. The resulting Gaussian fits were comparable to linear models with the same degrees of freedom, suggesting that the Gaussian model could provide a plausible perceptual assumption about stimulus spectra for a trichromatic visual system.

Systematic violations of von Kries rule reveal its limitations for explaining color and lightness constancy

Cone contrast remains constant, when the same object/background is seen under different illuminations-the von Kries rule [Shevell, Vis. Res. 18, 1649 (1978)]. Here we explore this idea using asymmetric color matching. We find that von Kries adaptation holds, regardless of whether chromatic constancy index is low or high. When illumination changes the stimulus luminance (reflectance), lightness constancy is weak and matching is dictated by object/background luminance contrast. When this contrast is masked or disrupted, lightness constancy mechanisms are more prominent. Thus von Kries adaptation is incompatible with lightness constancy, suggesting that cortical mechanisms must underlie color constancy, as expected from neurophysiological studies [Zeki, Nature 284, 412 (1980); Wild, Nature 313, 133 (1985)].

Uniform color spaces and natural image statistics

Many aspects of visual coding have been successfully predicted by starting from the statistics of natural scenes and then asking how the stimulus could be efficiently represented. We started from the representation of color characterized by uniform color spaces, and then asked what type of color environment they implied. These spaces are designed to represent equal perceptual differences in color discrimination or appearance by equal distances in the space. The relative sensitivity to different axes within the space might therefore reflect the gamut of colors in natural scenes. To examine this, we projected perceptually uniform distributions within the Munsell, CIE L(*)u(*)v(*) or CIE L(*)a(*)b(*) spaces into cone-opponent space. All were elongated along a bluish-yellowish axis reflecting covarying signals along the L-M and S-(L+M) cardinal axes, a pattern typical (though not identical) to many natural environments. In turn, color distributions from environments were more uniform when projected into the CIE L(*)a(*)b(*) perceptual space than when represented in a normalized cone-opponent space. These analyses suggest the bluish-yellowish bias in environmental colors might be an important factor shaping chromatic sensitivity, and also suggest that perceptually uniform color metrics could be derived from natural scene statistics and potentially tailored to specific environments.

Color changes in objects in natural scenes as a function of observation distance and weather conditions

We have analyzed the changes in the color of objects in natural scenes due to atmospheric scattering according to changes in the distance of observation. Hook-shaped curves were found in the chromaticity diagram when the object moved from zero distance to long distances, where the object chromaticity coordinates approached the color coordinates of the horizon. This trend is the result of the combined effect of attenuation in the direct light arriving to the observer from the object and the airlight added during its trajectory. Atmospheric scattering leads to a fall in the object's visibility, which is measurable as a difference in color between the object and the background (taken here to be the horizon). Focusing on color difference instead of luminance difference could produce different visibility values depending on the color tolerance used. We assessed the cone-excitation ratio constancy for several objects at different distances. Affine relationships were obtained when an object's cone excitations were represented both at zero distance and increasing distances. These results could help to explain color constancy in natural scenes for objects at different distances, a phenomenon that has been pointed out by different authors.

Computational color constancy: survey and experiments

Computational color constancy is a fundamental prerequisite for many computer vision applications. This paper presents a survey of many recent developments and state-of-the-art methods. Several criteria are proposed that are used to assess the approaches. A taxonomy of existing algorithms is proposed and methods are separated in three groups: static methods, gamut-based methods, and learning-based methods. Further, the experimental setup is discussed including an overview of publicly available datasets. Finally, various freely available methods, of which some are considered to be state of the art, are evaluated on two datasets.

The effect of photopigment optical density on the color vision of the anomalous trichromat

We present a theoretical model to estimate the influence of photopigment optical density (OD) on the color vision of anomalous trichromats. Photopigment spectral sensitivities are generated using the Lamb (1995) template, which we correct for OD and pre-receptoral filters. Sixteen hyperspectral images (Foster, Nascimento, & Amano, 2004; Nascimento, Ferreira, & Foster, 2002) are analyzed, and the signals produced in the post-receptoral channels calculated. In the case of anomalous trichromats whose two longer-wavelength cones have peak sensitivities that lie close together in the spectrum, color vision can be substantially enhanced if the cones differ in optical density by a realistic amount.

Hyperspectral optical imaging of human iris in vivo: characteristics of reflectance spectra

We report a hyperspectral imaging system to measure the reflectance spectra of real human irises with high spatial resolution. A set of ocular prosthesis was used as the control condition. Reflectance data were decorrelated by the principal-component analysis. The main conclusion is that spectral complexity of the human iris is considerable: between 9 and 11 principal components are necessary to account for 99% of the cumulative variance in human irises. Correcting image misalignments associated with spontaneous ocular movements did not influence this result. The data also suggests a correlation between the first principal component and different levels of melanin present in the irises. It was also found that although the spectral characteristics of the first five principal components were not affected by the radial and angular position of the selected iridal areas, they affect the higher-order ones, suggesting a possible influence of the iris texture. The results show that hyperspectral imaging in the iris, together with adequate spectroscopic analyses provide more information than conventional colorimetric methods, making hyperspectral imaging suitable for the characterization of melanin and the noninvasive diagnosis of ocular diseases and iris color.

Hyperspectral imaging of cuttlefish camouflage indicates good color match in the eyes of fish predators

Camouflage is a widespread phenomenon throughout nature and an important antipredator tactic in natural selection. Many visual predators have keen color perception, and thus camouflage patterns should provide some degree of color matching in addition to other visual factors such as pattern, contrast, and texture. Quantifying camouflage effectiveness in the eyes of the predator is a challenge from the perspectives of both biology and optical imaging technology. Here we take advantage of hyperspectral imaging (HSI), which records full-spectrum light data, to simultaneously visualize color match and pattern match in the spectral and the spatial domains, respectively. Cuttlefish can dynamically camouflage themselves on any natural substrate and, despite their colorblindness, produce body patterns that appear to have high-fidelity color matches to the substrate when viewed directly by humans or with RGB images. Live camouflaged cuttlefish on natural backgrounds were imaged using HSI, and subsequent spectral analysis revealed that most reflectance spectra of individual cuttlefish and substrates were similar, rendering the color match possible. Modeling color vision of potential di- and trichromatic fish predators of cuttlefish corroborated the spectral match analysis and demonstrated that camouflaged cuttlefish show good color match as well as pattern match in the eyes of fish predators. These findings (i) indicate the strong potential of HSI technology to enhance studies of biological coloration and (ii) provide supporting evidence that cuttlefish can produce color-coordinated camouflage on natural substrates despite lacking color vision.

Number of discernible colors for color-deficient observers estimated from the MacAdam limits

We estimated the number of colors perceived by color normal and color-deficient observers when looking at the theoretic limits of object-color stimuli. These limits, the optimal color stimuli, were computed for a color normal observer and CIE standard illuminant D65, and the resultant colors were expressed in the CIELAB and DIN99d color spaces. The corresponding color volumes for abnormal color vision were computed using models simulating for normal trichromatic observers the appearance for dichromats and anomalous trichomats. The number of colors perceived in each case was then computed from the color volumes enclosed by the optimal colors also known as MacAdam limits. It was estimated that dichromats perceive less than 1% of the colors perceived by normal trichromats and that anomalous trichromats perceive 50%-60% for anomalies in the medium-wavelength-sensitive and 60%-70% for anomalies in the long-wavelength-sensitive cones. Complementary estimates obtained similarly for the spectral locus of monochromatic stimuli suggest less impairment for color-deficient observers, a fact that is explained by the two-dimensional nature of the locus.

Color constancy

A quarter of a century ago, the first systematic behavioral experiments were performed to clarify the nature of color constancy-the effect whereby the perceived color of a surface remains constant despite changes in the spectrum of the illumination. At about the same time, new models of color constancy appeared, along with physiological data on cortical mechanisms and photographic colorimetric measurements of natural scenes. Since then, as this review shows, there have been many advances. The theoretical requirements for constancy have been better delineated and the range of experimental techniques has been greatly expanded; novel invariant properties of images and a variety of neural mechanisms have been identified; and increasing recognition has been given to the relevance of natural surfaces and scenes as laboratory stimuli. Even so, there remain many theoretical and experimental challenges, not least to develop an account of color constancy that goes beyond deterministic and relatively simple laboratory stimuli and instead deals with the intrinsically variable nature of surfaces and illuminations present in the natural world.

Effect of Scene Dimensionality on Colour Constancy with Real Three-Dimensional Scenes and Objects

The effect of scene dimensionality on colour constancy was tested with real scenes and objects. Observers viewed a three-dimensional (3-D) scene, or its two-dimensional (2-D) planar projection, through a large beam-splitter that projected the virtual image of a real test object (a cube or its 2-D projection) so that it appeared part of the scene. Test object and scene could be illuminated independently with high chromatic precision. In each trial, the illuminance of the scene changed abruptly from 25 000 K to 6700 K and the illuminant of the test object changed either consistently or inconsistently with it by a variable quantifiable amount. Observers had to decide whether the test object underwent a change in its materials. The extent of constancy obtained in the experiment was not influenced by scene dimensionality and varied significantly with the colour of the test object. These results suggest that color constancy in the conditions tested here may be determined by local spectral quantities.

Color constancy of red-green dichromats and anomalous trichromats

Purpose. Color-vision deficiency is associated with abnormalities in color matching and color discrimination, but its impact on the ability of people to judge the constancy of surface colors under different lights (color constancy) is less clear. This work had two aims: first, to quantify the degree of color constancy in subjects with congenital red-green color deficiency; second, to test whether the degree of color constancy in anomalous trichromats can be predicted from their Rayleigh anomaloscope matches. Methods. Color constancy of red-green color-deficient subjects was tested in a task requiring the discrimination of illuminant changes from surface-reflectance changes. Mondrian-like colored patterns, generated on the screen of a computer monitor, were used as stimuli to avoid the spatial cues provided by natural objects and scenes. Spectral reflectances were taken from the Munsell Book of Color and from natural scenes. Illuminants were taken from the daylight locus. Results. Protanopes and deuteranopes performed more poorly than normal trichromats with Munsell spectral reflectances but were less impaired with natural spectral reflectances. Protanomalous and deuteranomalous trichromats performed as well as, or almost as well as, normal trichromats, independent of the type of reflectance. Individual differences were not correlated with Rayleigh anomaloscope matches. Conclusions. Despite the evidence of clinical color-vision tests, red-green color-deficient persons are less disadvantaged than might be expected in their judgments of surface colors under different lights.

Approaching ideal observer efficiency in using color to retrieve information from natural scenes

Variations in illumination on a scene and trichromatic sampling by the eye limit inferences about scene content. The aim of this work was to elucidate these limits in relation to an ideal observer using color signals alone. Simulations were based on 50 hyperspectral images of natural scenes and daylight illuminants with correlated color temperatures 4000 K, 6500 K, and 25,000 K. Estimates were made of the (Shannon) information available from each scene, the redundancies in receptoral and postreceptoral coding, and the information retrieved by an observer identifying corresponding points across image pairs. For the largest illuminant difference, between 25,000 K and 4000 K, a postreceptoral transformation providing minimum redundancy yielded an efficiency of about 80% in the information retrieved. This increased to about 89% when the transformation was optimized directly for information retrieved, corresponding to an equivalent Gaussian noise amplitude of 3.0% or to a mean of 3.6 x 10(4) distinct identifiable points per scene. Using color signals to retrieve information from natural scenes can approach ideal observer efficiency levels.

Visualization of the spatial and spectral signals of orb-weaving spiders, Nephila pilipes, through the eyes of a honeybee

It is well known that the honeybee has good color vision. However, the spectral range in which the bee can see is different from that of the human eye. To study how bees view their world of colors, one has to see through the eyes of the bee, not the eyes of a human. A conventional way to examine the color signals that animals can detect is to measure the surface reflectance spectra and compute the quantum catches of each photoreceptor type based on its known spectral sensitivity. Color signal and color contrast are then determined from the loci of these quantum catches in the color space. While the point-by-point measurements of the reflectance spectra using a standard spectrometer have yielded a significant amount of data for analyzing color signals, the lack of spatial information and low sampling efficiency constrain their applications. Using a special filter coating technique, a set of filters with transmission spectra that were closely matched to the bee's sensitivity spectra of three photoreceptor types (UV, blue, and green) was custom made. By placing these filters in front of a UV/VIS-sensitive CCD camera and acquiring images sequentially, we could collect images of a bee's receptor with only three shots. This allowed a direct visualization of how bees view their world in a pseudo-color RGB display. With this imaging system, spatial and spectral signals of the orb-weaving spider, Nephila pilipes, were recorded,and color contrast images corresponding to the bee's spatial resolution were constructed and analyzed. The result not only confirmed that the color markings of N. pilipes are of high chromatic contrast to the eyes of a bee, but it also indicated that the spatial arrangement of these markings resemble flower patterns which may attract bees to visit them. Thus, it is likely that the orb-weaving spider (N. pilipes) deploys a similar strategy to that of the Australian crab spider (Thomisus spectabilis)to exploit the bee's pre-existing preference for flowers with color patterning.

Trichromatic red-green-blue camera used for the recovery of albedo and reflectance of rough-textured surfaces under different illumination conditions

Photometric-stereo techniques are based on the fact that image intensity depends upon the orientation of the surface with regard to the source of the illumination and its spectral reflectance. They are of special interest when dealing with rough surfaces because they usually present shadowed regions where sudden illumination changes might be found. In the present work we introduce an extension of the four-source photometric-stereo algorithm to color images that is able to recover the surface spectral reflectance of objects captured with a red-green-blue (RGB) camera. This method allows image rendering, even for rough-textured surfaces, under different directions of the impinging illumination. In addition, the introduction of spectral recovery techniques applied to the albedo and spectral reflectance from rough surfaces offers the possibility of image rendering for scenes captured under sources of illumination differing in spectral distribution. Using albedo instead of RGB information helps to avoid any shadows or highlights that might falsify results. One of the advantages of this spectral-based photometric-stereo method is that it can recover not only the albedo values, but also the spectral reflectance spectrum of an object's surface on a pixel-by-pixel basis, as can be done with more complex hyperspectral imaging devices involving a camera coupled to an extensive set of narrowband filters.

Using a trichromatic CCD camera for spectral skylight estimation

In a previous work [J. Opt. Soc. Am. A 24, 942-956 (2007)] we showed how to design an optimum multispectral system aimed at spectral recovery of skylight. Since high-resolution multispectral images of skylight could be interesting for many scientific disciplines, here we also propose a nonoptimum but much cheaper and faster approach to achieve this goal by using a trichromatic RGB charge-coupled device (CCD) digital camera. The camera is attached to a fish-eye lens, hence permitting us to obtain a spectrum of every point of the skydome corresponding to each pixel of the image. In this work we show how to apply multispectral techniques to the sensors' responses of a common trichromatic camera in order to obtain skylight spectra from them. This spectral information is accurate enough to estimate experimental values of some climate parameters or to be used in algorithms for automatic cloud detection, among many other possible scientific applications.

The number of discernible colors in natural scenes

The number of colors discernible by normal trichromats has been estimated for the idealized object-color solid. How well these estimates apply to natural scenes is an open question, as it is unknown how much their colors approach the theoretical limits. The aim of this work was to estimate the number of discernible colors based on a database of hyperspectral images of 50 natural scenes. The color volume of each scene was computed in the CIELAB color space and was analyzed using the CIEDE2000 color-difference formula. It was found that the color volume of the set of natural scenes was about 30% of the theoretical maximum for the full object-color solid, and it corresponded to a number of about 2.3 million discernible colors. Moreover, when the lightness dimension was ignored, only about 26,000 (1%) could be perceived as different colors. These results suggest that natural stimuli may be more constrained than expected from the analysis of the theoretical limits.

The number of discernible colors perceived by dichromats in natural scenes and the effects of colored lenses

The number of discernible colors perceived by normal trichromats when viewing natural scenes can be estimated by analyzing idealized color volumes or hyperspectral data obtained from actual scenes. The purpose of the present work was to estimate the relative impairment in chromatic diversity experienced by dichromats when viewing natural scenes and to investigate the effects of colored lenses. The estimates were obtained computationally from the analysis of hyperspectral images of natural scenes and using a quantitative model of dichromats' vision. The color volume corresponding to each scene was represented in CIELAB color space and segmented into cubes of unitary side. For normal trichromats, the number of discernible colors was estimated by counting the number of non-empty cubes. For dichromats, an algorithm simulating for normal observers the appearance of the scenes for dichromats was used, and the number of discernible colors was then counted as for normal trichromats. The effects of colored lenses were estimated by prior filtering the spectral radiance from the scenes with the spectral transmittance function of the lenses. It was found that in dichromatic vision the number of discernible colors was about 7% of normal trichromatic vision. With some colored lenses considerable improvements in chromatic diversity were obtained for trichromats; for dichromats, however, only modest improvements could be obtained with efficiency levels dependent on the combination of scene, lens and type of deficiency.

Calculation of retinal image quality for polychromatic light

Although the retinal image is typically polychromatic, few studies have examined polychromatic image quality in the human eye. We begin with a conceptual framework including the formulation of a psychophysical linking hypothesis that underlies the utility of image quality metrics based on the polychromatic point-spread function. We then outline strategies for computing polychromatic point-spread functions of the eye when monochromatic aberrations are known for only a single wavelength. Implementation problems and solutions for this strategy are described. Polychromatic image quality is largely unaffected by wavelength-dependent diffraction and higher-order chromatic aberration. However, accuracy is found to depend critically upon spectral sampling. Using typical aberrations from the Indiana Aberration Study, we assessed through-focus image quality for model eyes with and without chromatic aberrations using a polychromatic metric called the visual Strehl ratio. In the presence of typical levels of monochromatic aberrations, the effect of longitudinal chromatic aberration is greatly reduced. The effect of typical levels of transverse chromatic aberration is virtually eliminated in the presence of longitudinal chromatic aberration and monochromatic aberrations. Clinical value and limitations of the method are discussed.

Unsupervised illuminant estimation from natural scenes: an RGB digital camera suffices

A linear pseudo-inverse method for unsupervised illuminant recovery from natural scenes is presented. The algorithm, which uses a digital RGB camera, selects the naturally occurring bright areas (not necessarily the white ones) in natural images and converts the RGB digital counts directly into the spectral power distribution of the illuminants using a learning-based spectral procedure. Computations show a good spectral and colorimetric performance when only three sensors (a three-band RGB camera) are used. These results go against previous findings concerning the recovery of spectral reflectances and radiances, which claimed that the greater the number of sensors, the better the spectral performance. Combining the device with the appropriate computations can yield spectral information about objects and illuminants simultaneously, avoiding the need for spectroradiometric measurements. The method works well and needs neither a white reference located in the natural scene nor direct measurements of the spectral power distribution of the light.

Lie methods for color robot vision

We describe how Lie-theoretical methods can be used to analyze color related problems in machine vision. The basic observation is that the nonnegative nature of spectral color signals restricts these functions to be members of a limited, conical section of the larger Hilbert space of square-integrable functions. From this observation, we conclude that the space of color signals can be equipped with a coordinate system consisting of a half-axis and a unit ball with the Lorentz groups as natural transformation group. We introduce the theory of the Lorentz group SU(1, 1) as a natural tool for analyzing color image processing problems and derive some descriptions and algorithms that are useful in the investigation of dynamical color changes. We illustrate the usage of these results by describing how to compress, interpolate, extrapolate, and compensate image sequences generated by dynamical color changes.

Trichromatic reconstruction from the interleaved cone mosaic: Bayesian model and the color appearance of small spots

Observers use a wide range of color names, including white, to describe monochromatic flashes with a retinal size comparable to that of a single cone. We model such data as a consequence of information loss arising from trichromatic sampling. The model starts with the simulated responses of the individual L, M, and S cones actually present in the cone mosaic and uses these to reconstruct the L-, M-, and S-cone signals that were present at every image location. We incorporate the optics and the mosaic topography of individual observers, as well as the spatio-chromatic statistics of natural images. We simulated the experiment of H. Hofer, B. Singer, & D. R. Williams (2005) and predicted the color name on each simulated trial from the average chromaticity of the spot reconstructed by our model. Broad features of the data across observers emerged naturally as a consequence of the measured individual variation in the relative numbers of L, M, and S cones. The model's output is also consistent with the appearance of larger spots and of sinusoidal contrast modulations. Finally, the model makes testable predictions for future experiments that study how color naming varies with the fine structure of the retinal mosaic.

Color constancy: phenomenal or projective?

Naive observers viewed a sequence of colored Mondrian patterns, simulated on a color monitor. Each pattern was presented twice in succession, first under one daylight illuminant with a correlated color temperature of either 16,000 or 4000 K and then under the other, to test for color constancy. The observers compared the central square of the pattern across illuminants, either rating it for sameness of material appearance or sameness of hue and saturation or judging an objective property-that is, whether its change of color originated from a change in material or only from a change in illumination. Average color constancy indices were high for material appearance ratings and binary judgments of origin and low for hue-saturation ratings. Individuals' performance varied, but judgments of material and of hue and saturation remained demarcated. Observers seem able to separate phenomenal percepts from their ontological projections of mental appearance onto physical phenomena; thus, even when a chromatic change alters perceived hue and saturation, observers can reliably infer the cause, the constancy of the underlying surface spectral reflectance.

Seasonal variations in the color statistics of natural images

We examined how the distribution of colors in natural images varies as the seasons change. Images of natural outdoor scenes were acquired at locations in the Western Ghats, India, during monsoon and winter seasons and in the Sierra Nevada, USA, from spring to fall. The images were recorded with an RGB digital camera calibrated to yield estimates of the L, M, and S cone excitations and chromatic and luminance contrasts at each pixel. These were compared across time and location and were analyzed separately for regions of earth and sky. Seasonal climate changes alter both the average color in scenes and how the colors are distributed around the average. Arid periods are marked by a mean shift toward the +L pole of the L vs. M chromatic axis and a rotation in the color distributions away from the S vs. LM chromatic axis and toward an axis of bluish-yellowish variation, both primarily due to changes in vegetation. The form of the change was similar at the two locations suggesting that the color statistics of natural images undergo a characteristic pattern of temporal variation. We consider the implications of these changes for models of both visual sensitivity and color appearance.

Color naming, unique hues, and hue cancellation predicted from singularities in reflection properties

Psychophysical studies suggest that different colors have different perceptual status: red and blue for example are thought of as elementary sensations whereas yellowish green is not. The dominant account for such perceptual asymmetries attributes them to specificities of the neuronal representation of colors. Alternative accounts involve cultural or linguistic arguments. What these accounts have in common is the idea that there are no asymmetries in the physics of light and surfaces that could underlie the perceptual structure of colors, and this is why neuronal or cultural processes must be invoked as the essential underlying mechanisms that structure color perception. Here, we suggest a biological approach for surface reflection properties that takes into account only the information about light that is accessible to an organism given the photopigments it possesses, and we show that now asymmetries appear in the behavior of surfaces with respect to light. These asymmetries provide a classification of surface properties that turns out to be identical to the one observed in linguistic color categorization across numerous cultures, as pinned down by cross cultural studies. Further, we show that data from psychophysical studies about unique hues and hue cancellation are consistent with the viewpoint that stimuli reported by observers as special are those associated with this singularity-based categorization of surfaces under a standard illuminant. The approach predicts that unique blue and unique yellow should be aligned in chromatic space while unique red and unique green should not, a fact usually conjectured to result from nonlinearities in chromatic pathways.

Color constancy in natural scenes explained by global image statistics

To what extent do observers' judgments of surface color with natural scenes depend on global image statistics? To address this question, a psychophysical experiment was performed in which images of natural scenes under two successive daylights were presented on a computer-controlled high-resolution color monitor. Observers reported whether there was a change in reflectance of a test surface in the scene. The scenes were obtained with a hyperspectral imaging system and included variously trees, shrubs, grasses, ferns, flowers, rocks, and buildings. Discrimination performance, quantified on a scale of 0 to 1 with a color-constancy index, varied from 0.69 to 0.97 over 21 scenes and two illuminant changes, from a correlated color temperature of 25,000 K to 6700 K and from 4000 K to 6700 K. The best account of these effects was provided by receptor-based rather than colorimetric properties of the images. Thus, in a linear regression, 43% of the variance in constancy index was explained by the log of the mean relative deviation in spatial cone-excitation ratios evaluated globally across the two images of a scene. A further 20% was explained by including the mean chroma of the first image and its difference from that of the second image and a further 7% by the mean difference in hue. Together, all four global color properties accounted for 70% of the variance and provided a good fit to the effects of scene and of illuminant change on color constancy, and, additionally, of changing test-surface position. By contrast, a spatial-frequency analysis of the images showed that the gradient of the luminance amplitude spectrum accounted for only 5% of the variance.

Anomalous trichromats' judgments of surface color in natural scenes under different daylights

Deuteranomalous trichromacy, which affects medium-wavelength-sensitive cones, is more common than protanomalous trichromacy, which affects long-wavelength-sensitive cones. The aim of the present work was to test the extent to which these two kinds of anomalous trichromacy affect surface-color judgments in the natural world. Simulations of 18 natural scenes under different daylight illuminants were presented on a high-resolution color monitor to 7 deuteranomalous, 7 protanomalous, and 12 normal trichromatic observers, who had to discriminate between reflectance and illuminant changes in the images. Observers' ability to judge surface color was quantified by a standard color-constancy index. Deuteranomalous trichromats performed as well as normal trichromats, but protanomalous trichromats performed more poorly than both. The results are considered in relation to the spectral coverage of cones, rod intrusion, and the characterization of anomalous trichromacy by the Rayleigh match.

Frequency of metamerism in natural scenes

Estimates of the frequency of metameric surfaces, which appear the same to the eye under one illuminant but different under another, were obtained from 50 hyperspectral images of natural scenes. The degree of metamerism was specified with respect to a color-difference measure after allowing for full chromatic adaptation. The relative frequency of metameric pairs of surfaces, expressed as a proportion of all pairs of surfaces in a scene, was very low. Depending on the criterion degree of metamerism, it ranged from about 10(-6) to 10(-4) for the largest illuminant change tested, which was from a daylight of correlated color temperature 25,000 K to one of 4000 K. But, given pairs of surfaces that were indistinguishable under one of these illuminants, the conditional relative frequency of metamerism was much higher, from about 10(-2) to 10(-1), sufficiently large to affect visual inferences about material identity.

Visual sensitivity to color errors in images of natural scenes

Simple color-difference formulae and pictorial images have traditionally been used to estimate the visual impact of color errors introduced by image-reproduction processes. But the limited gamut of RGB cameras constrains such analyses, particularly of natural scenes. The purpose of this work was to estimate visual sensitivity to color errors introduced deliberately into pictures synthesized from hyperspectral images of natural scenes without gamut constraints and to compare discrimination thresholds expressed in CIELAB and S-CIELAB color spaces. From each original image, a set of approximate images with variable color errors were generated and displayed on a calibrated RGB color monitor. The threshold for perceptibility of the errors was determined in a paired-comparison experiment. In agreement with previous studies, it was found that discrimination between original and approximate images needed on average a CIELAB color difference ΔEab* of about 2.2. Although a large variation of performance across the nine images tested was found when errors were expressed in CIELAB units, little variation was obtained when they were expressed in S-CIELAB units.

Color constancy in natural scenes with and without an explicit illuminant cue

Observers can generally make reliable judgments of surface color in natural scenes despite changes in an illuminant that is out of view. This ability has sometimes been attributed to observers' estimating the spectral properties of the illuminant in order to compensate for its effects. To test this hypothesis, two surface-color-matching experiments were performed with images of natural scenes obtained from high-resolution hyperspectral images. In the first experiment, the sky illuminating the scene was directly visible to the observer, and its color was manipulated. In the second experiment, a large gray sphere was introduced into the scene so that its illumination by the sun and sky was also directly visible to the observer, and the color of that illumination was manipulated. Although the degree of color constancy varied across this and other variations of the images, there was no reliable effect of illuminant color. Even when the sky was eliminated from view, color constancy did not worsen. Judging surface color in natural scenes seems to be independent of an explicit illuminant cue.

Minimalist surface-colour matching

Some theories of surface-colour perception assume that observers estimate the illuminant on a scene so that its effects can be discounted. A critical test of this interpretation of colour constancy is whether surface-colour matching is worse when the number of surfaces in a scene is so small that any illuminant estimate is unreliable. In the experiment reported here, observers made asymmetric colour matches between pairs of simultaneously presented Mondrian-like patterns under different daylights. The patterns had either 49 surfaces or a minimal 2 surfaces. No significant effect of number was found, suggesting that illuminant estimates are unnecessary for surface-colour matching.

Information limits on identification of natural surfaces by apparent colour

By adaptational and other mechanisms, the visual system can compensate for moderate changes in the colour of the illumination on a scene. Although the colours of most surfaces are perceived to be constant ('colour constancy'), some are not. The effect of these residual colour changes on the ability of observers to identify surfaces by their apparent colour was determined theoretically from high-resolution hyperspectral images of natural scenes under different daylights with correlated colour temperatures 4,300 K, 6,500 K, and 25,000 K. Perceived differences between colours were estimated with an approximately uniform colour-distance measure. The information preserved under illuminant changes increased with the number of surfaces in the sample, but was limited to a relatively low asymptotic value, indicating the importance of physical factors in constraining identification by apparent colour.