Time-lapse ratios of cone excitations in natural scenes

Colour expectations across illumination changes

This study investigates human expectations towards naturalistic colour changes under varying illuminations. Understanding colour expectations is key to both scientific research on colour constancy and applications of colour and lighting in art and industry. We reanalysed data from asymmetric colour matches of a previous study and found that colour adjustments tended to align with illuminant-induced colour shifts predicted by naturalistic, rather than artificial, illuminants and reflectances. We conducted three experiments using hyperspectral images of naturalistic scenes to test if participants judged colour changes based on naturalistic illuminant and reflectance spectra as more plausible than artificial ones, which contradicted their expectations. When we consistently manipulated the illuminant (Experiment 1) and reflectance (Experiment 2) spectra across the whole scene, observers chose the naturalistic renderings significantly above the chance level (>25 %) but barely more often than any of the three artificial ones, collectively (>50 %). However, when we manipulated only one object/area's reflectance (Experiment 3), observers more reliably identified the version in which the object had a naturalistic reflectance like the rest of the scene. Results from Experiments 2-3 and additional analyses suggested that relational colour constancy strongly contributed to observer expectations, and stable cone-excitation ratios are not limited to naturalistic illuminants and reflectances but also occur for our artificial renderings. Our findings indicate that relational colour constancy and prior knowledge about surface colour shifts help to disambiguate surface colour identity under illumination changes, enabling human observers to recognise surface colours reliably in naturalistic conditions. Additionally, relational colour constancy may even be effective in many artificial conditions.

Misidentifying illuminant changes in natural scenes due to failures in relational colour constancy

The colours of surfaces in a scene may not appear constant with a change in the colour of the illumination. Yet even when colour constancy fails, human observers can usually discriminate changes in lighting from changes in surface reflecting properties. This operational ability has been attributed to the constancy of perceived colour relations between surfaces under illuminant changes, in turn based on approximately invariant spatial ratios of cone photoreceptor excitations. Natural deviations in these ratios may, however, lead to illuminant changes being misidentified. The aim of this work was to test whether such misidentifications occur with natural scenes and whether they are due to failures in relational colour constancy. Pairs of scene images from hyperspectral data were presented side-by-side on a computer-controlled display. On one side, the scene underwent illuminant changes and on the other side, it underwent the same changes but with images corrected for any residual deviations in spatial ratios. Observers systematically misidentified the corrected images as being due to illuminant changes. The frequency of errors increased with the size of the deviations, which were closely correlated with the estimated failures in relational colour constancy.

Sensory representation of surface reflectances: assessments with hyperspectral images

Specifying surface reflectances in a simple and perceptually informative way would be beneficial for many areas of research and application. We assessed whether a 3×3 matrix may be used to approximate how a surface reflectance modulates the sensory color signal across illuminants. We tested whether observers could discriminate between the model's approximate and accurate spectral renderings of hyperspectral images under narrowband and naturalistic, broadband illuminants for eight hue directions. Discriminating the approximate from the spectral rendering was possible with narrowband, but almost never with broadband illuminants. These results suggest that our model specifies the sensory information of reflectances across naturalistic illuminants with high fidelity, and with lower computational cost than spectral rendering.

Quantifying the spatial, temporal, angular and spectral structure of effective daylight in perceptually meaningful ways

We present a method to capture the 7-dimensional light field structure, and translate it into perceptually-relevant information. Our spectral cubic illumination method quantifies objective correlates of perceptually relevant diffuse and directed light components, including their variations over time, space, in color and direction, and the environment's response to sky and sunlight. We applied it "in the wild", capturing how light on a sunny day differs between light and shadow, and how light varies over sunny and cloudy days. We discuss the added value of our method for capturing nuanced lighting effects on scene and object appearance, such as chromatic gradients.

Efficient quantization of painting images by relevant colors

Realistic images often contain complex variations in color, which can make economical descriptions difficult. Yet human observers can readily reduce the number of colors in paintings to a small proportion they judge as relevant. These relevant colors provide a way to simplify images by effectively quantizing them. The aim here was to estimate the information captured by this process and to compare it with algorithmic estimates of the maximum information possible by colorimetric and general optimization methods. The images tested were of 20 conventionally representational paintings. Information was quantified by Shannon's mutual information. It was found that the estimated mutual information in observers' choices reached about 90% of the algorithmic maxima. For comparison, JPEG compression delivered somewhat less. Observers seem to be efficient at effectively quantizing colored images, an ability that may have applications in the real world.

Colour constancy failures expected in colourful environments

Colour constancy refers to the constant perceived or apparent colour of a surface despite changes in illumination spectrum. Laboratory measurements have often found it imperfect. The aim here was to estimate the frequency of constancy failures in natural outdoor environments and relate them to colorimetric surface properties. A computational analysis was performed with 50 hyperspectral reflectance images of outdoor scenes undergoing simulated daylight changes. For a chromatically adapted observer, estimated colour appearance changed noticeably for at least 5% of the surface area in 60% of scenes, and at least 10% of the surface area in 44% of scenes. Somewhat higher frequencies were found for estimated changes in perceived colour relations represented by spatial ratios of cone-photoreceptor excitations. These estimated changes correlated with surface chroma and saturation. Outdoors, the colour constancy of some individual surfaces seems likely to fail, particularly if those surfaces are colourful.

The Concept of Symmetry and the Theory of Perception

Perceptual constancy refers to the fact that the perceived geometrical and physical characteristics of objects remain constant despite transformations of the objects such as rigid motion. Perceptual constancy is essential in everything we do, like recognition of familiar objects and scenes, planning and executing visual navigation, visuomotor coordination, and many more. Perceptual constancy would not exist without the geometrical and physical permanence of objects: their shape, size, and weight. Formally, perceptual constancy and permanence of objects are invariants, also known in mathematics and physics as symmetries. Symmetries of the Laws of Physics received a central status due to mathematical theorems of Emmy Noether formulated and proved over 100 years ago. These theorems connected symmetries of the physical laws to conservation laws through the least-action principle. We show how Noether's theorem is applied to mirror-symmetrical objects and establishes mental shape representation (perceptual conservation) through the application of a simplicity (least-action) principle. This way, the formalism of Noether's theorem provides a computational explanation of the relation between the physical world and its mental representation.

The color appearance of curved transparent objects

Studies on colored transparent objects have elucidated potential mechanisms, but these studies have mainly focused on flat filters overlaying flat backgrounds. While they have provided valuable insight, these studies have not captured all aspects of transparency, like caustics, specular reflections/highlights, and shadows. Here, we investigate color-matching experiments with curved transparent objects for different matching stimuli: a uniform patch and a flat filter. Two instructions were tested: simply match the color of the glass object and the test element (patch and flat filter) or match the color of the dye that was used to tint the transparent object (patch). Observers’ matches differed from the mean, the most frequent, and the most saturated color of the transparent stimuli, whereas the brightest regions captured the chromaticity, but not the lightness, of patch matches. We applied four models from flat filter studies: the convergence model, the ratios of either the means (RMC) or standard deviations (RSD) of cone excitations, and a robust ratio model. The original convergence model does not fully generalize but does not perform poorly, and with modifications, we find that curved transparent objects cause a convergence of filtered colors toward a point in color space, similar to flat filters. Considering that, the RMC and robust ratio models generalized more than the RSD, with the RMC performing best across the stimuli we tested. We conclude that the RMC is probably the strongest factor for determining the color. The RSD seems instead to be related to the perceived “clarity” of glass objects.

Fluctuating environmental light limits number of surfaces visually recognizable by colour

Small changes in daylight in the environment can produce large changes in reflected light, even over short intervals of time. Do these changes limit the visual recognition of surfaces by their colour? To address this question, information-theoretic methods were used to estimate computationally the maximum number of surfaces in a sample that can be identified as the same after an interval. Scene data were taken from successive hyperspectral radiance images. With no illumination change, the average number of surfaces distinguishable by colour was of the order of 10,000. But with an illumination change, the average number still identifiable declined rapidly with change duration. In one condition, the number after two minutes was around 600, after 10 min around 200, and after an hour around 70. These limits on identification are much lower than with spectral changes in daylight. No recoding of the colour signal is likely to recover surface identity lost in this uncertain environment.

Spatio-chromatic information available from different neural layers via Gaussianization

How much visual information about the retinal images can be extracted from the different layers of the visual pathway? This question depends on the complexity of the visual input, the set of transforms applied to this multivariate input, and the noise of the sensors in the considered layer. Separate subsystems (e.g. opponent channels, spatial filters, nonlinearities of the texture sensors) have been suggested to be organized for optimal information transmission. However, the efficiency of these different layers has not been measured when they operate together on colorimetrically calibrated natural images and using multivariate information-theoretic units over the joint spatio-chromatic array of responses.In this work, we present a statistical tool to address this question in an appropriate (multivariate) way. Specifically, we propose an empirical estimate of the information transmitted by the system based on a recent Gaussianization technique. The total correlation measured using the proposed estimator is consistent with predictions based on the analytical Jacobian of a standard spatio-chromatic model of the retina-cortex pathway. If the noise at certain representation is proportional to the dynamic range of the response, and one assumes sensors of equivalent noise level, then transmitted information shows the following trends: (1) progressively deeper representations are better in terms of the amount of captured information, (2) the transmitted information up to the cortical representation follows the probability of natural scenes over the chromatic and achromatic dimensions of the stimulus space, (3) the contribution of spatial transforms to capture visual information is substantially greater than the contribution of chromatic transforms, and (4) nonlinearities of the responses contribute substantially to the transmitted information but less than the linear transforms.

Appearance of Maxwell's spot in images rendered using a cyan primary

The discovery of melanopsin as a third type of retinal photoreceptor, contributing to both perceptual vision and reflex light responses, represents a new opportunity to optimise the design of artificial light sources for practical applications and to generate experimental stimuli. In the case of emissive displays, multiprimary designs incorporating a cyan primary could be used to allow melanopic radiance to be controlled independent of colour and luminance. Here we explore the performance a five-primary (violet, cyan, green, yellow, red) display device and find an anomaly in colour appearance when the cyan primary is employed. The anomaly took the form of a reddish/pinkish tinge in the central visual field, consistent with descriptions of Maxwell’s spot. This effect was apparent in some full colour images and in uniform discs over a range of chromaticities. Its appearance in coloured discs correlated with differences in calculated colour coordinate between central and peripheral vision. A simulation indicated that inclusion of any primary with predominant output in the 470–500 nm range has the potential to produce such a discrepancy in central vs peripheral appearance. Applying an additional constraint in colour processing to reproduce naturally occurring differences in central vs peripheral colour coordinate eliminated appearance of the spot and produced acceptable colour images.

Form vision from melanopsin in humans

Detection and discrimination of spatial patterns is thought to originate with photoreception by rods and cones. Here, we investigated whether the inner-retinal photoreceptor melanopsin could represent a third origin for form vision. We developed a 4-primary visual display capable of presenting patterns differing in contrast for melanopsin vs cones, and generated spectrally distinct stimuli that were indistinguishable for cones (metamers) but presented contrast for melanopsin. Healthy observers could detect sinusoidal gratings formed by these metamers when presented in the peripheral retina at low spatial (≤0.8 cpd) and temporal (≤0.45 Hz) frequencies, and Michelson contrasts ≥14% for melanopsin. Metameric gratings became invisible at lower light levels (<10¹³ melanopsin photons cm⁻² sr⁻¹ s⁻¹) when rods are more active. The addition of metameric increases in melanopsin contrast altered appearance of greyscale representations of coarse gratings and a range of everyday images. These data identify melanopsin as a new potential origin for aspects of spatial vision in humans.

The perception of spatial patterns (form vision) is thought to rely on rod and cone cells in the retina. Here, the authors show that a third kind of retinal cell, melanopsin-expressing ganglion cells, can also detect form in humans, under particular conditions.

Hyperspectral imaging in color vision research: tutorial

This tutorial offers an introduction to terrestrial and close-range hyperspectral imaging and some of its uses in human color vision research. The main types of hyperspectral cameras are described together with procedures for image acquisition, postprocessing, and calibration for either radiance or reflectance data. Image transformations are defined for colorimetric representations, color rendering, and cone receptor and postreceptor coding. Several example applications are also presented. These include calculating the color properties of scenes, such as gamut volume and metamerism, and analyzing the utility of color in observer tasks, such as identifying surfaces under illuminant changes. The effects of noise and uncertainty are considered in both image acquisition and color vision applications.

Disentangling simultaneous changes of surface and illumination

Retinally incident light is an ambiguous product of spectral distributions of light in the environment and their interactions with reflecting, absorbing, and transmitting materials. An ideal color constant observer would unravel these confounded sources of information and account for changes in each factor. Scene statistics have been proposed as a way to compensate for changes in the illumination, but few theories consider changes of 3-dimensional surfaces. Here, we investigated the visual system's capacity to deal with simultaneous changes in illumination and surfaces. Spheres were imaged with a hyperspectral camera in a white box and their colors, as well as that of the illumination were varied along "red-green" and "blue-yellow" axes. Both the original hyperspectral images and replica scenes rendered with Mitsuba were used as stimuli, including rendered scenes with Glavens (Acta Psychologica, 2009, 132, 259-266). Observers viewed sequential, random pairs of our images, with either the whole scene, only the object, or only a part of the background being present. They judged how much the illuminant and object color changed on a scale of 0-100%. Observers could extract simultaneous illumination and reflectance changes when provided with a view of the whole scene, but global scene statistics did not fully account for their behavior, while local scene statistics improved the situation. There was no effect of color axis, shape, or simulated vs. original hyperspectral images. Observers appear to be making use of various sources of local information to complete the task.

Efficient Lossless Compression of Multitemporal Hyperspectral Image Data

Hyperspectral imaging (HSI) technology has been used for various remote sensing applications due to its excellent capability of monitoring regions-of-interest over a period of time. However, the large data volume of four-dimensional multitemporal hyperspectral imagery demands massive data compression techniques. While conventional 3D hyperspectral data compression methods exploit only spatial and spectral correlations, we propose a simple yet effective predictive lossless compression algorithm that can achieve significant gains on compression efficiency, by also taking into account temporal correlations inherent in the multitemporal data. We present an information theoretic analysis to estimate potential compression performance gain with varying configurations of context vectors. Extensive simulation results demonstrate the effectiveness of the proposed algorithm. We also provide in-depth discussions on how to construct the context vectors in the prediction model for both multitemporal HSI and conventional 3D HSI data.

HyTexiLa: High Resolution Visible and Near Infrared Hyperspectral Texture Images

We present a dataset of close range hyperspectral images of materials that span the visible and near infrared spectrums: HyTexiLa (Hyperspectral Texture images acquired in Laboratory). The data is intended to provide high spectral and spatial resolution reflectance images of 112 materials to study spatial and spectral textures. In this paper we discuss the calibration of the data and the method for addressing the distortions during image acquisition. We provide a spectral analysis based on non-negative matrix factorization to quantify the spectral complexity of the samples and extend local binary pattern operators to the hyperspectral texture analysis. The results demonstrate that although the spectral complexity of each of the textures is generally low, increasing the number of bands permits better texture classification, with the opponent band local binary pattern feature giving the best performance.

The Verriest Lecture: Color vision in an uncertain world

The natural world is optically unconstrained. Surface properties may vary from one point to another, and reflected light may vary from one instant to the next. The aim of this work is to quantify some of the physical failures of color vision performance that result from uncertainty. In computational simulations with images of vegetated and nonvegetated outdoor scenes, it is shown that color provides an unreliable guide to surface identity. It is also shown that changes in illuminant may cause colors to no longer match and the relations between individual colors to vary. These failures are generally well described by a measure of the randomness of the colors in scenes, the Shannon entropy. Although uncertainty is intrinsic to the environment, its consequences for color vision can be predicted.

A Physics-Based Deep Learning Approach to Shadow Invariant Representations of Hyperspectral Images

This paper proposes the Relit Spectral Angle-Stacked Autoencoder, a novel unsupervised feature learning approach for mapping pixel reflectances to illumination invariant encodings. This work extends the Spectral Angle-Stacked Autoencoder so that it can learn a shadow-invariant mapping. The method is inspired by a deep learning technique, Denoising Autoencoders, with the incorporation of a physics-based model for illumination such that the algorithm learns a shadow invariant mapping without the need for any labelled training data, additional sensors, a priori knowledge of the scene or the assumption of Planckian illumination. The method is evaluated using datasets captured from several different cameras, with experiments to demonstrate the illumination invariance of the features and how they can be used practically to improve the performance of high-level perception algorithms that operate on images acquired outdoors.

Motion of glossy objects does not promote separation of lighting and surface colour

The surface properties of an object, such as texture, glossiness or colour, provide important cues to its identity. However, the actual visual stimulus received by the eye is determined by both the properties of the object and the illumination. We tested whether operational colour constancy for glossy objects (the ability to distinguish changes in spectral reflectance of the object, from changes in the spectrum of the illumination) was affected by rotational motion of either the object or the light source. The different chromatic and geometric properties of the specular and diffuse reflections provide the basis for this discrimination, and we systematically varied specularity to control the available information. Observers viewed animations of isolated objects undergoing either lighting or surface-based spectral transformations accompanied by motion. By varying the axis of rotation, and surface patterning or geometry, we manipulated: (i) motion-related information about the scene, (ii) relative motion between the surface patterning and the specular reflection of the lighting, and (iii) image disruption caused by this motion. Despite large individual differences in performance with static stimuli, motion manipulations neither improved nor degraded performance. As motion significantly disrupts frame-by-frame low-level image statistics, we infer that operational constancy depends on a high-level scene interpretation, which is maintained in all conditions.