Statistics of spatial cone-excitation ratios in natural scenes

Colour expectations across illumination changes

This study investigates human expectations towards naturalistic colour changes under varying illuminations. Understanding colour expectations is key to both scientific research on colour constancy and applications of colour and lighting in art and industry. We reanalysed data from asymmetric colour matches of a previous study and found that colour adjustments tended to align with illuminant-induced colour shifts predicted by naturalistic, rather than artificial, illuminants and reflectances. We conducted three experiments using hyperspectral images of naturalistic scenes to test if participants judged colour changes based on naturalistic illuminant and reflectance spectra as more plausible than artificial ones, which contradicted their expectations. When we consistently manipulated the illuminant (Experiment 1) and reflectance (Experiment 2) spectra across the whole scene, observers chose the naturalistic renderings significantly above the chance level (>25 %) but barely more often than any of the three artificial ones, collectively (>50 %). However, when we manipulated only one object/area's reflectance (Experiment 3), observers more reliably identified the version in which the object had a naturalistic reflectance like the rest of the scene. Results from Experiments 2-3 and additional analyses suggested that relational colour constancy strongly contributed to observer expectations, and stable cone-excitation ratios are not limited to naturalistic illuminants and reflectances but also occur for our artificial renderings. Our findings indicate that relational colour constancy and prior knowledge about surface colour shifts help to disambiguate surface colour identity under illumination changes, enabling human observers to recognise surface colours reliably in naturalistic conditions. Additionally, relational colour constancy may even be effective in many artificial conditions.

A Bayesian observer model reveals a prior for natural daylights in hue perception

Incorporating statistical characteristics of stimuli in perceptual processing can be highly beneficial for reliable estimation from noisy sensory measurements but may generate perceptual bias. According to Bayesian inference, perceptual biases arise from the integration of internal priors with noisy sensory inputs. In this study, we used a Bayesian observer model to derive biases and priors in hue perception based on discrimination data for hue ensembles with varying levels of chromatic noise. Our results showed that discrimination thresholds for isoluminant stimuli with hue defined by azimuth angle in cone-opponent color space exhibited a bimodal pattern, with lowest thresholds near a non-cardinal blue-yellow axis that aligns closely with the variation of natural daylights. Perceptual biases showed zero crossings around this axis, indicating repulsion away from yellow and attraction towards blue. These biases could be explained by the Bayesian observer model through a non-uniform prior with a preference for blue. Our findings suggest that visual processing takes advantage of knowledge of the distribution of colors in natural environments for hue perception.

Colorimetric Evaluation of a Reintegration via Spectral Imaging-Case Study: Nasrid Tiling Panel from the Alhambra of Granada (Spain)

Color reintegration is a restoration treatment that involves applying paint or colored plaster to an object of cultural heritage to facilitate its perception and understanding. This study examines the impact of lighting on the visual appearance of one such restored piece: a tiled skirting panel from the Nasrid period (1238-1492), permanently on display at the Museum of the Alhambra (Spain). Spectral images in the range of 380-1080 nm were obtained using a hyperspectral image scanner. CIELAB and CIEDE2000 color coordinates at each pixel were computed assuming the CIE 1931 standard colorimetric observer and considering ten relevant illuminants proposed by the International Commission on Illumination (CIE): D65 plus nine white LEDs. Four main hues (blue, green, yellow, and black) can be distinguished in the original and reintegrated areas. For each hue, mean color difference from the mean (MCDM), CIEDE2000 average distances, volumes, and overlapping volumes were computed in the CIELAB space by comparing the original and the reintegrated zones. The study reveals noticeable average color differences between the original and reintegrated areas within tiles: 6.0 and 4.7 CIEDE2000 units for the yellow and blue tiles (with MCDM values of 3.7 and 4.5 and 5.8 and 7.2, respectively), and 16.6 and 17.8 CIEDE2000 units for the black and green tiles (with MCDM values of 13.2 and 12.2 and 10.9 and 11.3, respectively). The overlapping volume of CIELAB clouds of points corresponding to the original and reintegrated areas ranges from 35% to 50%, indicating that these areas would be perceived as different by observers with normal color vision for all four tiles. However, average color differences between the original and reintegrated areas changed with the tested illuminants by less than 2.6 CIEDE2000 units. Our current methodology provides useful quantitative results for evaluation of the color appearance of a reintegrated area under different light sources, helping curators and museum professionals to choose optimal lighting.

Predicted effectiveness of EnChroma multi-notch filters for enhancing color perception in anomalous trichromats

EnChroma filters are aids designed to improve color vision for anomalous trichromats. Their use is controversial because the results of lab-based assessments of their effectiveness have so far largely failed to agree with positive anecdotal reports. However, the effectiveness of EnChroma filters will vary depending on the conditions of viewing, including whether the stimuli are broadband reflective surfaces or colors presented on RGB displays, whether illumination spectra are broadband or narrowband, the transmission spectra of particular filters, and the cone spectral sensitivity functions of the observer. We created a model of anomalous trichromatic color vision to predict the effects of EnChroma filters on the color signals impaired in anomalous trichromacy. Using the model we varied illumination, filter type and observer cone sensitivity functions, and tested the effect of presenting colors as broadband reflective surfaces or on RGB displays. We also used hyperspectral images to assess the impact of the filters on anomalous trichromats' color vision for natural scenes. Model results predicted that the filters should be broadly effective at enhancing anomalous trichromats' equivalent to L/(L + M) chromatic contrasts under a range of viewing conditions, but are substantially more effective for deuteranomals than for protanomals. The filters are predicted to be more effective for broadband reflective surfaces presented under broadband illuminants than for surfaces presented under narrowband illuminants or for colors presented on RGB displays. Since the potential impacts of contrast adaptation and perceptual learning are not considered in the model, it needs to be empirically validated. Results of empirical tests of the effects of EnChroma filters on deuteranomalous color vision in comparison with model predictions are presented in an accompanying paper (Somers et al., in prep.).

RGB color constancy using multispectral pixel information

Multispectral imaging is a technique that captures data across several bands of the light spectrum, and it can be useful in many computer vision fields, including color constancy. We propose a method that exploits multispectral imaging for illuminant estimation, and then applies illuminant correction in the raw RGB domain to achieve computational color constancy. Our proposed method is composed of two steps: first, a selected number of existing camera-independent algorithms for illuminant estimation, originally designed for RGB data, are applied in generalized form to work with multispectral data. We demonstrate that the sole multispectral extension of such algorithms is not sufficient to achieve color constancy, and thus we introduce a second step, in which we re-elaborate the multispectral estimations before conversion into raw RGB with the use of the camera response function. Our results on the NUS dataset show that an improvement of 60% in the color constancy performance, measured in terms of reproduction angular error, can be obtained according to our method when compared to the traditional raw RGB pipeline.

Color constancy in real-world settings

Color constancy denotes the ability to assign a particular and stable color percept to an object, irrespective of its surroundings and illumination. The light reaching the eye confounds illumination and spectral reflectance of the object, making the recovery of constant object color an ill-posed problem. How good the visual system is at accomplishing this task is still a matter of heated debate, despite more than a 100 years of research. Depending on the laboratory task and the specific cues available to observers, color constancy was found to be at levels ranging between 15% and 80%, which seems incompatible with the relatively stable color appearance of objects around us and the consistent usage of color names in real life. Here, we show close-to-perfect color constancy using real objects in a natural task and natural environmental conditions, chosen to mimic the role of color constancy in everyday life. Participants had to identify the color of a (non-present) item familiar to them in an office room under five different experimental illuminations. They mostly selected the same colored Munsell chip as their match to the absent object, even though the light reaching the eye in each case differed substantially. Our results demonstrate that color constancy under ideal conditions in the real world can indeed be exceptionally good. We found it to be as good as visual memory permits and not generally compromised by sensory uncertainty.

Misidentifying illuminant changes in natural scenes due to failures in relational colour constancy

The colours of surfaces in a scene may not appear constant with a change in the colour of the illumination. Yet even when colour constancy fails, human observers can usually discriminate changes in lighting from changes in surface reflecting properties. This operational ability has been attributed to the constancy of perceived colour relations between surfaces under illuminant changes, in turn based on approximately invariant spatial ratios of cone photoreceptor excitations. Natural deviations in these ratios may, however, lead to illuminant changes being misidentified. The aim of this work was to test whether such misidentifications occur with natural scenes and whether they are due to failures in relational colour constancy. Pairs of scene images from hyperspectral data were presented side-by-side on a computer-controlled display. On one side, the scene underwent illuminant changes and on the other side, it underwent the same changes but with images corrected for any residual deviations in spatial ratios. Observers systematically misidentified the corrected images as being due to illuminant changes. The frequency of errors increased with the size of the deviations, which were closely correlated with the estimated failures in relational colour constancy.

Colour vision is aligned with natural scene statistics at 4 months of age

Visual perception in adult humans is thought to be tuned to represent the statistical regularities of natural scenes. For example, in adults, visual sensitivity to different hues shows an asymmetry which coincides with the statistical regularities of colour in the natural world. Infants are sensitive to statistical regularities in social and linguistic stimuli, but whether or not infants' visual systems are tuned to natural scene statistics is currently unclear. We measured colour discrimination in infants to investigate whether or not the visual system can represent chromatic scene statistics in very early life. Our results reveal the earliest association between vision and natural scene statistics that has yet been found: even as young as 4 months of age, colour vision is aligned with the distributions of colours in natural scenes. RESEARCH HIGHLIGHTS: We find infants' colour sensitivity is aligned with the distribution of colours in the natural world, as it is in adults. At just 4 months, infants' visual systems are tailored to extract and represent the statistical regularities of the natural world. This points to a drive for the human brain to represent statistical regularities even at a young age.

Analysis of Hyperspectral Data to Develop an Approach for Document Images

Hyperspectral data analysis is being utilized as an effective and compelling tool for image processing, providing unprecedented levels of information and insights for various applications. In this manuscript, we have compiled and presented a comprehensive overview of recent advances in hyperspectral data analysis that can provide assistance for the development of customized techniques for hyperspectral document images. We review the fundamental concepts of hyperspectral imaging, discuss various techniques for data acquisition, and examine state-of-the-art approaches to the preprocessing, feature extraction, and classification of hyperspectral data by taking into consideration the complexities of document images. We also explore the possibility of utilizing hyperspectral imaging for addressing critical challenges in document analysis, including document forgery, ink age estimation, and text extraction from degraded or damaged documents. Finally, we discuss the current limitations of hyperspectral imaging and identify future research directions in this rapidly evolving field. Our review provides a valuable resource for researchers and practitioners working on document image processing and highlights the potential of hyperspectral imaging for addressing complex challenges in this domain.

Discrimination of natural colors in anomalous trichromacy and the effects of EnChroma and Vino filters

It is still unclear how well anomalous trichromats discriminate natural colors and whether commercial spectral filters improve performance in these conditions. We show that anomalous trichromats have good color discrimination with colors drawn from natural environments. It is only about 14% poorer, on average, than normal trichromats in our sample of thirteen anomalous trichromats. No measurable effect of the filters on discrimination was found, even after 8 hours of continuous use. Computations of cone and post-receptoral signals show only a modest increase in medium-to-long-wavelength difference signals, which may explain the absent effect of the filters.

Filter design and color correction for the X-cube prism 3CCD camera

For the X-cube prism three-charge-coupled-device (3CCD) camera, the spectra of the designed dichroic films in the X-cube prism shift with changes in the angle of incident light, producing non-uniformity of color on the image plane. We considered the influence of the incident angle on color performance in filter design and directly optimized the thin film to improve color consistency. An optical model was constructed to calculate the distribution of camera spectral sensitivity and independently correct the non-uniform color on the image plane. Results showed that the optimization and correction methods could significantly improve the color performance of the X-cube prism 3CCD camera.

The effect of a short-wave filtering contact lens on color appearance

We assessed the effect of a contact lens that filters short-wavelength (SW) visible light on color appearance. These effects were modeled and measured by direct comparison to a clear contact lens. Sixty-one subjects were enrolled, and 58 completed as cohort; 31 were 18 to 39 years old (mean ± SD, 29.6 ± 5.6), 27 were 40 to 65 years old (50.1 ± 8.1). A double-masked contralateral design was used; participants randomly wore a SW-filtering contact lens on one eye and a clear control lens on the other eye. Subjects then mixed three primaries (including a short-wave primary, strongly within the absorbance of the test lens) until a perceived perfect neutral white was achieved with each eye. Color appearance was quantified using chromaticity coordinates measured with a spectral radiometer within a custom-built tricolorimeter. Color vision in natural scenes was simulated using hyperspectral images and cone fundamentals based on a standard observer. Overall, the chromaticity coordinates of matches that were set using the SW-filtering contact lens (n = 58; x = 0.345, y = 0.325, u' = 0.222, v' = 0.470) and clear contact lens (n = 58; x = 0.344, y = 0.325, u' = 0.223, v' = 0.471) were not significantly different, regardless of age group. Simulations indicated that, for natural scenes, the SW-filtering contact lens that was evaluated changes L/(L+M) and S/(L+M) chromatic contrast by no more than -1.4% to +1.1% and -36.9% to +5.0%, respectively. Tricolorimetry was used to measure color appearance in subjects wearing a SW-filtering lens in one eye and a clear lens in the other, and the results indicate that imparting a subtle tint to a contact lens, as in the SW-filtering lens that was evaluated, does not alter color appearance for younger or older subjects. A model of color vision predicted little effect of the lens on chromatic contrast for natural scenes.

Feasibility of a Real-Time Embedded Hyperspectral Compressive Sensing Imaging System

Hyperspectral imaging has been attracting considerable interest as it provides spectrally rich acquisitions useful in several applications, such as remote sensing, agriculture, astronomy, geology and medicine. Hyperspectral devices based on compressive acquisitions have appeared recently as an alternative to conventional hyperspectral imaging systems and allow for data-sampling with fewer acquisitions than classical imaging techniques, even under the Nyquist rate. However, compressive hyperspectral imaging requires a reconstruction algorithm in order to recover all the data from the raw compressed acquisition. The reconstruction process is one of the limiting factors for the spread of these devices, as it is generally time-consuming and comes with a high computational burden. Algorithmic and material acceleration with embedded and parallel architectures (e.g., GPUs and FPGAs) can considerably speed up image reconstruction, making hyperspectral compressive systems suitable for real-time applications. This paper provides an in-depth analysis of the required performance in terms of computing power, data memory and bandwidth considering a compressive hyperspectral imaging system and a state-of-the-art reconstruction algorithm as an example. The results of the analysis show that real-time application is possible by combining several approaches, namely, exploitation of system matrix sparsity and bandwidth reduction by appropriately tuning data value encoding.

Validation of a Hyperspectral Imaging System for Color Measurement of In-Vivo Dental Structures

A full comprehension of colorimetric relationships within and between teeth is key for aesthetic success of a dental restoration. In this sense, hyperspectral imaging can provide point-wise reliable measurements of the tooth surface, which can serve for this purpose. The aim of this study was to use a hyperspectral imaging system for the colorimetric characterization of 4 in-vivo maxillary anterior teeth and to cross-check the results with similar studies carried out with other measuring systems in order to validate the proposed capturing protocol. Hyperspectral reflectance images (Specim IQ), of the upper central (UCI) and lateral incisors (ULI), were captured on 30 participants. CIE-L*a*b* values were calculated for the incisal (I), middle (M) and cervical (C) third of each target tooth. ΔEab* and ΔE00 total color differences were computed between different tooth areas and adjacent teeth, and evaluated according to the perceptibility (PT) and acceptability (AT) thresholds for dentistry. Non-perceptible color differences were found between UCIs and ULIs. Mean color differences between UCI and ULI exceeded AT (ΔEab* = 7.39-7.42; ΔE00 = 5.71-5.74) in all cases. Large chromatic variations between I, M and C areas of the same tooth were registered (ΔEab* = 5.01-6.07 and ΔE00 = 4.07-5.03; ΔEab* = 5.80-8.16 and ΔE00 = 4.37-5.15; and ΔEab* = 5.42-5.92 and ΔE00 = 3.87-4.16 between C and M, C and I and M and I, respectively). The use of a hyperspectral camera has proven to be a reliable and effective method for color evaluation of in-vivo natural teeth.

Understanding insect colour constancy

Colour constancy is the ability to recognize the colour of objects despite spectral changes in the natural illumination. As such, this phenomenon is important for most organisms with good colour vision, and it has been intensively studied in humans and primates. Colour constancy is also documented for several species of insects, which is not surprising given the ecological importance of colour vision. But how do insects, with their small brains, solve the complex problem of colour vision and colour constancy? In an interspecies approach, this review reports on behavioural studies on colour constancy in bees, butterflies, moths and humans, corresponding computational models and possible neurophysiological correlates. This article is part of the theme issue 'Understanding colour vision: molecular, physiological, neuronal and behavioural studies in arthropods'.

CVD-MET: an image difference metric designed for analysis of color vision deficiency aids

Color vision deficiency (CVD) has gained in relevance in the last decade, with a surge of proposals for aid systems that aim to improve the color discrimination capabilities of CVD subjects. This paper focuses on the proposal of a new metric called CVD-MET, that can evaluate the efficiency and naturalness of these systems through a set of images using a simulation of the subject's vision. In the simulation, the effect of chromatic adaptation is introduced via CIECAM02, which is relevant for the evaluation of passive aids (color filters). To demonstrate the potential of the CVD-MET, an evaluation of a representative set of passive and active aids is carried out both with conventional image quality metrics and with CVD-MET. The results suggest that the active aids (recoloration algorithms) are in general more efficient and produce more natural images, although the changes that are introduced do not shift the CVD's perception of the scene towards the normal observer's perception.

Low Rank Tensor Completion With Poisson Observations

Poisson observations for videos are important models in video processing and computer vision. In this paper, we study the third-order tensor completion problem with Poisson observations. The main aim is to recover a tensor based on a small number of its Poisson observation entries. A existing matrix-based method may be applied to this problem via the matricized version of the tensor. However, this method does not leverage on the global low-rankness of a tensor and may be substantially suboptimal. Our approach is to consider the maximum likelihood estimate of the Poisson distribution, and utilize the Kullback-Leibler divergence for the data-fitting term to measure the observations and the underlying tensor. Moreover, we propose to employ a transformed tensor nuclear norm ball constraint and a bounded constraint of each entry, where the transformed tensor nuclear norm is used to get a lower transformed multi-rank tensor with suitable unitary transformation matrices. We show that the upper bound of the error of the estimator of the proposed model is less than that of the existing matrix-based method. Also an information theoretic lower error bound is established. An alternating direction method of multipliers is developed to solve the resulting convex optimization model. Extensive numerical experiments on synthetic data and real-world datasets are presented to demonstrate the effectiveness of our proposed model compared with existing tensor completion methods.

An image reconstruction framework for characterizing initial visual encoding

We developed an image-computable observer model of the initial visual encoding that operates on natural image input, based on the framework of Bayesian image reconstruction from the excitations of the retinal cone mosaic. Our model extends previous work on ideal observer analysis and evaluation of performance beyond psychophysical discrimination, takes into account the statistical regularities of the visual environment, and provides a unifying framework for answering a wide range of questions regarding the visual front end. Using the error in the reconstructions as a metric, we analyzed variations of the number of different photoreceptor types on human retina as an optimal design problem. In addition, the reconstructions allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling, and how these vary with eccentricity. Furthermore, in simulations of color deficiencies and interferometric experiments, we found that the reconstructed images provide a reasonable proxy for modeling subjects' percepts. Lastly, we used the reconstruction-based observer for the analysis of psychophysical threshold, and found notable interactions between spatial frequency and chromatic direction in the resulting spatial contrast sensitivity function. Our method is widely applicable to experiments and applications in which the initial visual encoding plays an important role.

Luminosity thresholds of colored surfaces are determined by their upper-limit luminances empirically internalized in the visual system

We typically have a fairly good idea whether a given object is self-luminous or illuminated, but it is not fully understood how we make this judgment. This study aimed to identify determinants of the luminosity threshold, a luminance level at which a surface begins to appear self-luminous. We specifically tested a hypothesis that our visual system knows the maximum luminance level that a surface can reach under the physical constraint that a surface cannot reflect more light than any incident light and applies this prior to determine the luminosity thresholds. Observers were presented with a 2-degree circular test field surrounded by numerous overlapping colored circles and luminosity thresholds were measured as a function of (i) the chromaticity of the test field, (ii) the shape of surrounding color distribution, and (iii) the color of the illuminant of the surrounding colors. We found that the luminosity thresholds peaked around the chromaticity of test illuminants and decreased as the purity of the test chromaticity increased. However, the loci of luminosity thresholds across chromaticities were nearly invariant to the shape of the surrounding color distribution and generally resembled the loci drawn from theoretical upper-limit luminances and upper-limit luminance boundaries of real objects. These trends were particularly evident for illuminants on the black-body locus and did not hold well under atypical illuminants, such as magenta or green. These results support the idea that our visual system empirically internalizes the gamut of surface colors under natural illuminants and a given object appears self-luminous when its luminance exceeds this internalized upper-limit luminance.

Naturalness and aesthetics of colors - Preference for color compositions perceived as natural

What makes a colored image, e.g. an abstract painting or a landscape, look pleasing? We hypothesized that a preference for complex color compositions, such as paintings and images of natural scenes, might be related to how natural the colors are perceived. We tested this possibility with two experiments in which the degree of naturalness of images was manipulated by rotating their color gamut rigidly in the color space CIELAB. This changed just the hue composition, but preserved saturation and lightness. In the first experiment we obtained individual scaling curves for perceived naturalness and for preference as a function of the angle of gamut rotation for a small set of images. The naturalness and preference scaling curves were found to be largely similar and their maxima were close to the original image. In the second experiment, we tested whether this effect generalized to a larger set of images. We used a simultaneous 5AFC procedure where in each trial participants had to select the most natural or the most preferred image from five different rotations of the color gamut. The results confirmed the first experiment and showed that, in general, the images perceived as the more natural tend to be the ones that are preferred. Together these results show that perceived naturalness and preference are indeed perceptually closely related and may be driven by related mechanisms.

Color Can Shorten Breakthrough Times in Continuous Flash Suppression through Increased Salience and Task Relevance

Color can enhance the perception of relevant stimuli by increasing their salience and guiding visual search towards stimuli that match a task-relevant color. Using Continuous Flash Suppression (CFS), the current study investigated whether color facilitates the discrimination of targets that are difficult to perceive due to interocular suppression. Gabor patterns of two or four cycles per degree (cpd) were shown as targets to the non-dominant eye of human participants. CFS masks were presented at a rate of 10 Hz to the dominant eye, and participants had the task to report the target’s orientation as soon as they could discriminate it. The 2-cpd targets were robustly suppressed and resulted in much longer response times compared to 4-cpd targets. Moreover, only for 2-cpd targets, two color-related effects were evident. First, in trials where targets and CFS masks had different colors, targets were reported faster than in trials where targets and CFS masks had the same color. Second, targets with a known color, either cyan or yellow, were reported earlier than targets whose color was randomly cyan or yellow. The results suggest that the targets’ entry to consciousness may have been speeded by color-mediated effects relating to increased (bottom-up) salience and (top-down) task relevance.

Color for object recognition: Hue and chroma sensitivity in the deep features of convolutional neural networks

In this work, we examined the color tuning of units in the hidden layers of AlexNet, VGG-16 and VGG-19 convolutional neural networks and their relevance for the successful recognition of an object. We first selected the patches for which the units are maximally responsive among the 1.2 M images of the ImageNet training dataset. We segmented these patches using a k-means clustering algorithm on their chromatic distribution. Then we independently varied the color of these segments, both in hue and chroma, to measure the unit's chromatic tuning. The models exhibited properties at times similar or opposed to the known chromatic processing of biological system. We found that, similarly to the most anterior occipital visual areas in primates, the last convolutional layer exhibited high color sensitivity. We also found the gradual emergence of single to double opponent kernels. Contrary to cells in the visual system, however, these kernels were selective for hues that gradually transit from being broadly distributed in early layers, to mainly falling along the blue-orange axis in late layers. In addition, we found that the classification performance of our models varies as we change the color of our stimuli following the models' kernels properties. Performance was highest for colors the kernels maximally responded to, and images responsible for the activation of color sensitive kernels were more likely to be mis-classified as we changed their color. These observations were shared by all three networks, thus suggesting that they are general properties of current convolutional neural networks trained for object recognition.

Colors and Things

How many colors are there? Quoted numbers range from ten million to a dozen. Are colors object properties? Opinions range all the way from of course they are to no, colors are just mental paint. These questions are ill-posed. We submit that the way to tackle such questions is to adopt a biological approach, based on the evolutionary past of hominins. Hunter-gatherers in tundra or savannah environments have various, mutually distinct uses for color. Color differences aid in segmenting the visual field, whereas color qualia aid in recognizing objects. Classical psychophysics targets the former, but mostly ignores the latter, whereas experimental phenomenology, for instance in color naming, is relevant for recognition. Ecological factors, not anatomical/physiological ones, limit the validity of qualia as distinguishing signs. Spectral databases for varieties of daylight and object reflectance factors allow one to model this. The two questions are really one. A valid question that may replace both is how many distinguishing signs does color vision offer in the hominin Umwelt? The answer turns out to be about a thousand. The reason is that colors are formally not object properties but pragmatically are useful distinguishing signs.

Mixed 2D/3D Convolutional Network for Hyperspectral Image Super-Resolution

Deep learning-based hyperspectral image super-resolution (SR) methods have achieved great success recently. However, there are two main problems in the previous works. One is to use the typical three-dimensional convolution analysis, resulting in more parameters of the network. The other is not to pay more attention to the mining of hyperspectral image spatial information, when the spectral information can be extracted. To address these issues, in this paper, we propose a mixed convolutional network (MCNet) for hyperspectral image super-resolution. We design a novel mixed convolutional module (MCM) to extract the potential features by 2D/3D convolution instead of one convolution, which enables the network to more mine spatial features of hyperspectral image. To explore the effective features from 2D unit, we design the local feature fusion to adaptively analyze from all the hierarchical features in 2D units. In 3D unit, we employ spatial and spectral separable 3D convolution to extract spatial and spectral information, which reduces unaffordable memory usage and training time. Extensive evaluations and comparisons on three benchmark datasets demonstrate that the proposed approach achieves superior performance in comparison to existing state-of-the-art methods.

Spectral Image Processing for Museum Lighting Using CIE LED Illuminants

This work presents a spectral color-imaging procedure for the detailed colorimetric study of real artworks under arbitrary illuminants. The results demonstrate this approach to be a powerful tool for art and heritage professionals when deciding which illumination to use in museums, or which conservation or restoration techniques best maintain the color appearance of the original piece under any illuminant. Spectral imaging technology overcomes the limitations of common area-based point-measurement devices such as spectrophotometers, allowing a local study either pixelwise or by selected areas. To our knowledge, this is the first study available that uses the proposed CIE (Commission Internationale de l'Éclairage) light-emitting diode (LED) illuminants in the context of art and heritage science, comparing them with the three main CIE illuminants A, D50, and D65. For this, the corresponding colors under D65 have been calculated using a chromatic adaptation transform analogous to the one in CIECAM02. For the sample studied, the CIE LED illuminants with the lowest average CIEDE2000 color differences from the standard CIE illuminants are LED-V1 for A and LED-V2 for D50 and D65, with 1.23, 1.07, and 1.57 units, respectively. The work studied is a Moorish epigraphic frieze of plasterwork with a tiled skirting from the Nasrid period (12th-15th centuries) exhibited in the Museum of the Alhambra (Granada, Spain).

Reconciling the statistics of spectral reflectance and colour

The spectral reflectance function of a surface specifies the fraction of the illumination reflected by it at each wavelength. Jointly with the illumination spectral density, this function determines the apparent colour of the surface. Models for the distribution of spectral reflectance functions in the natural environment are considered. The realism of the models is assessed in terms of the individual reflectance functions they generate, and in terms of the overall distribution of colours which they give rise to. Both realism assessments are made in comparison to empirical datasets. Previously described models (PCA- and fourier-based) of reflectance function statistics are evaluated, as are improved versions; and also a novel model, which synthesizes reflectance functions as a sum of sigmoid functions. Key model features for realism are identified. The new sigmoid-sum model is shown to be the most realistic, generating reflectance functions that are hard to distinguish from real ones, and accounting for the majority of colours found in natural images with the exception of an abundance of vegetation green and sky blue.

Coregistered optical coherence tomography and frequency-encoded multispectral imaging for spectrally sparse color imaging

We present a system combining optical coherence tomography (OCT) and multispectral imaging (MSI) for coregistered structural imaging and surface color imaging. We first describe and numerically validate an optimization model to guide the selection of the MSI wavelengths and their relative intensities. We then demonstrate the integration of this model into an all-fiber bench-top system. We implement frequency-domain multiplexing for the MSI to enable concurrent acquisition of both OCT and MSI at OCT acquisition rates. Such a system could be implemented in endoscopic practices to provide multimodal, high-resolution imaging of deep organ structures that are currently inaccessible to standard video endoscopes.

Appearance of Maxwell's spot in images rendered using a cyan primary

The discovery of melanopsin as a third type of retinal photoreceptor, contributing to both perceptual vision and reflex light responses, represents a new opportunity to optimise the design of artificial light sources for practical applications and to generate experimental stimuli. In the case of emissive displays, multiprimary designs incorporating a cyan primary could be used to allow melanopic radiance to be controlled independent of colour and luminance. Here we explore the performance a five-primary (violet, cyan, green, yellow, red) display device and find an anomaly in colour appearance when the cyan primary is employed. The anomaly took the form of a reddish/pinkish tinge in the central visual field, consistent with descriptions of Maxwell’s spot. This effect was apparent in some full colour images and in uniform discs over a range of chromaticities. Its appearance in coloured discs correlated with differences in calculated colour coordinate between central and peripheral vision. A simulation indicated that inclusion of any primary with predominant output in the 470–500 nm range has the potential to produce such a discrepancy in central vs peripheral appearance. Applying an additional constraint in colour processing to reproduce naturally occurring differences in central vs peripheral colour coordinate eliminated appearance of the spot and produced acceptable colour images.

Hyperspectral environmental illumination maps: characterizing directional spectral variation in natural environments

Objects placed in real-world scenes receive incident light from every direction, and the spectral content of this light may vary from one direction to another. In computer graphics, environmental illumination is approximated using maps that specify illumination at a point as a function of incident angle. However, to-date, existing public databases of environmental illumination specify only three colour channels (RGB). We have captured a new set of 12 environmental illumination maps (eight outdoor scenes; four indoor scenes) using a hyperspectral imaging system with 33 spectral channels. The data reveal a striking directional variation of spectral distribution of lighting in natural environments. We discuss limitations of using daylight models to describe natural environmental illumination.

Compressive spectral imaging with diffractive lenses

Compressive spectral imaging enables the reconstruction of an entire 3D spectral cube from a few multiplexed images. Here we develop a novel compressive spectral imaging technique using diffractive lenses. Our technique uses a coded aperture to spatially modulate the optical field from the scene and a diffractive lens such as a photon sieve for both dispersion and focusing. Measurement diversity is achieved by changing the focusing behavior of the diffractive lens. The 3D spectral cube is then reconstructed from highly compressed measurements taken with a monochrome detector. A fast sparse recovery method is developed to solve this large-scale inverse problem. The performance is illustrated for various scenarios with different compression ratios through simulations. The results demonstrate that promising reconstruction performance can be achieved at high compression levels. This opens up new possibilities for high-resolution spectral imaging with simpler and low cost designs.

HyTexiLa: High Resolution Visible and Near Infrared Hyperspectral Texture Images

We present a dataset of close range hyperspectral images of materials that span the visible and near infrared spectrums: HyTexiLa (Hyperspectral Texture images acquired in Laboratory). The data is intended to provide high spectral and spatial resolution reflectance images of 112 materials to study spatial and spectral textures. In this paper we discuss the calibration of the data and the method for addressing the distortions during image acquisition. We provide a spectral analysis based on non-negative matrix factorization to quantify the spectral complexity of the samples and extend local binary pattern operators to the hyperspectral texture analysis. The results demonstrate that although the spectral complexity of each of the textures is generally low, increasing the number of bands permits better texture classification, with the opponent band local binary pattern feature giving the best performance.

Color constancy of color reproductions in art paintings

Popular color reproductions of art paintings such as postcards are intended to remind viewers of the original works. It is, however, unclear how well the quality of the reproductions is preserved under various illuminations. Color constancy of the reproductions in relation to colors in the original paintings was estimated computationally with hyperspectral images of 15th-century Flemish paintings, 20th-century modern abstract paintings, and their corresponding postcards with a series of illuminants: the CIE daylight D65 with correlated color temperature (CCT) 6500 K, daylight D40, fluorescent lamps F2 and F11, and a LED lamp designed for museums with CCT approximately 3500-4000 K. Despite large colorimetric differences between the types of art paintings and between the illuminants simulated, local areas showed good color constancy: skin areas in the Flemish paintings ranged from 0.76 to 0.81, whereas nonskin areas ranged from 0.19 to 0.68. This result suggests that viewers may be able to achieve color constancy with the reproduction postcards by disregarding inconsistent colors representations from the original paintings caused by changes in illumination conditions.

A Data Set for Camera-Independent Color Constancy

In this paper, we provide a novel data set designed for Camera-independent color constancy research. Camera independence corresponds to the robustness of an algorithm's performance when it runs on images of the same scene taken by different cameras. Accordingly, the images in our database correspond to several laboratory and field scenes each of which is captured by three different cameras with minimal registration errors. The laboratory scenes are also captured under five different illuminations. The spectral responses of cameras and the spectral power distributions of the laboratory light sources are also provided, as they may prove beneficial for training future algorithms to achieve color constancy. For a fair evaluation of future methods, we provide guidelines for supervised methods with indicated training, validation, and testing partitions. Accordingly, we evaluate two recently proposed convolutional neural network-based color constancy algorithms as baselines for future research. As a side contribution, this data set also includes images taken by a mobile camera with color shading corrected and uncorrected results. This allows research on the effect of color shading as well.

Color naming across languages reflects color use

What determines how languages categorize colors? We analyzed results of the World Color Survey (WCS) of 110 languages to show that despite gross differences across languages, communication of chromatic chips is always better for warm colors (yellows/reds) than cool colors (blues/greens). We present an analysis of color statistics in a large databank of natural images curated by human observers for salient objects and show that objects tend to have warm rather than cool colors. These results suggest that the cross-linguistic similarity in color-naming efficiency reflects colors of universal usefulness and provide an account of a principle (color use) that governs how color categories come about. We show that potential methodological issues with the WCS do not corrupt information-theoretic analyses, by collecting original data using two extreme versions of the color-naming task, in three groups: the Tsimane', a remote Amazonian hunter-gatherer isolate; Bolivian-Spanish speakers; and English speakers. These data also enabled us to test another prediction of the color-usefulness hypothesis: that differences in color categorization between languages are caused by differences in overall usefulness of color to a culture. In support, we found that color naming among Tsimane' had relatively low communicative efficiency, and the Tsimane' were less likely to use color terms when describing familiar objects. Color-naming among Tsimane' was boosted when naming artificially colored objects compared with natural objects, suggesting that industrialization promotes color usefulness.

Improving color constancy by discounting the variation of camera spectral sensitivity

It is an ill-posed problem to recover the true scene colors from a color biased image by discounting the effects of scene illuminant and camera spectral sensitivity (CSS) at the same time. Most color constancy (CC) models have been designed to first estimate the illuminant color, which is then removed from the color biased image to obtain an image taken under white light, without the explicit consideration of CSS effect on CC. This paper first studies the CSS effect on illuminant estimation arising in the inter-dataset-based CC (inter-CC), i.e., training a CC model on one dataset and then testing on another dataset captured by a distinct CSS. We show the clear degradation of existing CC models for inter-CC application. Then a simple way is proposed to overcome such degradation by first learning quickly a transform matrix between the two distinct CSSs (CSS-1 and CSS-2). The learned matrix is then used to convert the data (including the illuminant ground truth and the color-biased images) rendered under CSS-1 into CSS-2, and then train and apply the CC model on the color-biased images under CSS-2 without the need of burdensome acquiring of the training set under CSS-2. Extensive experiments on synthetic and real images show that our method can clearly improve the inter-CC performance for traditional CC algorithms. We suggest that, by taking the CSS effect into account, it is more likely to obtain the truly color constant images invariant to the changes of both illuminant and camera sensors.

How daylight influences high-order chromatic descriptors in natural images

Despite the global and local daylight changes naturally occurring in natural scenes, the human visual system usually adapts quite well to those changes, developing a stable color perception. Nevertheless, the influence of daylight in modeling natural image statistics is not fully understood and has received little attention. The aim of this work was to analyze the influence of daylight changes in different high-order chromatic descriptors (i.e., color volume, color gamut, and number of discernible colors) derived from 350 color images, which were rendered under 108 natural illuminants with Correlated Color Temperatures (CCT) from 2735 to 25,889 K. Results suggest that chromatic and luminance information is almost constant and does not depend on the CCT of the illuminant for values above 14,000 K. Nevertheless, differences between the red-green and blue-yellow image components were found below that CCT, with most of the statistical descriptors analyzed showing local extremes in the range 2950 K-6300 K. Uniform regions and areas of the images attracting observers' attention were also considered in this analysis and were characterized by their patchiness index and their saliency maps. Meanwhile, the results of the patchiness index do not show a clear dependence on CCT, and it is remarkable that a significant reduction in the number of discernible colors (58% on average) was found when the images were masked with their corresponding saliency maps. Our results suggest that chromatic diversity, as defined in terms of the discernible colors, can be strongly reduced when an observer scans a natural scene. These findings support the idea that a reduction in the number of discernible colors will guide visual saliency and attention. Whatever the modeling is mediating the neural representation of natural images, natural image statistics, it is clear that natural image statistics should take into account those local maxima and minima depending on the daylight illumination and the reduction of the number of discernible colors when salient regions are considered.

An evaluation of different measures of color saturation

We investigated how well seven saturation measures defined in CIECAM02, HSV, DKL, LAB, LUV, and CIE 1931 xyY color spaces correspond to human perception of saturation. We used a paradigm that allowed us to measure the perceived saturation of several standard color stimuli in many different directions of color space. We implemented this paradigm at different levels of luminance and varied background luminance relative to the luminance of our color stimuli in order to ensure the generality of our approach. We found that varying background luminance changed the relative saturation of the standard colors. Raising the overall luminance level did not have such an effect. We compared the results of our measurements to the predictions of the seven saturation measures. All of the measures could predict our observers' judgments of saturation reasonably well. The measures that are based on measurements of discrimination thresholds (LUV, LAB, CIECAM02) performed best on average. However, some of the perceptual effects induced by changing background luminance could not be predicted by any measure.

The colors of paintings and viewers' preferences

One hypothesis to explain the aesthetics of paintings is that it depends on the extent to which they mimic natural image statistics. In fact, paintings and natural scenes share several statistical image regularities but the colors of paintings seem generally more biased towards red than natural scenes. Is the particular option for colors in each painting, even if less naturalistic, critical for perceived beauty? Here we show that it is. In the experiments, 50 naïve observers, unfamiliar with the 10 paintings tested, could rotate the color gamut of the paintings and select the one producing the best subjective impression. The distributions of angles obtained are described by normal distributions with maxima deviating, on average, only 7 degrees from the original gamut orientation and full width at half maximum just above the threshold to perceive a chromatic change in the paintings. Crucially, for data pooled across observers and abstract paintings the maximum of the distribution was at zero degrees, i.e., the same as the original. This demonstrates that artists know what chromatic compositions match viewers' preferences and that the option for less naturalistic colors does not constrain the aesthetic value of paintings.

Perception of saturation in natural scenes

We measured how well perception of color saturation in natural scenes can be predicted by different measures that are available in the literature. We presented 80 color images of natural scenes or their gray-scale counterparts to our observers, who were asked to choose the pixel from each image that appeared to be the most saturated. We compared our observers' choices to the predictions of seven popular saturation measures. For the color images, all of the measures predicted perception of saturation quite well, with CIECAM02 performing best. Differences between the measures were small but systematic. When gray-scale images were viewed, observers still chose pixels whose counterparts in the color images were saturated above average. This indicates that image structure and prior knowledge can be relevant to perception of saturation. Nevertheless, our results also show that saturation in natural scenes can be specified quite well without taking these factors into account.

Low levels of specularity support operational color constancy, particularly when surface and illumination geometry can be inferred

We tested whether surface specularity alone supports operational color constancy-the ability to discriminate changes in illumination or reflectance. Observers viewed short animations of illuminant or reflectance changes in rendered scenes containing a single spherical surface and were asked to classify the change. Performance improved with increasing specularity, as predicted from regularities in chromatic statistics. Peak performance was impaired by spatial rearrangements of image pixels that disrupted the perception of illuminated surfaces but was maintained with increased surface complexity. The characteristic chromatic transformations that are available with nonzero specularity are useful for operational color constancy, particularly if accompanied by appropriate perceptual organization.

Statistics of colors in paintings and natural scenes

Painters reproduce some spatial statistical regularities of natural scenes. To what extent they replicate their color statistics is an open question. We investigated this question by analyzing the colors of 50 natural scenes of rural and urban environments and 44 paintings with abstract and figurative compositions. The analysis was carried out using hyperspectral imaging data from both sets and focused on the gamut and distribution of colors in the CIELAB space. The results showed that paintings, like natural scenes, have gamuts with elongated shapes in the yellow-blue direction but more tilted to the red direction. It was also found that the fraction of discernible colors, expressed as a function of the number of occurrences in the scene or painting, is well described by power laws. These have similar distribution of slopes in a log-log scale for paintings and natural scenes. These features are observed in both abstract and figurative compositions. These results suggest that the underlying chromatic structure of artistic compositions generally follows the main statistical features of the natural environment.

Spatial distributions of local illumination color in natural scenes

In natural complex environments, the elevation of the sun and the presence of occluding objects and mutual reflections cause variations in the spectral composition of the local illumination across time and location. Unlike the changes in time and their consequences for color appearance and constancy, the spatial variations of local illumination color in natural scenes have received relatively little attention. The aim of the present work was to characterize these spatial variations by spectral imaging. Hyperspectral radiance images were obtained from 30 rural and urban scenes in which neutral probe spheres were embedded. The spectra of the local illumination at 17 sample points on each sphere in each scene were extracted and a total of 1904 chromaticity coordinates and correlated color temperatures (CCTs) derived. Maximum differences in chromaticities over spheres and over scenes were similar. When data were pooled over scenes, CCTs ranged from 3000 K to 20,000 K, a variation of the same order of magnitude as that occurring over the day. Any mechanisms that underlie stable surface color perception in natural scenes need to accommodate these large spatial variations in local illumination color.

Time-lapse ratios of cone excitations in natural scenes

The illumination in natural environments varies through the day. Stable inferences about surface color might be supported by spatial ratios of cone excitations from the reflected light, but their invariance has been quantified only for global changes in illuminant spectrum. The aim here was to test their invariance under natural changes in both illumination spectrum and geometry, especially in the distribution of shadows. Time-lapse hyperspectral radiance images were acquired from five outdoor vegetated and nonvegetated scenes. From each scene, 10,000 pairs of points were sampled randomly and ratios measured across time. Mean relative deviations in ratios were generally large, but when sampling was limited to short distances or moderate time intervals, they fell below the level for detecting violations in ratio invariance. When illumination changes with uneven geometry were excluded, they fell further, to levels obtained with global changes in illuminant spectrum alone. Within sampling constraints, ratios of cone excitations, and also of opponent-color combinations, provide an approximately invariant signal for stable surface-color inferences, despite spectral and geometric variations in scene illumination.

Recovering of weather degraded images based on RGB response ratio constancy

Images captured under bad weather conditions suffer from poor contrast and visibility. These effects are noticeable for haze, mist, fog, or dust storms. We have proposed a recovering method for images captured for several adverse weather conditions based on the RGB response ratio constancy under illuminant changes. This algorithm improves the visibility, contrast, and color in degraded images with low computational times. We obtain results similar to those from previously published deweathering methods but with no prior information about the image content or atmospheric parameters needed.

What is white?

To shed light on the perceptual basis of the color white, we measured settings of unique white in a dark surround. We find that settings reliably show more variability in an oblique (blue-yellow) direction in color space than along the cardinal axes of the cone-opponent mechanisms. This is against the idea that white perception arises at the null point of the cone-opponent mechanisms, but one alternative possibility is that it occurs through calibration to the visual environment. We found that the locus of maximum variability in settings lies close to the locus of natural daylights, suggesting that variability may result from uncertainty about the color of the illuminant. We tested this by manipulating uncertainty. First, we altered the extent to which the task was absolute (requiring knowledge of the illumination) or relative. We found no clear effect of this factor on the reduction in sensitivity in the blue-yellow direction. Second, we provided a white surround as a cue to the illumination or left the surround dark. Sensitivity was selectively worse in the blue-yellow direction when the surround was black than when it was white. Our results can be functionally related to the statistics of natural images, where a greater blue-yellow dispersion is characteristic of both reflectances (where anisotropy is weak) and illuminants (where it is very pronounced). Mechanistically, the results could suggest a neural signal responsive to deviations from the blue-yellow locus or an adaptively matched range of contrast response functions for signals that encode different directions in color space.