Frequency of metamerism in natural scenes

Colour expectations across illumination changes

This study investigates human expectations towards naturalistic colour changes under varying illuminations. Understanding colour expectations is key to both scientific research on colour constancy and applications of colour and lighting in art and industry. We reanalysed data from asymmetric colour matches of a previous study and found that colour adjustments tended to align with illuminant-induced colour shifts predicted by naturalistic, rather than artificial, illuminants and reflectances. We conducted three experiments using hyperspectral images of naturalistic scenes to test if participants judged colour changes based on naturalistic illuminant and reflectance spectra as more plausible than artificial ones, which contradicted their expectations. When we consistently manipulated the illuminant (Experiment 1) and reflectance (Experiment 2) spectra across the whole scene, observers chose the naturalistic renderings significantly above the chance level (>25 %) but barely more often than any of the three artificial ones, collectively (>50 %). However, when we manipulated only one object/area's reflectance (Experiment 3), observers more reliably identified the version in which the object had a naturalistic reflectance like the rest of the scene. Results from Experiments 2-3 and additional analyses suggested that relational colour constancy strongly contributed to observer expectations, and stable cone-excitation ratios are not limited to naturalistic illuminants and reflectances but also occur for our artificial renderings. Our findings indicate that relational colour constancy and prior knowledge about surface colour shifts help to disambiguate surface colour identity under illumination changes, enabling human observers to recognise surface colours reliably in naturalistic conditions. Additionally, relational colour constancy may even be effective in many artificial conditions.

How the orientation of the color gamut of natural scenes influences color discrimination in red-green dichromacy

In natural scenes, visual discrimination of colored surfaces by individuals with X-linked dichromacy is known to be only a little poorer than in normal trichromacy. This surprising result may be related to the properties of the colors of these scenes, like the shape and orientation of the color gamut, uneven frequency, and a considerable variation in lightness. It is unclear, however, how much each of these factors contributes to the small impairment in discrimination, in particular, what is the contribution of the orientation of the gamut. We measured the discrimination of colors from natural scenes by six normal trichromats and six dichromats. Colors were drawn either from the original color gamut of the scenes or from gamut-rotated versions of the scenes. Pairs of colors were randomly drawn from hyperspectral images of one rural and one urban environment and presented on a screen. As expected, dichromats were only a little poorer than normal trichromats at discrimination but the disadvantage varied systematically with the orientation of the color gamut by a factor of about three with a minimum around a yellow-green axis. Dichromats also took longer to respond, and the response times were modulated with the orientation of the color gamut in a similar way as the loss in discrimination. For the scenes tested here, these results imply an important impact of the orientation of the gamut on discrimination. They also indicate that the predominantly yellow-blue orientation of the gamut of natural scene might not be optimal for discrimination in dichromacy.

Predicted effectiveness of EnChroma multi-notch filters for enhancing color perception in anomalous trichromats

EnChroma filters are aids designed to improve color vision for anomalous trichromats. Their use is controversial because the results of lab-based assessments of their effectiveness have so far largely failed to agree with positive anecdotal reports. However, the effectiveness of EnChroma filters will vary depending on the conditions of viewing, including whether the stimuli are broadband reflective surfaces or colors presented on RGB displays, whether illumination spectra are broadband or narrowband, the transmission spectra of particular filters, and the cone spectral sensitivity functions of the observer. We created a model of anomalous trichromatic color vision to predict the effects of EnChroma filters on the color signals impaired in anomalous trichromacy. Using the model we varied illumination, filter type and observer cone sensitivity functions, and tested the effect of presenting colors as broadband reflective surfaces or on RGB displays. We also used hyperspectral images to assess the impact of the filters on anomalous trichromats' color vision for natural scenes. Model results predicted that the filters should be broadly effective at enhancing anomalous trichromats' equivalent to L/(L + M) chromatic contrasts under a range of viewing conditions, but are substantially more effective for deuteranomals than for protanomals. The filters are predicted to be more effective for broadband reflective surfaces presented under broadband illuminants than for surfaces presented under narrowband illuminants or for colors presented on RGB displays. Since the potential impacts of contrast adaptation and perceptual learning are not considered in the model, it needs to be empirically validated. Results of empirical tests of the effects of EnChroma filters on deuteranomalous color vision in comparison with model predictions are presented in an accompanying paper (Somers et al., in prep.).

Are ipRGCs involved in human color vision? Hints from physiology, psychophysics, and natural image statistics

Human photoreceptors consist of cones, rods, and melanopsin-expressing intrinsically photosensitive retinal ganglion cells (ipRGCs). First studied in circadian regulation and pupillary control, ipRGCs project to a variety of brain centers suggesting a broader involvement beyond non-visual functions. IpRGC responses are stable, long-lasting, and with a particular codification of photoreceptor signals. In comparison with the transient and adaptive nature of cone and rod signals, ipRGCs' signaling might provide an ecological advantage to different attributes of color vision. Previous studies have indicated melanopsin's influence on visual responses yet its contribution to color perception in humans remains debated. We summarized evidence and hypotheses (from physiology, psychophysics, and natural image statistics) about direct and indirect involvement of ipRGCs in human color vision, by first briefly assessing the current knowledge about the role of melanopsin and ipRGCs in vision and codification of spectral signals. We then approached the question about melanopsin activation eliciting a color percept, discussing studies using the silent substitution method. Finally, we explore various avenues through which ipRGCs might impact color perception indirectly, such as through involvement in peripheral color matching, post-receptoral pathways, color constancy, long-term chromatic adaptation, and chromatic induction. While there is consensus about the role of ipRGCs in brightness perception, confirming its direct contribution to human color perception requires further investigation. We proposed potential approaches for future research, emphasizing the need for empirical validation and methodological thoroughness to elucidate the exact role of ipRGCs in human color vision.

RGB color constancy using multispectral pixel information

Multispectral imaging is a technique that captures data across several bands of the light spectrum, and it can be useful in many computer vision fields, including color constancy. We propose a method that exploits multispectral imaging for illuminant estimation, and then applies illuminant correction in the raw RGB domain to achieve computational color constancy. Our proposed method is composed of two steps: first, a selected number of existing camera-independent algorithms for illuminant estimation, originally designed for RGB data, are applied in generalized form to work with multispectral data. We demonstrate that the sole multispectral extension of such algorithms is not sufficient to achieve color constancy, and thus we introduce a second step, in which we re-elaborate the multispectral estimations before conversion into raw RGB with the use of the camera response function. Our results on the NUS dataset show that an improvement of 60% in the color constancy performance, measured in terms of reproduction angular error, can be obtained according to our method when compared to the traditional raw RGB pipeline.

Color constancy in real-world settings

Color constancy denotes the ability to assign a particular and stable color percept to an object, irrespective of its surroundings and illumination. The light reaching the eye confounds illumination and spectral reflectance of the object, making the recovery of constant object color an ill-posed problem. How good the visual system is at accomplishing this task is still a matter of heated debate, despite more than a 100 years of research. Depending on the laboratory task and the specific cues available to observers, color constancy was found to be at levels ranging between 15% and 80%, which seems incompatible with the relatively stable color appearance of objects around us and the consistent usage of color names in real life. Here, we show close-to-perfect color constancy using real objects in a natural task and natural environmental conditions, chosen to mimic the role of color constancy in everyday life. Participants had to identify the color of a (non-present) item familiar to them in an office room under five different experimental illuminations. They mostly selected the same colored Munsell chip as their match to the absent object, even though the light reaching the eye in each case differed substantially. Our results demonstrate that color constancy under ideal conditions in the real world can indeed be exceptionally good. We found it to be as good as visual memory permits and not generally compromised by sensory uncertainty.

Misidentifying illuminant changes in natural scenes due to failures in relational colour constancy

The colours of surfaces in a scene may not appear constant with a change in the colour of the illumination. Yet even when colour constancy fails, human observers can usually discriminate changes in lighting from changes in surface reflecting properties. This operational ability has been attributed to the constancy of perceived colour relations between surfaces under illuminant changes, in turn based on approximately invariant spatial ratios of cone photoreceptor excitations. Natural deviations in these ratios may, however, lead to illuminant changes being misidentified. The aim of this work was to test whether such misidentifications occur with natural scenes and whether they are due to failures in relational colour constancy. Pairs of scene images from hyperspectral data were presented side-by-side on a computer-controlled display. On one side, the scene underwent illuminant changes and on the other side, it underwent the same changes but with images corrected for any residual deviations in spatial ratios. Observers systematically misidentified the corrected images as being due to illuminant changes. The frequency of errors increased with the size of the deviations, which were closely correlated with the estimated failures in relational colour constancy.

Performance Comparison of Classical Methods and Neural Networks for Colour Correction

Colour correction is the process of converting RAW RGB pixel values of digital cameras to a standard colour space such as CIE XYZ. A range of regression methods including linear, polynomial and root-polynomial least-squares have been deployed. However, in recent years, various neural network (NN) models have also started to appear in the literature as an alternative to classical methods. In the first part of this paper, a leading neural network approach is compared and contrasted with regression methods. We find that, although the neural network model supports improved colour correction compared with simple least-squares regression, it performs less well than the more advanced root-polynomial regression. Moreover, the relative improvement afforded by NNs, compared to linear least-squares, is diminished when the regression methods are adapted to minimise a perceptual colour error. Problematically, unlike linear and root-polynomial regressions, the NN approach is tied to a fixed exposure (and when exposure changes, the afforded colour correction can be quite poor). We explore two solutions that make NNs more exposure-invariant. First, we use data augmentation to train the NN for a range of typical exposures and second, we propose a new NN architecture which, by construction, is exposure-invariant. Finally, we look into how the performance of these algorithms is influenced when models are trained and tested on different datasets. As expected, the performance of all methods drops when tested with completely different datasets. However, we noticed that the regression methods still outperform the NNs in terms of colour correction, even though the relative performance of the regression methods does change based on the train and test datasets.

On the utility of Colour in shape analysis: An introduction to Colour science via palaeographical case studies

In this article, we explore the use of colour for the analysis of shapes in digital images. We argue that colour can provide unique information that is not available from shape alone, and that familiarity with the interdisciplinary field of colour science is essential for unlocking the potential of colour. Within this perspective, we offer an illustrated overview of the colour-related aspects of image management and processing, perceptual psychology, and cultural studies, using for exemplary purposes case studies focused on computational palaeography. We also discuss the changing roles of colour in society and the sciences, and provide technical solutions for using digital colour effectively, highlighting the impact of human factors. The article concludes with an annotated bibliography. This work is a primer, and its intended readership are scholars and computer scientists unfamiliar with colour science.

Illuminant metamerism between natural teeth and zirconia restorations evaluated with a chromatic adaptation transform

STATEMENT OF PROBLEM

Little is known about the effect of illuminant metamerism between natural teeth and zirconia restorations, despite their increasing clinical popularity.

PURPOSE

The purpose of this in vitro study was to compare illuminant metamerism between pairs of natural teeth and layered zirconia restorations and pairs of natural teeth and monolithic zirconia restorations under 10 different illuminants and analyze their metameric potential.

MATERIAL AND METHODS

Spectral reflectance factors were obtained from 10 pairs of extracted natural teeth and layered zirconia restorations and 28 pairs of extracted natural teeth and monolithic multilayer zirconia restorations. Each pair showed a color match that was within the visual threshold for clinical acceptability (CIEDE2000≤1.8). A special index of metamerism for the change of illuminant (Milm) was calculated from the CIEDE2000 color difference equation. Descriptive statistics and the one-sample t test were used to analyze the results for the Milm and for both groups of layered and monolithic zirconia restorations (α=.05).

RESULTS

Layered zirconia restorations reached a mean ±standard deviation value for Milm=0.3 ±0.2 and Milm=0.5 ±0.4 for monolithic zirconia restorations (P<.01).

CONCLUSIONS

The effect of illuminant metamerism between natural teeth and zirconia crowns was weak and generally within the clinical acceptability limit.

Little information loss with red-green color deficient vision in natural environments

Inherited color vision deficiency affects red-green discrimination in about one in twelve men from European populations. Its effects have been studied mainly in primitive foraging but also in detecting blushing and breaking camouflage. Yet there is no obvious relationship between these specific tasks and vision in the real world. The aim here was to quantify the impact of color vision deficiency by estimating computationally the information available to observers about colored surfaces in natural scenes. With representative independent sets of 50 and 100 hyperspectral images, estimated information was found to be only a little less in red-green color vision deficiency than in normal trichromacy. Colorimetric analyses revealed the importance of large lightness variations within scenes, small redness-greenness variations, and uneven frequencies of different colored surfaces. While red-green color vision deficiency poses challenges in some tasks, it has much less effect on gaining information from natural environments.

Discrimination of natural colors in anomalous trichromacy and the effects of EnChroma and Vino filters

It is still unclear how well anomalous trichromats discriminate natural colors and whether commercial spectral filters improve performance in these conditions. We show that anomalous trichromats have good color discrimination with colors drawn from natural environments. It is only about 14% poorer, on average, than normal trichromats in our sample of thirteen anomalous trichromats. No measurable effect of the filters on discrimination was found, even after 8 hours of continuous use. Computations of cone and post-receptoral signals show only a modest increase in medium-to-long-wavelength difference signals, which may explain the absent effect of the filters.

Synthetic hyperspectral array video database with applications to cross-spectral reconstruction and hyperspectral video coding

In this paper, a synthetic hyperspectral video database is introduced. Since it is impossible to record ground-truth hyperspectral videos, this database offers the possibility to leverage the evaluation of algorithms in diverse applications. For all scenes, depth maps are provided as well to yield the position of a pixel in all spatial dimensions as well as the reflectance in spectral dimension. Two novel algorithms for two different applications are proposed to prove the diversity of applications that can be addressed by this novel database. First, a cross-spectral image reconstruction algorithm is extended to exploit the temporal correlation between two consecutive frames. The evaluation using this hyperspectral database shows an increase in peak signal-to-noise ratio (PSNR) of up to 5.6 dB dependent on the scene. Second, a hyperspectral video coder is introduced, which extends an existing hyperspectral image coder by exploiting temporal correlation. The evaluation shows rate savings of up to 10% depending on the scene.

Sensory representation of surface reflectances: assessments with hyperspectral images

Specifying surface reflectances in a simple and perceptually informative way would be beneficial for many areas of research and application. We assessed whether a 3×3 matrix may be used to approximate how a surface reflectance modulates the sensory color signal across illuminants. We tested whether observers could discriminate between the model's approximate and accurate spectral renderings of hyperspectral images under narrowband and naturalistic, broadband illuminants for eight hue directions. Discriminating the approximate from the spectral rendering was possible with narrowband, but almost never with broadband illuminants. These results suggest that our model specifies the sensory information of reflectances across naturalistic illuminants with high fidelity, and with lower computational cost than spectral rendering.

Dataset artificial augmentation with a small number of training samples for reflectance estimation

The accuracy of the spectral reflectance estimation approaches highly depends on the amount, coverage, and representation of valid samples in the training dataset. We present a dataset artificial augmentation approach with a small number of actual training samples by light source spectra tuning. Then, the reflectance estimation process is carried out with our augmented color samples for commonly used datasets (IES, Munsell, Macbeth, Leeds). Finally, the impact of the augmented color sample number is investigated using different augmented color sample numbers. The results show that our proposed approach can artificially augment the color samples from CCSG 140 color samples to 13791 color samples and even more. The reflectance estimation performances with augmented color samples are much higher than with the benchmark CCSG datasets for all tested datasets (IES, Munsell, Macbeth, Leeds, as well as a real-scene hyperspectral reflectance database). It indicates that the proposed dataset augmentation approach is practical for improving the reflectance estimation performances.

The effect of a short-wave filtering contact lens on color appearance

We assessed the effect of a contact lens that filters short-wavelength (SW) visible light on color appearance. These effects were modeled and measured by direct comparison to a clear contact lens. Sixty-one subjects were enrolled, and 58 completed as cohort; 31 were 18 to 39 years old (mean ± SD, 29.6 ± 5.6), 27 were 40 to 65 years old (50.1 ± 8.1). A double-masked contralateral design was used; participants randomly wore a SW-filtering contact lens on one eye and a clear control lens on the other eye. Subjects then mixed three primaries (including a short-wave primary, strongly within the absorbance of the test lens) until a perceived perfect neutral white was achieved with each eye. Color appearance was quantified using chromaticity coordinates measured with a spectral radiometer within a custom-built tricolorimeter. Color vision in natural scenes was simulated using hyperspectral images and cone fundamentals based on a standard observer. Overall, the chromaticity coordinates of matches that were set using the SW-filtering contact lens (n = 58; x = 0.345, y = 0.325, u' = 0.222, v' = 0.470) and clear contact lens (n = 58; x = 0.344, y = 0.325, u' = 0.223, v' = 0.471) were not significantly different, regardless of age group. Simulations indicated that, for natural scenes, the SW-filtering contact lens that was evaluated changes L/(L+M) and S/(L+M) chromatic contrast by no more than -1.4% to +1.1% and -36.9% to +5.0%, respectively. Tricolorimetry was used to measure color appearance in subjects wearing a SW-filtering lens in one eye and a clear lens in the other, and the results indicate that imparting a subtle tint to a contact lens, as in the SW-filtering lens that was evaluated, does not alter color appearance for younger or older subjects. A model of color vision predicted little effect of the lens on chromatic contrast for natural scenes.

Feasibility of a Real-Time Embedded Hyperspectral Compressive Sensing Imaging System

Hyperspectral imaging has been attracting considerable interest as it provides spectrally rich acquisitions useful in several applications, such as remote sensing, agriculture, astronomy, geology and medicine. Hyperspectral devices based on compressive acquisitions have appeared recently as an alternative to conventional hyperspectral imaging systems and allow for data-sampling with fewer acquisitions than classical imaging techniques, even under the Nyquist rate. However, compressive hyperspectral imaging requires a reconstruction algorithm in order to recover all the data from the raw compressed acquisition. The reconstruction process is one of the limiting factors for the spread of these devices, as it is generally time-consuming and comes with a high computational burden. Algorithmic and material acceleration with embedded and parallel architectures (e.g., GPUs and FPGAs) can considerably speed up image reconstruction, making hyperspectral compressive systems suitable for real-time applications. This paper provides an in-depth analysis of the required performance in terms of computing power, data memory and bandwidth considering a compressive hyperspectral imaging system and a state-of-the-art reconstruction algorithm as an example. The results of the analysis show that real-time application is possible by combining several approaches, namely, exploitation of system matrix sparsity and bandwidth reduction by appropriately tuning data value encoding.

Spectral Reflectance Reconstruction of Organ Tissue Based on Metameric Black and Lattice Regression

In this study, a new approach is proposed for the restoration of reflectance information on organ samples using a commercial camera. This novel approach is comprised of three stages. In the first stage, a color clustering method is utilized to extract the representative colors of the organ samples as well as their corresponding spectral reflectance. In the second stage, the spectral reflectance is decomposed into two separate parts, i.e., the fundamental stimulus spectrum and the metameric black following the matrix-R theory, and the latter is further utilized to form a look-up table (LUT) via a lattice regression model. Finally, the reflectance information can be easily retrieved by referring to the newly built LUT. The performance of the proposed method was investigated, along with that of six other commonly adopted methods, through a physical experiment using real, measured organ samples. The results demonstrate that the proposed method outperformed all the other methods in terms of both colorimetric and spectral metrics, indicating that it is a promising strategy for organ sample reflectance restoration.

pHSCNN: CNN-based hyperspectral recovery from a pair of RGB images

To increase the fidelity of hyperspectral recovery from RGB images, we propose a pairwise-image-based hyperspectral convolutional neural network (pHSCNN) to recover hyperspectral images from a pair of RGB images, obtained by the same color sensor with and without an optical filter in front of the imaging lens. The proposed method avoids the pitfall of requiring multiple color sensors to obtain different RGB images and achieves higher accuracy than recovery from single RGB image. Besides, pHSCNN can also optimize the optical filter to further improve the performance. To experiment on real data, we built a dual-camera hyperspectral imaging system and created a real-captured hyperspectral-RGB dataset. Experimental results demonstrate the superiority of pHSCNN with the highest accuracy of the recovered hyperspectral signature perceptually and numerically.

NEUTRON: Neural particle swarm optimization for material-aware inverse design of structural color

Designing optical structures for generating structural colors is challenging because of the complex relationship between the optical structures and the color perceived by human eyes. Machine learning-based approaches have been developed to expedite this design process. However, existing methods solely focus on structural parameters of the optical design, which could lead to suboptimal color generation because of the inability to optimize the selection of materials. To address this issue, an approach known as Neural Particle Swarm Optimization is proposed in this paper. The proposed method achieves high design accuracy and efficiency on two structural color design tasks; the first task is designing environment-friendly alternatives to chrome coatings, and the second task concerns reconstructing pictures with multilayer optical thin films. Several designs that could replace chrome coatings have been discovered; pictures with more than 200,000 pixels and thousands of unique colors can be accurately reconstructed in a few hours.

Colour constancy failures expected in colourful environments

Colour constancy refers to the constant perceived or apparent colour of a surface despite changes in illumination spectrum. Laboratory measurements have often found it imperfect. The aim here was to estimate the frequency of constancy failures in natural outdoor environments and relate them to colorimetric surface properties. A computational analysis was performed with 50 hyperspectral reflectance images of outdoor scenes undergoing simulated daylight changes. For a chromatically adapted observer, estimated colour appearance changed noticeably for at least 5% of the surface area in 60% of scenes, and at least 10% of the surface area in 44% of scenes. Somewhat higher frequencies were found for estimated changes in perceived colour relations represented by spatial ratios of cone-photoreceptor excitations. These estimated changes correlated with surface chroma and saturation. Outdoors, the colour constancy of some individual surfaces seems likely to fail, particularly if those surfaces are colourful.

Spectral-Spatial Boundary Detection in Hyperspectral Images

In this paper, we propose a novel method for boundary detection in close-range hyperspectral images. This method can effectively predict the boundaries of objects of similar colour but different materials. To effectively extract the material information in the image, the spatial distribution of the spectral responses of different materials or endmembers is first estimated by hyperspectral unmixing. The resulting abundance map represents the fraction of each endmember spectra at each pixel. The abundance map is used as a supportive feature such that the spectral signature and the abundance vector for each pixel are fused to form a new spectral feature vector. Then different spectral similarity measures are adopted to construct a sparse spectral-spatial affinity matrix that characterizes the similarity between the spectral feature vectors of neighbouring pixels within a local neighborhood. After that, a spectral clustering method is adopted to produce eigenimages. Finally, the boundary map is constructed from the most informative eigenimages. We created a new HSI dataset and use it to compare the proposed method with four alternative methods, one for hyperspectral image and three for RGB image. The results exhibit that our method outperforms the alternatives and can cope with several scenarios that methods based on colour images cannot handle.

Luminosity thresholds of colored surfaces are determined by their upper-limit luminances empirically internalized in the visual system

We typically have a fairly good idea whether a given object is self-luminous or illuminated, but it is not fully understood how we make this judgment. This study aimed to identify determinants of the luminosity threshold, a luminance level at which a surface begins to appear self-luminous. We specifically tested a hypothesis that our visual system knows the maximum luminance level that a surface can reach under the physical constraint that a surface cannot reflect more light than any incident light and applies this prior to determine the luminosity thresholds. Observers were presented with a 2-degree circular test field surrounded by numerous overlapping colored circles and luminosity thresholds were measured as a function of (i) the chromaticity of the test field, (ii) the shape of surrounding color distribution, and (iii) the color of the illuminant of the surrounding colors. We found that the luminosity thresholds peaked around the chromaticity of test illuminants and decreased as the purity of the test chromaticity increased. However, the loci of luminosity thresholds across chromaticities were nearly invariant to the shape of the surrounding color distribution and generally resembled the loci drawn from theoretical upper-limit luminances and upper-limit luminance boundaries of real objects. These trends were particularly evident for illuminants on the black-body locus and did not hold well under atypical illuminants, such as magenta or green. These results support the idea that our visual system empirically internalizes the gamut of surface colors under natural illuminants and a given object appears self-luminous when its luminance exceeds this internalized upper-limit luminance.

Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision

In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists' eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at materialsinpaintings.tudelft.nl. We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.

Naturalness and aesthetics of colors - Preference for color compositions perceived as natural

What makes a colored image, e.g. an abstract painting or a landscape, look pleasing? We hypothesized that a preference for complex color compositions, such as paintings and images of natural scenes, might be related to how natural the colors are perceived. We tested this possibility with two experiments in which the degree of naturalness of images was manipulated by rotating their color gamut rigidly in the color space CIELAB. This changed just the hue composition, but preserved saturation and lightness. In the first experiment we obtained individual scaling curves for perceived naturalness and for preference as a function of the angle of gamut rotation for a small set of images. The naturalness and preference scaling curves were found to be largely similar and their maxima were close to the original image. In the second experiment, we tested whether this effect generalized to a larger set of images. We used a simultaneous 5AFC procedure where in each trial participants had to select the most natural or the most preferred image from five different rotations of the color gamut. The results confirmed the first experiment and showed that, in general, the images perceived as the more natural tend to be the ones that are preferred. Together these results show that perceived naturalness and preference are indeed perceptually closely related and may be driven by related mechanisms.

Fluctuating environmental light limits number of surfaces visually recognizable by colour

Small changes in daylight in the environment can produce large changes in reflected light, even over short intervals of time. Do these changes limit the visual recognition of surfaces by their colour? To address this question, information-theoretic methods were used to estimate computationally the maximum number of surfaces in a sample that can be identified as the same after an interval. Scene data were taken from successive hyperspectral radiance images. With no illumination change, the average number of surfaces distinguishable by colour was of the order of 10,000. But with an illumination change, the average number still identifiable declined rapidly with change duration. In one condition, the number after two minutes was around 600, after 10 min around 200, and after an hour around 70. These limits on identification are much lower than with spectral changes in daylight. No recoding of the colour signal is likely to recover surface identity lost in this uncertain environment.

Hyperspectral Recovery from RGB Images using Gaussian Processes

We propose to recover spectral details from RGB images of known spectral quantization by modeling natural spectra under Gaussian Processes and combining them with the RGB images. Our technique exploits Process Kernels to model the relative smoothness of reflectance spectra, and encourages non-negativity in the resulting signals for better estimation of the reflectance values. The Gaussian Processes are inferred in sets using clusters of spatio-spectrally correlated hyperspectral training patches. Each set is transformed to match the spectral quantization of the test RGB image. We extract overlapping patches from the RGB image and match them to the hyperspectral training patches by spectrally transforming the latter. The RGB patches are encoded over the transformed Gaussian Processes related to those hyperspectral patches and the resulting image is constructed by combining the codes with the original processes. Our approach infers the desired Gaussian Processes under a fully Bayesian model inspired by Beta-Bernoulli Process, for which we also present the inference procedure. A thorough evaluation using three hyperspectral datasets demonstrates the effective extraction of spectral details from RGB images by the proposed technique.

Hyperspectral Pansharpening Based on Spectral Constrained Adversarial Autoencoder

Hyperspectral (HS) imaging is conducive to better describing and understanding the subtle differences in spectral characteristics of different materials due to sufficient spectral information compared with traditional imaging systems. However, it is still challenging to obtain high resolution (HR) HS images in both the spectral and spatial domains. Different from previous methods, we first propose spectral constrained adversarial autoencoder (SCAAE) to extract deep features of HS images and combine with the panchromatic (PAN) image to competently represent the spatial information of HR HS images, which is more comprehensive and representative. In particular, based on the adversarial autoencoder (AAE) network, the SCAAE network is built with the added spectral constraint in the loss function so that spectral consistency and a higher quality of spatial information enhancement can be ensured. Then, an adaptive fusion approach with a simple feature selection rule is induced to make full use of the spatial information contained in both the HS image and PAN image. Specifically, the spatial information from two different sensors is introduced into a convex optimization equation to obtain the fusion proportion of the two parts and estimate the generated HR HS image. By analyzing the results from the experiments executed on the tested data sets through different methods, it can be found that, in CC, SAM, and RMSE, the performance of the proposed algorithm is improved by about 1.42%, 13.12%, and 29.26% respectively on average which is preferable to the well-performed method HySure. Compared to the MRA-based method, the improvement of the proposed method in in the above three indexes is 17.63%, 0.83%, and 11.02%, respectively. Moreover, the results are 0.87%, 22.11%, and 20.66%, respectively, better than the PCA-based method, which fully illustrated the superiority of the proposed method in spatial information preservation. All the experimental results demonstrate that the proposed method is superior to the state-of-the-art fusion methods in terms of subjective and objective evaluations.

Hyperspectral environmental illumination maps: characterizing directional spectral variation in natural environments

Objects placed in real-world scenes receive incident light from every direction, and the spectral content of this light may vary from one direction to another. In computer graphics, environmental illumination is approximated using maps that specify illumination at a point as a function of incident angle. However, to-date, existing public databases of environmental illumination specify only three colour channels (RGB). We have captured a new set of 12 environmental illumination maps (eight outdoor scenes; four indoor scenes) using a hyperspectral imaging system with 33 spectral channels. The data reveal a striking directional variation of spectral distribution of lighting in natural environments. We discuss limitations of using daylight models to describe natural environmental illumination.

A low-cost hyperspectral scanner for natural imaging and the study of animal colour vision above and under water

Hyperspectral imaging is a widely used technology for industrial and scientific purposes, but the high cost and large size of commercial setups have made them impractical for most basic research. Here, we designed and implemented a fully open source and low-cost hyperspectral scanner based on a commercial spectrometer coupled to custom optical, mechanical and electronic components. We demonstrate our scanner's utility for natural imaging in both terrestrial and underwater environments. Our design provides sub-nm spectral resolution between 350-950 nm, including the UV part of the light spectrum which has been mostly absent from commercial solutions and previous natural imaging studies. By comparing the full light spectra from natural scenes to the spectral sensitivity of animals, we show how our system can be used to identify subtle variations in chromatic details detectable by different species. In addition, we have created an open access database for hyperspectral datasets collected from natural scenes in the UK and India. Together with comprehensive online build- and use-instructions, our setup provides an inexpensive and customisable solution to gather and share hyperspectral imaging data.

Compressive spectral imaging via deformable mirror and colored-mosaic detector

Compressive Spectral Imaging (CSI) is an emerging technology that aims at reconstructing a spectral image from a limited set of two-dimensional projections. To capture these projections, CSI architectures often combine light dispersive elements with coded apertures or programmable spatial light modulators. This work introduces a novel and compact CSI architecture based on a deformable mirror and a colored-filter detector to produce compressive spatio-spectral projections without the need of a grating or prism. Alongside, we propose a tensor-based reconstruction algorithm to recover the spatial-spectral information from the compressed measurements. Experimental results on both simulated and real datasets demonstrate efficacy of the proposed acquisition architecture and the especially crafted inversion algorithms.

Tensor Completion via Nonlocal Low-Rank Regularization

Tensor completion (TC), aiming to recover original high-order data from its degraded observations, has recently drawn much attention in hyperspectral images (HSIs) domain. Generally, the widely used TC methods formulate the rank minimization problem with a convex trace norm penalty, which shrinks all singular values equally, and may generate a much biased solution. Besides, these TC methods assume the whole high-order data is of low-rank, which may fail to recover the detail information in high-order data with diverse and complex structures. In this paper, a novel nonlocal low-rank regularization-based TC (NLRR-TC) method is proposed for HSIs, which includes two main steps. In the first step, an initial completion result is generated by the proposed low-rank regularization-based TC (LRR-TC) model, which combines the logarithm of the determinant with the tensor trace norm. This model can more effectively approximate the tensor rank, since the logarithm function values can be adaptively tuned for each input. In the second step, the nonlocal spatial-spectral similarity is integrated into the LRR-TC model, to obtain the final completion result. Specifically, the initial completion result is first divided into groups of nonlocal similar cubes (each group forms a 3-D tensor), and then the LRR-TC is applied to each group. Since similar cubes within each group contain similar structures, each 3-D tensor should have low-rank property, and thus further improves the completion result. Experimental results demonstrate that the proposed NLRR-TC method outperforms state-of-the-art HSIs completion techniques.

Form vision from melanopsin in humans

Detection and discrimination of spatial patterns is thought to originate with photoreception by rods and cones. Here, we investigated whether the inner-retinal photoreceptor melanopsin could represent a third origin for form vision. We developed a 4-primary visual display capable of presenting patterns differing in contrast for melanopsin vs cones, and generated spectrally distinct stimuli that were indistinguishable for cones (metamers) but presented contrast for melanopsin. Healthy observers could detect sinusoidal gratings formed by these metamers when presented in the peripheral retina at low spatial (≤0.8 cpd) and temporal (≤0.45 Hz) frequencies, and Michelson contrasts ≥14% for melanopsin. Metameric gratings became invisible at lower light levels (<10¹³ melanopsin photons cm⁻² sr⁻¹ s⁻¹) when rods are more active. The addition of metameric increases in melanopsin contrast altered appearance of greyscale representations of coarse gratings and a range of everyday images. These data identify melanopsin as a new potential origin for aspects of spatial vision in humans.

The perception of spatial patterns (form vision) is thought to rely on rod and cone cells in the retina. Here, the authors show that a third kind of retinal cell, melanopsin-expressing ganglion cells, can also detect form in humans, under particular conditions.

Hyperspectral imaging in color vision research: tutorial

This tutorial offers an introduction to terrestrial and close-range hyperspectral imaging and some of its uses in human color vision research. The main types of hyperspectral cameras are described together with procedures for image acquisition, postprocessing, and calibration for either radiance or reflectance data. Image transformations are defined for colorimetric representations, color rendering, and cone receptor and postreceptor coding. Several example applications are also presented. These include calculating the color properties of scenes, such as gamut volume and metamerism, and analyzing the utility of color in observer tasks, such as identifying surfaces under illuminant changes. The effects of noise and uncertainty are considered in both image acquisition and color vision applications.

The colors of natural scenes benefit dichromats

Dichromacy impairs color vision and impoverishes the discrimination of surface colors in natural scenes. Computational estimates based on hyperspectral imaging data from natural scenes suggest that dichromats can discriminate only about 10% of the colors discriminated by normal trichromats. These estimates, however, assume that the colors are equally frequent. Yet, pairs of colors confused by dichromats may be rare and thus have small impact on overall perceived chromatic diversity. This study estimated, empirically, how much dichromats are disadvantaged in discriminating surface colors drawn from natural scenes. The stimulus for the experiment was a scene made of real three-dimensional objects painted with matte white paint and illuminated by a spectrally tunable light source. In each trial the observers saw the scene illuminated by two spectra in two successive time intervals and had to indicate whether the colors perceived in the objects in the two intervals were the same or different. The spectra were drawn randomly from hyperspectral data of natural scenes and therefore represented natural spectral statistics. Four normal trichromats and four dichromats carried out the experiment. It was found that the number of pairs that could be discriminated by dichromats was almost 70% of those discriminated by normal trichromats, a proportion much higher than anticipated from estimates of discernible colors. Moreover, data from model simulations show that normal trichromats and dichromats use lightness differences for discrimination in about 40% and 50% of the discriminable pairs, respectively. Together these results suggest that the color distributions of natural scenes benefit the color vision of dichromats.

Integration of point cloud data and hyperspectral imaging as a data gathering methodology for refurbishment projects using building information modelling (BIM)

Purpose

Building information modelling (BIM) is a digital representation of the physical and functional characteristics of a building. Its use offers a range of benefits in terms of achieving the efficient design, construction, operation and maintenance of buildings. Applying BIM at the outset of a new build project should be relatively easy. However, it is often problematic to apply BIM techniques to an existing building, for example, as part of a refurbishment project or as a tool supporting the facilities management strategy, because of inadequacies in the previous management of the dataset that characterises the facility in question. These inadequacies may include information on as built geometry and materials of construction. By the application of automated retrospective data gathering for use in BIM, such problems should be largely overcome and significant benefits in terms of efficiency gains and cost savings should be achieved.

Design/methodology/approach

Laser scanning can be used to collect geometrical and spatial information in the form of a 3D point cloud, and this technique is already used. However, as a point cloud representation does not contain any semantic information or geometrical context, such point cloud data must refer to external sources of data, such as building specification and construction materials, to be in used in BIM.

Findings

Hyperspectral imaging techniques can be applied to provide both spectral and spatial information of scenes as a set of high-resolution images. Integrating of a 3D point cloud into hyperspectral images would enable accurate identification and classification of surface materials and would also convert the 3D representation to BIM.

Originality/value

This integrated approach has been applied in other areas, for example, in crop management. The transfer of this approach to facilities management and construction would improve the efficiency and automation of the data transition from building pathology to BIM. In this study, the technological feasibility and advantages of the integration of laser scanning and hyperspectral imaging (the latter not having previously been used in the construction context in its own right) is discussed, and an example of the use of a new integration technique is presented, applied for the first time in the context of buildings.

Fast High-Dimensional Bilateral and Nonlocal Means Filtering

Existing fast algorithms for bilateral and nonlocal means filtering mostly work with grayscale images. They cannot easily be extended to high-dimensional data such as color and hyperspectral images, patch-based data, flow-fields, etc. In this paper, we propose a fast algorithm for high-dimensional bilateral and nonlocal means filtering. Unlike existing approaches, where the focus is on approximating the data (using quantization) or the filter kernel (via analytic expansions), we locally approximate the kernel using weighted and shifted copies of a Gaussian, where the weights and shifts are inferred from the data. The algorithm emerging from the proposed approximation essentially involves clustering and fast convolutions, and is easy to implement. Moreover, a variant of our algorithm comes with a guarantee (bound) on the approximation error, which is not enjoyed by existing algorithms.We present some results for high-dimensional bilateral and nonlocal means filtering to demonstrate the speed and accuracy of our proposal. Moreover, we also show that our algorithm can outperform state-of-the-art fast approximations in terms of accuracy and timing.

Illumination separation of non-Lambertian scenes from a single hyperspectral image

In this paper, we propose a general framework to estimate the spectrum of the illumination from global specular information in a single hyperspectral image. By utilizing the specular independent subspace, we iteratively separate the reflectance components and shape a weight scheme in order to find specular-contaminated pixels. After that, the illumination can be directly estimated by factorizing the weighted specular-contaminated pixels. The proposed method enables a direct and effective decomposition of the illumination and reflectance components from a single hyperspectral image. We demonstrate the robustness and accuracy of our method on simulation and real experiments. Moreover, we capture a hyperspectral image dataset with ground-truth illumination to quantitative compare the performance.

Color Perception: Objects, Constancy, and Categories

Color has been scientifically investigated by linking color appearance to colorimetric measurements of the light that enters the eye. However, the main purpose of color perception is not to determine the properties of incident light, but to aid the visual perception of objects and materials in our environment. We review the state of the art on object colors, color constancy, and color categories to gain insight into the functional aspects of color perception. The common ground between these areas of research is that color appearance is tightly linked to the identification of objects and materials and the communication across observers. In conclusion, we argue that research should focus on how color processing is adapted to the surface properties of objects in the natural environment in order to bridge the gap between the known early stages of color perception and the subjective appearance of color.

HyTexiLa: High Resolution Visible and Near Infrared Hyperspectral Texture Images

We present a dataset of close range hyperspectral images of materials that span the visible and near infrared spectrums: HyTexiLa (Hyperspectral Texture images acquired in Laboratory). The data is intended to provide high spectral and spatial resolution reflectance images of 112 materials to study spatial and spectral textures. In this paper we discuss the calibration of the data and the method for addressing the distortions during image acquisition. We provide a spectral analysis based on non-negative matrix factorization to quantify the spectral complexity of the samples and extend local binary pattern operators to the hyperspectral texture analysis. The results demonstrate that although the spectral complexity of each of the textures is generally low, increasing the number of bands permits better texture classification, with the opponent band local binary pattern feature giving the best performance.

Adaptive filter design via a gradient thresholding algorithm for compressive spectral imaging

Sensing a spectral image data cube has traditionally been a time-consuming task since it requires a scanning process. In contrast, compressive spectral imaging (CSI) has attracted widespread interest since it requires fewer samples than scanning systems to acquire the data cube, thus improving the sensing speed. CSI captures linear projections of the scene, and then a reconstruction algorithm estimates the underlying scene. One notable CSI architecture is the color coded aperture snapshot spectral imager (C-CASSI), which employs pixelated filter arrays as the coding patterns to spatially and spectrally encode the incoming light. Up to date works on C-CASSI have used non-adaptive color coded apertures. Non-adaptive sampling ignores prior information about the signal to design the coding patterns. Therefore, this work proposes a method to adaptively design the color coded aperture, such that the quality of image reconstruction is improved. In more detail, this work introduces a gradient thresholding algorithm, which computes the consecutive color coded aperture from a rapidly reconstructed low-resolution version of the data cube. The successive adaptive patterns enable recovering a data cube in the presence of Gaussian noise with higher image quality. Real reconstructions and simulations evidence an improvement of up to 3 dB in the quality of image reconstruction of the proposed method in comparison with state-of-the-art non-adaptive techniques.

Image quality degradation of object-color metamer mismatching in digital camera color reproduction

Metamer mismatching is a phenomenon where two objects that are colorimetrically indistinguishable under one lighting condition become distinguishable under another one. Due to the unavailability of spectral information, metamer mismatching introduces an inherent uncertainty into cameras' color reproduction. To investigate the degree of image quality degradation by the metamer mismatching, a large spectral reflectance database was compiled in this study to search the object-color metamers sets of the spectra in hyperspectral images. Then, metamer-degraded images were constructed and compared with the ground truth images by directional statistics-based color similarity index image quality assessment metrics to evaluate the perceptual image degradation. The results indicate that the object-color metamer mismatching has only little impact on the image quality degradation, whereas the inappropriate selection of color correction matrices involved with the illumination metamerism is the primary factor for the accuracy decrease in the digital camera color reproduction.

Hyperspectral database of fruits and vegetables

We have built a hyperspectral database of 42 fruits and vegetables. Both the outside (skin) and inside of the objects were imaged. We used a Specim VNIR HS-CL-30-V8E-OEM mirror-scanning hyperspectral camera and took pictures at a spatial resolution of ∼57  px/deg by 800 pixels at a wavelength resolution of ∼1.12  nm. A stable, broadband illuminant was used. Images and software are freely available on our webserver (http://www.allpsych.uni-giessen.de/GHIFVD; pronounced "gift"). We performed two kinds of analyses on these images. First, when comparing the insides and outsides of the objects, we observed that the insides were lighter than the skins, and that the hues of the insides and skins were significantly correlated (circular correlation=0.638). Second, we compared the color distribution within each object to corresponding human color discrimination thresholds. We found a significant correlation (0.75) between the orientation of ellipses fit to the chromaticity distributions of our fruits and vegetables with the orientations of interpolated MacAdam discrimination ellipses. This indicates a close relationship between sensory processing and the characteristics of environmental objects.

The Verriest Lecture: Color vision in an uncertain world

The natural world is optically unconstrained. Surface properties may vary from one point to another, and reflected light may vary from one instant to the next. The aim of this work is to quantify some of the physical failures of color vision performance that result from uncertainty. In computational simulations with images of vegetated and nonvegetated outdoor scenes, it is shown that color provides an unreliable guide to surface identity. It is also shown that changes in illuminant may cause colors to no longer match and the relations between individual colors to vary. These failures are generally well described by a measure of the randomness of the colors in scenes, the Shannon entropy. Although uncertainty is intrinsic to the environment, its consequences for color vision can be predicted.

Color metamerism and the structure of illuminant space

The colors of two surfaces might appear exactly alike under one illuminant while varying under others. This is due to the metamerism phenomenon in which physically distinct reflectance spectra result in identical cone photoreceptor excitations. The existence of such metameric pairs can potentially cause great ambiguities for our visual perception by challenging phenomena such as color constancy. We investigated frequency and magnitude of metamerism in a wide range of scenarios by studying a large set of surface reflectance spectra illuminated under numerous natural and artificial sources of light. Our results extend previous studies in the literature by demonstrating that metamers are indeed relatively infrequent. Potentially troublesome cases in which two surfaces with an identical color under one illuminant appear very differently under a second illuminant are exceedingly rare. We used the frequency of metameric pairs in combination with non-metric multidimensional scaling to establish a new representation of illuminants based on metamerism. This approach imposes a systematic structure onto the representation of illuminants and allows to better prognosticate the likelihood of metamers under new illuminants.

Color constancy of color reproductions in art paintings

Popular color reproductions of art paintings such as postcards are intended to remind viewers of the original works. It is, however, unclear how well the quality of the reproductions is preserved under various illuminations. Color constancy of the reproductions in relation to colors in the original paintings was estimated computationally with hyperspectral images of 15th-century Flemish paintings, 20th-century modern abstract paintings, and their corresponding postcards with a series of illuminants: the CIE daylight D65 with correlated color temperature (CCT) 6500 K, daylight D40, fluorescent lamps F2 and F11, and a LED lamp designed for museums with CCT approximately 3500-4000 K. Despite large colorimetric differences between the types of art paintings and between the illuminants simulated, local areas showed good color constancy: skin areas in the Flemish paintings ranged from 0.76 to 0.81, whereas nonskin areas ranged from 0.19 to 0.68. This result suggests that viewers may be able to achieve color constancy with the reproduction postcards by disregarding inconsistent colors representations from the original paintings caused by changes in illumination conditions.