Hyperspectral imaging in color vision research: tutorial

Hyperspectral Imaging Database of Human Facial Skin

The perceived color of human skin is the result of the interaction of environmental lighting with the skin. Only by resorting to human skin spectral reflectance, it is possible to obtain physical outcomes of this interaction. The purpose of this work was to provide a cured and validated database of hyperspectral images of human faces, useful for several applications, such as psychophysics-based research, object recognition, and material modeling. The hyperspectral imaging data from 29 human faces with different skin tones and sexes, under constant lighting and controlled movements, were described and characterized. Each hyperspectral image, which comprised spectral reflectance of the whole face from 400 to 720 nm in 10 nm steps at each pixel, was analyzed between and within nine facial positions located at different areas of the face. Simultaneously, spectral measurements at the same nine facial positions using conventional local point and/or contact devices were used to ascertain the data. It was found that the spectral reflectance profile changed between skin tones, subjects, and facial locations. Important local variations of the spectral reflectance profile showed that extra care is needed when considering average values from conventional devices at the same area of measurement.

Colour expectations across illumination changes

This study investigates human expectations towards naturalistic colour changes under varying illuminations. Understanding colour expectations is key to both scientific research on colour constancy and applications of colour and lighting in art and industry. We reanalysed data from asymmetric colour matches of a previous study and found that colour adjustments tended to align with illuminant-induced colour shifts predicted by naturalistic, rather than artificial, illuminants and reflectances. We conducted three experiments using hyperspectral images of naturalistic scenes to test if participants judged colour changes based on naturalistic illuminant and reflectance spectra as more plausible than artificial ones, which contradicted their expectations. When we consistently manipulated the illuminant (Experiment 1) and reflectance (Experiment 2) spectra across the whole scene, observers chose the naturalistic renderings significantly above the chance level (>25 %) but barely more often than any of the three artificial ones, collectively (>50 %). However, when we manipulated only one object/area's reflectance (Experiment 3), observers more reliably identified the version in which the object had a naturalistic reflectance like the rest of the scene. Results from Experiments 2-3 and additional analyses suggested that relational colour constancy strongly contributed to observer expectations, and stable cone-excitation ratios are not limited to naturalistic illuminants and reflectances but also occur for our artificial renderings. Our findings indicate that relational colour constancy and prior knowledge about surface colour shifts help to disambiguate surface colour identity under illumination changes, enabling human observers to recognise surface colours reliably in naturalistic conditions. Additionally, relational colour constancy may even be effective in many artificial conditions.

How the orientation of the color gamut of natural scenes influences color discrimination in red-green dichromacy

In natural scenes, visual discrimination of colored surfaces by individuals with X-linked dichromacy is known to be only a little poorer than in normal trichromacy. This surprising result may be related to the properties of the colors of these scenes, like the shape and orientation of the color gamut, uneven frequency, and a considerable variation in lightness. It is unclear, however, how much each of these factors contributes to the small impairment in discrimination, in particular, what is the contribution of the orientation of the gamut. We measured the discrimination of colors from natural scenes by six normal trichromats and six dichromats. Colors were drawn either from the original color gamut of the scenes or from gamut-rotated versions of the scenes. Pairs of colors were randomly drawn from hyperspectral images of one rural and one urban environment and presented on a screen. As expected, dichromats were only a little poorer than normal trichromats at discrimination but the disadvantage varied systematically with the orientation of the color gamut by a factor of about three with a minimum around a yellow-green axis. Dichromats also took longer to respond, and the response times were modulated with the orientation of the color gamut in a similar way as the loss in discrimination. For the scenes tested here, these results imply an important impact of the orientation of the gamut on discrimination. They also indicate that the predominantly yellow-blue orientation of the gamut of natural scene might not be optimal for discrimination in dichromacy.

Misidentifying illuminant changes in natural scenes due to failures in relational colour constancy

The colours of surfaces in a scene may not appear constant with a change in the colour of the illumination. Yet even when colour constancy fails, human observers can usually discriminate changes in lighting from changes in surface reflecting properties. This operational ability has been attributed to the constancy of perceived colour relations between surfaces under illuminant changes, in turn based on approximately invariant spatial ratios of cone photoreceptor excitations. Natural deviations in these ratios may, however, lead to illuminant changes being misidentified. The aim of this work was to test whether such misidentifications occur with natural scenes and whether they are due to failures in relational colour constancy. Pairs of scene images from hyperspectral data were presented side-by-side on a computer-controlled display. On one side, the scene underwent illuminant changes and on the other side, it underwent the same changes but with images corrected for any residual deviations in spatial ratios. Observers systematically misidentified the corrected images as being due to illuminant changes. The frequency of errors increased with the size of the deviations, which were closely correlated with the estimated failures in relational colour constancy.

Little information loss with red-green color deficient vision in natural environments

Inherited color vision deficiency affects red-green discrimination in about one in twelve men from European populations. Its effects have been studied mainly in primitive foraging but also in detecting blushing and breaking camouflage. Yet there is no obvious relationship between these specific tasks and vision in the real world. The aim here was to quantify the impact of color vision deficiency by estimating computationally the information available to observers about colored surfaces in natural scenes. With representative independent sets of 50 and 100 hyperspectral images, estimated information was found to be only a little less in red-green color vision deficiency than in normal trichromacy. Colorimetric analyses revealed the importance of large lightness variations within scenes, small redness-greenness variations, and uneven frequencies of different colored surfaces. While red-green color vision deficiency poses challenges in some tasks, it has much less effect on gaining information from natural environments.

Design of non-Gaussian multispectral shortwave infrared filters assessed by surface spectral reflectances on the ECOSTRESS library

This paper addresses the multispectral filter design problem for spectral ranges where a viewing subspace is not defined. The methodology of color filter design is extended to this case, which allows the optimization of custom filter transmittance that meets the physical constraints of available fabrication methods. Multispectral shortwave infrared filters are then designed for two scenarios: spectral reconstruction and false-color representation. The Monte Carlo method is used to verify the filter performance degradation due to deviations in fabrication. The results obtained indicate that the proposed method is useful for designing multispectral filters to be fabricated using generic processes without any additional constraints.

Iridescence from Total Internal Reflection at 3D Microscale Interfaces: Mechanistic Insights and Spectral Analysis

An experimental investigation and the optical modeling of the structural coloration produced from total internal reflection interference within 3D microstructures are described. Ray-tracing simulations coupled with color visualization and spectral analysis techniques are used to model, examine, and rationalize the iridescence generated for a range of microgeometries, including hemicylinders and truncated hemispheres, under varying illumination conditions. An approach to deconstruct the observed iridescence and complex far-field spectral features into its elementary components and systematically link them to ray trajectories that emanate from the illuminated microstructures is demonstrated. The results are compared with experiments, wherein microstructures are fabricated with methods such as chemical etching, multiphoton lithography, and grayscale lithography. Microstructure arrays patterned on surfaces with varying orientation and size lead to unique color-traveling optical effects and highlight opportunities for how total internal reflection interference can be used to create customizable reflective iridescence. The findings herein provide a robust conceptual framework for rationalizing this multibounce interference mechanism and establish approaches for characterizing and tailoring the optical and iridescent properties of microstructured surfaces.

Sensory representation of surface reflectances: assessments with hyperspectral images

Specifying surface reflectances in a simple and perceptually informative way would be beneficial for many areas of research and application. We assessed whether a 3×3 matrix may be used to approximate how a surface reflectance modulates the sensory color signal across illuminants. We tested whether observers could discriminate between the model's approximate and accurate spectral renderings of hyperspectral images under narrowband and naturalistic, broadband illuminants for eight hue directions. Discriminating the approximate from the spectral rendering was possible with narrowband, but almost never with broadband illuminants. These results suggest that our model specifies the sensory information of reflectances across naturalistic illuminants with high fidelity, and with lower computational cost than spectral rendering.

Information gains from commercial spectral filters in anomalous trichromacy

Red-green color discrimination is compromised in anomalous trichromacy, the most common inherited color vision deficiency. This computational analysis tested whether three commercial optical filters with medium-to-long-wavelength stop bands increased information about colored surfaces. The surfaces were sampled from 50 hyperspectral images of outdoor scenes. At best, potential gains in the effective number of surfaces discriminable solely by color reached 9% in protanomaly and 15% in deuteranomaly, much less than with normal trichromacy. Gains were still less with lower scene illumination and more severe color vision deficiency. Stop-band filters may offer little improvement in objective real-world color discrimination.

Colour constancy failures expected in colourful environments

Colour constancy refers to the constant perceived or apparent colour of a surface despite changes in illumination spectrum. Laboratory measurements have often found it imperfect. The aim here was to estimate the frequency of constancy failures in natural outdoor environments and relate them to colorimetric surface properties. A computational analysis was performed with 50 hyperspectral reflectance images of outdoor scenes undergoing simulated daylight changes. For a chromatically adapted observer, estimated colour appearance changed noticeably for at least 5% of the surface area in 60% of scenes, and at least 10% of the surface area in 44% of scenes. Somewhat higher frequencies were found for estimated changes in perceived colour relations represented by spatial ratios of cone-photoreceptor excitations. These estimated changes correlated with surface chroma and saturation. Outdoors, the colour constancy of some individual surfaces seems likely to fail, particularly if those surfaces are colourful.

Visible Light Spectrum Extraction from Diffraction Images by Deconvolution and the Cepstrum

Accurate color determination in variable lighting conditions is difficult and requires special devices. We considered the task of extracting the visible light spectrum using ordinary camera sensors, to facilitate low-cost color measurements using consumer equipment. The approach uses a diffractive element attached to a standard camera and a computational algorithm for forming the light spectrum from the resulting diffraction images. We present two machine learning algorithms for this task, based on alternative processing pipelines using deconvolution and cepstrum operations, respectively. The proposed methods were trained and evaluated on diffraction images collected using three cameras and three illuminants to demonstrate the generality of the approach, measuring the quality by comparing the recovered spectra against ground truth measurements collected using a hyperspectral camera. We show that the proposed methods are able to reconstruct the spectrum, and, consequently, the color, with fairly good accuracy in all conditions, but the exact accuracy depends on the specific camera and lighting conditions. The testing procedure followed in our experiments suggests a high degree of confidence in the generalizability of our results; the method works well even for a new illuminant not seen in the development phase.

Leveraging colour-based pseudo-labels to supervise saliency detection in hyperspectral image datasets

Saliency detection mimics the natural visual attention mechanism that identifies an imagery region to be salient when it attracts visual attention more than the background. This image analysis task covers many important applications in several fields such as military science, ocean research, resources exploration, disaster and land-use monitoring tasks. Despite hundreds of models have been proposed for saliency detection in colour images, there is still a large room for improving saliency detection performances in hyperspectral imaging analysis. In the present study, an ensemble learning methodology for saliency detection in hyperspectral imagery datasets is presented. It enhances saliency assignments yielded through a robust colour-based technique with new saliency information extracted by taking advantage of the abundance of spectral information on multiple hyperspectral images. The experiments performed with the proposed methodology provide encouraging results, also compared to several competitors.

Naturalness and aesthetics of colors - Preference for color compositions perceived as natural

What makes a colored image, e.g. an abstract painting or a landscape, look pleasing? We hypothesized that a preference for complex color compositions, such as paintings and images of natural scenes, might be related to how natural the colors are perceived. We tested this possibility with two experiments in which the degree of naturalness of images was manipulated by rotating their color gamut rigidly in the color space CIELAB. This changed just the hue composition, but preserved saturation and lightness. In the first experiment we obtained individual scaling curves for perceived naturalness and for preference as a function of the angle of gamut rotation for a small set of images. The naturalness and preference scaling curves were found to be largely similar and their maxima were close to the original image. In the second experiment, we tested whether this effect generalized to a larger set of images. We used a simultaneous 5AFC procedure where in each trial participants had to select the most natural or the most preferred image from five different rotations of the color gamut. The results confirmed the first experiment and showed that, in general, the images perceived as the more natural tend to be the ones that are preferred. Together these results show that perceived naturalness and preference are indeed perceptually closely related and may be driven by related mechanisms.

A new transformation of cone responses to opponent color responses

It is widely agreed that the color vision process moves quickly from cone receptors to opponent color cells in the retina and lateral geniculate nucleus. Many workers have proposed the transformation or coding of long, medium, short (LMS) cone responses to r - g, y - b opponent color chromatic responses (unique hues) on the following basis: That L, M, S cones represent Red, Green, and Blue hues, with Yellow represented by (L + M), while r - g and y - b represent the opponent pairs of unique hues. The traditional coding from cones to opponent colors is that L - M gives r - g, while (L + M) - S gives y - b. This convention is open to several criticisms, and a new coding is required. A literature search produced 16 studies of cone responses LMS and 15 studies of spectral (i.e., ygb) opponent color chromatic responses, in terms of response wavelength peaks. Comparative analysis of the two sets of studies shows the means are almost identical (within 3 nm; i.e., L = y, M = g, S = b). Further, the response curves of LMS are very similar shapes to ygb. In sum, each set can directly transform to the other on this proposed coding: (S + L) - M gives r - g, while L - S gives y - b. This coding activates neural operations in the cardinal directions r - g and y - b.

Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation

Fruit maturity is a critical factor in the supply chain, consumer preference, and agriculture industry. Most classification methods on fruit maturity identify only two classes: ripe and unripe, but this paper estimates six maturity stages of papaya fruit. Deep learning architectures have gained respect and brought breakthroughs in unimodal processing. This paper suggests a novel non-destructive and multimodal classification using deep convolutional neural networks that estimate fruit maturity by feature concatenation of data acquired from two imaging modes: visible-light and hyperspectral imaging systems. Morphological changes in the sample fruits can be easily measured with RGB images, while spectral signatures that provide high sensitivity and high correlation with the internal properties of fruits can be extracted from hyperspectral images with wavelength range in between 400 nm and 900 nm—factors that must be considered when building a model. This study further modified the architectures: AlexNet, VGG16, VGG19, ResNet50, ResNeXt50, MobileNet, and MobileNetV2 to utilize multimodal data cubes composed of RGB and hyperspectral data for sensitivity analyses. These multimodal variants can achieve up to 0.90 F1 scores and 1.45% top-2 error rate for the classification of six stages. Overall, taking advantage of multimodal input coupled with powerful deep convolutional neural network models can classify fruit maturity even at refined levels of six stages. This indicates that multimodal deep learning architectures and multimodal imaging have great potential for real-time in-field fruit maturity estimation that can help estimate optimal harvest time and other in-field industrial applications.

Fluctuating environmental light limits number of surfaces visually recognizable by colour

Small changes in daylight in the environment can produce large changes in reflected light, even over short intervals of time. Do these changes limit the visual recognition of surfaces by their colour? To address this question, information-theoretic methods were used to estimate computationally the maximum number of surfaces in a sample that can be identified as the same after an interval. Scene data were taken from successive hyperspectral radiance images. With no illumination change, the average number of surfaces distinguishable by colour was of the order of 10,000. But with an illumination change, the average number still identifiable declined rapidly with change duration. In one condition, the number after two minutes was around 600, after 10 min around 200, and after an hour around 70. These limits on identification are much lower than with spectral changes in daylight. No recoding of the colour signal is likely to recover surface identity lost in this uncertain environment.

Abrupt darkening under high dynamic range (HDR) luminance invokes facilitation for high-contrast targets and grouping by luminance similarity

When scanning across a scene, luminance can vary by up to 100,000-to-1 (high dynamic range, HDR), requiring multiple normalizing mechanisms spanning from the retina to the cortex to support visual acuity and recognition. Vision models based on standard dynamic range (SDR) luminance contrast ratios below 100-to-1 have limited ability to generalize to real-world scenes with HDR luminance. To characterize how orientation and luminance are linked in brain mechanisms for luminance normalization, we measured orientation discrimination of Gabor targets under HDR luminance dynamics. We report a novel phenomenon, that abrupt 10- to 100-fold darkening engages contextual facilitation, distorting the apparent orientation of a high-contrast central target. Surprisingly, facilitation was influenced by grouping by luminance similarity, as well as by the degree of luminance variability in the surround. These results challenge vision models based solely on activity normalization and raise new questions that will lead to models that perform better in real-world scenes.

How Good Are RGB Cameras Retrieving Colors of Natural Scenes and Paintings?-A Study Based on Hyperspectral Imaging

RGB digital cameras (RGB) compress the spectral information into a trichromatic system capable of approximately representing the actual colors of objects. Although RGB digital cameras follow the same compression philosophy as the human eye (OBS), the spectral sensitivity is different. To what extent they provide the same chromatic experiences is still an open question, especially with complex images. We addressed this question by comparing the actual colors derived from spectral imaging with those obtained with RGB cameras. The data from hyperspectral imaging of 50 natural scenes and 89 paintings was used to estimate the chromatic differences between OBS and RGB. The corresponding color errors were estimated and analyzed in the color spaces CIELAB (using the color difference formulas ΔE^*_ab and CIEDE2000), J_za_zb_z, and iCAM06. In CIELAB the most frequent error (using ΔE^*_ab) found was 5 for both paintings and natural scenes, a similarity that held for the other spaces tested. In addition, the distribution of errors across the color space shows that the errors are small in the achromatic region and increase with saturation. Overall, the results indicate that the chromatic errors estimated are close to the acceptance error and therefore RGB digital cameras are able to produce quite realistic colors of complex scenarios.

Transmission filters forming orthogonal basis for spectral imaging purposes

Hyperspectral imaging has become a common technique in many different applications, enabling accurate identification of materials based on their optical properties; however, it requires complex and expensive technical implementation. A less expensive way to produce spectral data, spectral estimation, suffers from complex mathematics and limited accuracy. We introduce a novel, to the best of our knowledge, method where spectral reflectance curves can be reconstructed from the measured camera responses without complex mathematics. We have simulated the method with seven non-negative broadband transmission filters extracted from Munsell color data through principal component analysis and used sensitivity and noise levels characteristic of the Retiga 4000DC 12-bit monochrome camera. The method is sensitive to noise but produces sufficient reproduction accuracy even with six filters.

Hyperspectral Imaging for Skin Feature Detection: Advances in Markerless Tracking for Spine Surgery

In spinal surgery, surgical navigation is an essential tool for safe intervention, including the placement of pedicle screws without injury to nerves and blood vessels. Commercially available systems typically rely on the tracking of a dynamic reference frame attached to the spine of the patient. However, the reference frame can be dislodged or obscured during the surgical procedure, resulting in loss of navigation. Hyperspectral imaging (HSI) captures a large number of spectral information bands across the electromagnetic spectrum, providing image information unseen by the human eye. We aim to exploit HSI to detect skin features in a novel methodology to track patient position in navigated spinal surgery. In our approach, we adopt two local feature detection methods, namely a conventional handcrafted local feature and a deep learning-based feature detection method, which are compared to estimate the feature displacement between different frames due to motion. To demonstrate the ability of the system in tracking skin features, we acquire hyperspectral images of the skin of 17 healthy volunteers. Deep-learned skin features are detected and localized with an average error of only 0.25 mm, outperforming the handcrafted local features with respect to the ground truth based on the use of optical markers.

Semisupervised classification of hyperspectral images with low-rank representation kernel

A semisupervised deformed kernel function, using low-rank representation with consideration of local geometrical structure of data, is presented for the classification of hyperspectral images. The proposed method incorporates the wealth of unlabeled information to deal with the limited labeled samples situation as a common case in HSIs applications. The proposed kernel needs to be computed before training the classifier, e.g., a support vector machine, and it relies on combining the standard radial basis function kernel based on labeled information and the low-rank representation kernel obtained using all available (labeled and unlabeled) information. The low-rank representation kernel can overcome the difficulties of clustering methods that are used to construct the kernels such as bagged kernel and multi-scale bagged kernel. The experimental results of two well-known HSIs data sets demonstrate the effectiveness of the proposed method in comparison with cluster kernels obtained using traditional clustering methods and graph learning methods.

Computing the relevant colors that describe the color palette of paintings

In this paper, we introduce an innovative parameter that allows us to evaluate the so-called "relevant colors" in a painting; in other words, the number of colors that would stand out for an observer when just glancing at a painting. These relevant colors allow us to characterize the color palette of a scene and, on this basis, those discernible colors that are colorimetrically different within the scene. We tried to carry out this characterization of the chromatic range of paints according to authors and styles. We used a collection of 4,266 paintings by 91 painters, from which we extracted various parameters that are exclusively colorimetric to characterize the range of colors. After this refinement of the set of selected colors, our algorithm obtained an average number of 18 relevant colors, which partially agreed with the total 11-15 basic color names usually found in other categorical color studies.

Multispectral time-of-flight imaging using light-emitting diodes

Multispectral and 3-D imaging are useful for a wide variety of applications, adding valuable spectral and depth information for image analysis. Single-photon avalanche diode (SPAD) based imaging systems provide photon time-of-arrival information, and can be used for imaging with time-correlated single photon counting techniques. Here we demonstrate an LED based synchronised illumination system, where temporally structured light can be used to relate time-of-arrival to specific wavelengths, thus recovering reflectance information. Cross-correlation of the received multi-peak histogram with a reference measurement yields a time delay, allowing depth information to be determined with cm-scale resolution despite the long sequence of relatively wide (∼10 ns) pulses. Using commercial LEDs and a SPAD imaging array, multispectral 3-D imaging is demonstrated across 9 visible wavelength bands.

Hyperspectral environmental illumination maps: characterizing directional spectral variation in natural environments

Objects placed in real-world scenes receive incident light from every direction, and the spectral content of this light may vary from one direction to another. In computer graphics, environmental illumination is approximated using maps that specify illumination at a point as a function of incident angle. However, to-date, existing public databases of environmental illumination specify only three colour channels (RGB). We have captured a new set of 12 environmental illumination maps (eight outdoor scenes; four indoor scenes) using a hyperspectral imaging system with 33 spectral channels. The data reveal a striking directional variation of spectral distribution of lighting in natural environments. We discuss limitations of using daylight models to describe natural environmental illumination.

Tuning color and saving energy with spatially variable laser illumination

Previous studies have shown that the radiant flux that needs to be emitted by an illumination system, can be significantly reduced by optimizing its spectral power distribution to the object reflectance spectra, without inducing perceptible chroma or hue shifts of the illuminated objects. In this paper, the idea is explored to vary the spectral power distribution at different positions in the illuminated scene, in order to tailor the color appearance of objects. For this, a spatially variable, laser diode based illumination system is considered with three primaries and large color gamut. The color rendering performance of the illumination system is quantified via the IES TM-30-2018 method. It is shown that it is possible to reach the maximal color gamut score that is theoretically allowed by the corresponding color fidelity score. This is a unique property of an illumination system with a spatially variable spectral power distribution. The radiant flux requirements of this laser diode based illumination system are theoretically investigated for various color rendering settings, showing reduced power requirements for higher color gamut. The possibility to tune color rendering is also experimentally demonstrated with a set-up that consists of a commercially available laser projector with a hyperspectral camera. By including a feedback optimization algorithm, it is possible to reach the targeted color rendering performance.