Natural metamers

Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learning

The ability to acquire abstract knowledge is a hallmark of human intelligence and is believed by many to be one of the core differences between humans and neural network models. Agents can be endowed with an inductive bias towards abstraction through meta-learning, where they are trained on a distribution of tasks that share some abstract structure that can be learned and applied. However, because neural networks are hard to interpret, it can be difficult to tell whether agents have learned the underlying abstraction, or alternatively statistical patterns that are characteristic of that abstraction. In this work, we compare the performance of humans and agents in a meta-reinforcement learning paradigm in which tasks are generated from abstract rules. We define a novel methodology for building "task metamers" that closely match the statistics of the abstract tasks but use a different underlying generative process, and evaluate performance on both abstract and metamer tasks. We find that humans perform better at abstract tasks than metamer tasks whereas common neural network architectures typically perform worse on the abstract tasks than the matched metamers. This work provides a foundation for characterizing differences between humans and machine learning that can be used in future work towards developing machines with more human-like behavior.

Matched illumination: using light modulation as a proxy for a color filter that makes a camera more colorimetric

In previous work, it was shown that a camera can theoretically be made more colorimetric-its RGBs become more linearly related to XYZ tristimuli-by placing a specially designed color filter in the optical path. While the prior art demonstrated the principle, the optimal color-correction filters were not actually manufactured. In this paper, we provide a novel way of creating the color filtering effect without making a physical filter: we modulate the spectrum of the light source by using a spectrally tunable lighting system to recast the prefiltering effect from a lighting perspective. According to our method, if we wish to measure color under a D65 light, we relight the scene with a modulated D65 spectrum where the light modulation mimics the effect of color prefiltering in the prior art. We call our optimally modulated light, the matched illumination. In the experiments, using synthetic and real measurements, we show that color measurement errors can be reduced by about 50% or more on simulated data and 25% or more on real images when the matched illumination is used.

Designing Color Filters That Make Cameras More Colorimetric

When we place a colored filter in front of a camera the effective camera response functions are equal to the given camera spectral sensitivities multiplied by the filter spectral transmittance. In this article, we solve for the filter which returns the modified sensitivities as close to being a linear transformation from the color matching functions of the human visual system as possible. When this linearity condition - sometimes called the Luther condition- is approximately met, the 'camera+filter' system can be used for accurate color measurement. Then, we reformulate our filter design optimisation for making the sensor responses as close to the CIEXYZ tristimulus values as possible given the knowledge of real measured surfaces and illuminants spectra data. This data-driven method in turn is extended to incorporate constraints on the filter (smoothness and bounded transmission). Also, because how the optimisation is initialised is shown to impact on the performance of the solved-for filters, a multi-initialisation optimisation is developed. Experiments demonstrate that, by taking pictures through our optimised color filters, we can make cameras significantly more colorimetric.

Colour and illumination in computer vision

In computer vision, illumination is considered to be a problem that needs to be 'solved'. The colour cast due to illumination is removed to support colour-based image recognition and stable tracking (in and out of shadows), among other tasks. In this paper, I review historical and current algorithms for illumination estimation. In the classical approach, the illuminant colour is estimated by an ever more sophisticated analysis of simple image summary statistics often followed by a bias correction step. Bias correction is a function applied to the estimates made by a given illumination estimation algorithm to correct consistent errors in the estimations. Most recently, the full power, and much higher complexity, of deep learning has been deployed (where, effectively, the definition of the image statistics of interest and the type of analysis carried out are found as part of an overall optimization). In this paper, I challenge the orthodoxy of deep learning, i.e. that it is the best approach for illuminant estimation. We instead focus on the final bias correction stage found in many simple illumination estimation algorithms. There are two key insights in our method. First, we argue that the bias must be corrected in an exposure invariant way. Second, we show that this bias correction amounts to 'solving for a homography'. Homography-based illuminant estimation is shown to deliver leading illumination estimation performance (at a very small fraction of the complexity of deep learning methods).

G-Channel Restoration for RWB CFA with Double-Exposed W Channel

In this paper, we propose a green (G)-channel restoration for a red–white–blue (RWB) color filter array (CFA) image sensor using the dual sampling technique. By using white (W) pixels instead of G pixels, the RWB CFA provides high-sensitivity imaging and an improved signal-to-noise ratio compared to the Bayer CFA. However, owing to this high sensitivity, the W pixel values become rapidly over-saturated before the red–blue (RB) pixel values reach the appropriate levels. Because the missing G color information included in the W channel cannot be restored with a saturated W, multiple captures with dual sampling are necessary to solve this early W-pixel saturation problem. Each W pixel has a different exposure time when compared to those of the R and B pixels, because the W pixels are double-exposed. Therefore, a RWB-to-RGB color conversion method is required in order to restore the G color information, using a double-exposed W channel. The proposed G-channel restoration algorithm restores G color information from the W channel by considering the energy difference caused by the different exposure times. Using the proposed method, the RGB full-color image can be obtained while maintaining the high-sensitivity characteristic of the W pixels.

Metrics and statistics of frequency of occurrence of metamerism in consumer cameras for natural scenes

This paper presents metrics and statistics of frequency of occurrence of metamerism in three consumer cameras, viz., Canon 1D Mark III, Nikon D40, and Sony α7, using spectral and RGB images of natural scenes. Both sensor metamerism and observer metamerism of the cameras' sensors are studied. We use the concept of dissimilarity of two spectral power distributions in the spectral domain and the RGB domain for studying the occurrence sensor metamerism. Specifically, we use angular difference and digital equivalence approaches for this purpose. For studying the occurrence observer metamerism, we use the weighted Nimeroff's index for dissimilarity in the spectral domain with respect to the CIE color space along with the conventionally used CIE LAB color difference for dissimilarity in the CIE color space. The statistics of the frequency of occurrence of metamerism are generated on a dataset of 423 spectral images of indoor scenes in 5 illumination conditions and outdoor scenes in natural illumination conditions. Experiments show that about 18%-22% of the pixels in the images are metameric in the sense of angular difference. It is also observed that 1%-4% of the colors that would have appeared similar to human eyes are reproduced as distinct colors in the cameras. Dataset and details can be found at https://sites.google.com/site/dilipprasad/source-codes.

Generalized spectral decomposition: a theory and practice to spectral reconstruction

The generalized spectral decomposition (GSD) theorem is introduced, and the generalized fundamental stimulus and metameric black are analyzed to show how they convey the valuable features in terms of color information. The suggestion would be considered as the generalization of Cohen and Kappauf's matrix R theory and its later application in parameric correction by Fairman. The GSD theorem provides a modular model whose arguments can be elaborately set up for high-performance spectral recovery. It is also shown that the suggested methods for spectral decomposition and/or spectral reconstruction proposed by different researchers could be considered as special cases of GSD.

Analytic solution for separating spectra into illumination and surface reflectance components

The measured light spectrum is the result of an illuminant interacting with a surface. The illuminant spectral power distribution multiplies the surface spectral reflectance function to form a color signal--the light spectrum that gives rise to our perception. Disambiguation of the two factors, illuminant and surface, is difficult without prior knowledge. Previously [IEEE Trans. Pattern Anal. Mach. Intell.12, 966 (1990); J. Opt. Soc. Am. A21, 1825 (2004)], one approach to this problem applied a finite-dimensional basis function model to recover the separate illuminant and surface reflectance components that make up the color signal, using principal component bases for lights and for reflectances. We introduce the idea of making use of finite-dimensional models of logarithms of spectra for this problem. Recognizing that multiplications turn into additions in such a formulation, we can replace the original iterative method with a direct, analytic algorithm with no iteration, resulting in a speedup of several orders of magnitude. Moreover, in the new, logarithm-based approach, it is straightforward to further design new basis functions, for both illuminant and reflectance simultaneously, such that the initial basis function coefficients derived from the input color signal are optimally mapped onto separate coefficients that produce spectra that more closely approximate the illuminant and the surface reflectance for any given dimensionality. This is accomplished by using an extra bias correction step that maps the analytically determined basis function coefficients onto the optimal coefficient set, separately for lights and surfaces, for the training set. The analytic equation plus the bias correction is then used for unknown input color signals.

A subspace matching color filter design methodology for a multispectral imaging system

In this paper, we present a methodology to design filters for an imaging system to improve the accuracy of the spectral measurements for families of reflective surfaces. We derive the necessary and sufficient conditions that the sensor space of the system must obey in order to measure the spectral reflectance of the surfaces accurately. Through simulations, we show how these conditions can be applied to design filters using a set of sample spectral data acquired from extracted teeth. For this set of data, we also compare our results to those of Wolski's method, a conventional filter design method which produces filters that recover tristimulus values of surfaces accurately under several illuminants. We show that our method produces filters that capture the spectral reflectance better given the same number of measurements. The errors in predicting the color of the sample data are much lower under every test illuminant when the filters designed with our method are used.

Metamer-set-based approach to estimating surface reflectance from camera RGB

We present an approach to estimating the reflectance of a surface given its camera response. In this approach we first solve the general form of this problem: we calculate the set of all possible surface reflectances, called the metamer set, and then choose a member from this set. Three possibilities in choosing a single reflectance are described here. First, we assume that all reflectances are equally likely and minimize worst-case error. Second, we adopt the assumption that reflectances follow a normal probability distribution and maximize this probability. Finally, we assume that reflectances are smooth and maximize this property. The results of our experiments show that there is significant benefit from the proposed approach in terms of the accuracy of the estimation compared with that of standard estimation methods. Moreover, the present approach introduces a notion of robustness of estimates in the form of error bars.

Estimating visible through near-infrared spectral reflectance from a sensor radiance spectrum

We show that surface spectral reflectance can be separated from illumination effects in visible through near-infrared (350 nm-1740 nm) hyperspectral data by using only the information in a single radiance spectrum. The separation method exploits the fact that reflectance and illumination spectra typically lie in distinct subspaces. We present a comparison of a linear and a nonlinear algorithm for the separation. These algorithms compute an estimate of the spectral reflectance up to a scaling factor. In addition, we present an iterative method that is used to determine the starting point for the nonlinear algorithm. We also develop a method for selecting the dimension of the reflectance and illumination subspaces that is appropriate for material identification applications. The accuracy of the separation methods is quantified by application to noisy visible through near-infrared spectral data with a database of 107 materials and 3000 illumination spectra. The utility of the separation method for material identification is demonstrated with the same database. The results show that accurate reflectance recovery and material identification is possible by use of visible through near-infrared spectral data over the outdoor environmental conditions represented in this data set.

Spectral estimation theory: beyond linear but before Bayesian

Most color-acquisition devices capture spectral signals by acquiring only three samples, critically undersampling the spectral information. We analyze the problem of estimating high-dimensional spectral signals from low-dimensional device responses. We begin with the theory and geometry of linear estimation methods. These methods use linear models to characterize the likely input signals and reduce the number of estimation parameters. Next, we introduce two submanifold estimation methods. These methods are based on the observation that for many data sets the deviation between the signal and the linear estimate is systematic; the methods incorporate knowledge of these systematic deviations to improve upon linear estimation methods. We describe the geometric intuition of these methods and evaluate the submanifold method on hyperspectral image data.

Multispectral processing without spectra

It is often the case that multiplications of whole spectra, component by component, must be carried out,for example when light reflects from or is transmitted through materials. This leads to particularly taxing calculations, especially in spectrally based ray tracing or radiosity in graphics, making a full-spectrum method prohibitively expensive. Nevertheless, using full spectra is attractive because of the many important phenomena that can be modeled only by using all the physics at hand. We apply to the task of spectral multiplication a method previously used in modeling RGB-based light propagation. We show that we can often multiply spectra without carrying out spectral multiplication. In previous work [J. Opt. Soc. Am. A 11, 1553 (1994)] we developed a method called spectral sharpening, which took camera RGBs to a special sharp basis that was designed to render illuminant change simple to model. Specifically, in the new basis, one can effectively model illuminant change by using a diagonal matrix rather than the 3 x 3 linear transform that results from a three-component finite-dimensional model [G. Healey and D. Slater, J. Opt. Soc. Am. A 11, 3003 (1994)]. We apply this idea of sharpening to the set of principal components vectors derived from a representative set of spectra that might reasonably be encountered in a given application. With respect to the sharp spectral basis, we show that spectral multiplications can be modeled as the multiplication of the basis coefficients. These new product coefficients applied to the sharp basis serve to accurately reconstruct the spectral product. Although the method is quite general, we show how to use spectral modeling by taking advantage of metameric surfaces, ones that match under one light but not another, for tasks such as volume rendering. The use of metamers allows a user to pick out or merge different volume structures in real time simply by changing the lighting.

Color constancy at a pixel

In computational terms we can solve the color constancy problem if device red, green, and blue sensor responses, or RGB's, for surfaces seen under an unknown illuminant can be mapped to corresponding RGB's under a known reference light. In recent years almost all authors have argued that this three-dimensional problem is too hard. It is argued that because a bright light striking a dark surface results in the same physical spectra as those of a dim light incident on a light surface, the magnitude of RGB's cannot be recovered. Consequently, modern color constancy algorithms attempt only to recover image chromaticities under the reference light: They solve a two-dimensional problem. While significant progress has been made toward achieving chromaticity constancy, recent work has shown that the most advanced algorithms are unable to render chromaticity stable enough so that it can be used as a cue for object recognition [B. V. Funt, K. Bernard, and L. Martin, in Proceedings of the Fifth European Conference on Computer Vision (European Vision Society, Springer-Verlag, Berlin, 1998), Vol. II, p. 445.] We take this reductionist approach a little further and look at the one-dimensional color constancy problem. We ask, Is there a single color coordinate, a function of image chromaticities, for which the color constancy problem can be solved? Our answer is an emphatic yes. We show that there exists a single invariant color coordinate, a function of R, G, and B, that depends only on surface reflectance. Two corollaries follow. First, given an RGB image of a scene viewed under any illuminant, we can trivially synthesize the same gray-scale image (we simply code the invariant coordinate as a gray scale). Second, this result implies that we can solve the one-dimensional color constancy problem at a pixel (in scenes with no color diversity whatsoever). We present experiments that show that invariant gray-scale histograms are a stable feature for object recognition. Indexing on invariant distributions supports almost perfect recognition for a dataset of 11 objects viewed under five colored lights. In contrast, object recognition based on chromaticity histograms (post-color constancy preprocessing) delivers much poorer recognition.