Improving the perception of low-light enhanced images

Improving images captured under low-light conditions has become an important topic in computational color imaging, as it has a wide range of applications. Most current methods are either based on handcrafted features or on end-to-end training of deep neural networks that mostly focus on minimizing some distortion metric -such as PSNR or SSIM- on a set of training images. However, the minimization of distortion metrics does not mean that the results are optimal in terms of perception (i.e. perceptual quality). As an example, the perception-distortion trade-off states that, close to the optimal results, improving distortion results in worsening perception. This means that current low-light image enhancement methods -that focus on distortion minimization- cannot be optimal in the sense of obtaining a good image in terms of perception errors. In this paper, we propose a post-processing approach in which, given the original low-light image and the result of a specific method, we are able to obtain a result that resembles as much as possible that of the original method, but, at the same time, giving an improvement in the perception of the final image. More in detail, our method follows the hypothesis that in order to minimally modify the perception of an input image, any modification should be a combination of a local change in the shading across a scene and a global change in illumination color. We demonstrate the ability of our method quantitatively using perceptual blind image metrics such as BRISQUE, NIQE, or UNIQUE, and through user preference tests.

Performance Comparison of Classical Methods and Neural Networks for Colour Correction

Colour correction is the process of converting RAW RGB pixel values of digital cameras to a standard colour space such as CIE XYZ. A range of regression methods including linear, polynomial and root-polynomial least-squares have been deployed. However, in recent years, various neural network (NN) models have also started to appear in the literature as an alternative to classical methods. In the first part of this paper, a leading neural network approach is compared and contrasted with regression methods. We find that, although the neural network model supports improved colour correction compared with simple least-squares regression, it performs less well than the more advanced root-polynomial regression. Moreover, the relative improvement afforded by NNs, compared to linear least-squares, is diminished when the regression methods are adapted to minimise a perceptual colour error. Problematically, unlike linear and root-polynomial regressions, the NN approach is tied to a fixed exposure (and when exposure changes, the afforded colour correction can be quite poor). We explore two solutions that make NNs more exposure-invariant. First, we use data augmentation to train the NN for a range of typical exposures and second, we propose a new NN architecture which, by construction, is exposure-invariant. Finally, we look into how the performance of these algorithms is influenced when models are trained and tested on different datasets. As expected, the performance of all methods drops when tested with completely different datasets. However, we noticed that the regression methods still outperform the NNs in terms of colour correction, even though the relative performance of the regression methods does change based on the train and test datasets.

A Rehabilitation of Pixel-Based Spectral Reconstruction from RGB Images

Recently, many deep neural networks (DNN) have been proposed to solve the spectral reconstruction (SR) problem: recovering spectra from RGB measurements. Most DNNs seek to learn the relationship between an RGB viewed in a given spatial context and its corresponding spectra. Significantly, it is argued that the same RGB can map to different spectra depending on the context with respect to which it is seen and, more generally, that accounting for spatial context leads to improved SR. However, as it stands, DNN performance is only slightly better than the much simpler pixel-based methods where spatial context is not used. In this paper, we present a new pixel-based algorithm called A++ (an extension of the A+ sparse coding algorithm). In A+, RGBs are clustered, and within each cluster, a designated linear SR map is trained to recover spectra. In A++, we cluster the spectra instead in an attempt to ensure neighboring spectra (i.e., spectra in the same cluster) are recovered by the same SR map. A polynomial regression framework is developed to estimate the spectral neighborhoods given only the RGB values in testing, which in turn determines which mapping should be used to map each testing RGB to its reconstructed spectrum. Compared to the leading DNNs, not only does A++ deliver the best results, it is parameterized by orders of magnitude fewer parameters and has a significantly faster implementation. Moreover, in contradistinction to some DNN methods, A++ uses pixel-based processing, which is robust to image manipulations that alter the spatial context (e.g., blurring and rotations). Our demonstration on the scene relighting application also shows that, while SR methods, in general, provide more accurate relighting results compared to the traditional diagonal matrix correction, A++ provides superior color accuracy and robustness compared to the top DNN methods.

Matched illumination: using light modulation as a proxy for a color filter that makes a camera more colorimetric

In previous work, it was shown that a camera can theoretically be made more colorimetric-its RGBs become more linearly related to XYZ tristimuli-by placing a specially designed color filter in the optical path. While the prior art demonstrated the principle, the optimal color-correction filters were not actually manufactured. In this paper, we provide a novel way of creating the color filtering effect without making a physical filter: we modulate the spectrum of the light source by using a spectrally tunable lighting system to recast the prefiltering effect from a lighting perspective. According to our method, if we wish to measure color under a D65 light, we relight the scene with a modulated D65 spectrum where the light modulation mimics the effect of color prefiltering in the prior art. We call our optimally modulated light, the matched illumination. In the experiments, using synthetic and real measurements, we show that color measurement errors can be reduced by about 50% or more on simulated data and 25% or more on real images when the matched illumination is used.

On the Optimization of Regression-Based Spectral Reconstruction

Spectral reconstruction (SR) algorithms attempt to recover hyperspectral information from RGB camera responses. Recently, the most common metric for evaluating the performance of SR algorithms is the Mean Relative Absolute Error (MRAE)-an ℓ1 relative error (also known as percentage error). Unsurprisingly, the leading algorithms based on Deep Neural Networks (DNN) are trained and tested using the MRAE metric. In contrast, the much simpler regression-based methods (which actually can work tolerably well) are trained to optimize a generic Root Mean Square Error (RMSE) and then tested in MRAE. Another issue with the regression methods is-because in SR the linear systems are large and ill-posed-that they are necessarily solved using regularization. However, hitherto the regularization has been applied at a spectrum level, whereas in MRAE the errors are measured per wavelength (i.e., per spectral channel) and then averaged. The two aims of this paper are, first, to reformulate the simple regressions so that they minimize a relative error metric in training-we formulate both ℓ2 and ℓ1 relative error variants where the latter is MRAE-and, second, we adopt a per-channel regularization strategy. Together, our modifications to how the regressions are formulated and solved leads to up to a 14% increment in mean performance and up to 17% in worst-case performance (measured with MRAE). Importantly, our best result narrows the gap between the regression approaches and the leading DNN model to around 8% in mean accuracy.

Designing Color Filters That Make Cameras More Colorimetric

When we place a colored filter in front of a camera the effective camera response functions are equal to the given camera spectral sensitivities multiplied by the filter spectral transmittance. In this article, we solve for the filter which returns the modified sensitivities as close to being a linear transformation from the color matching functions of the human visual system as possible. When this linearity condition - sometimes called the Luther condition- is approximately met, the 'camera+filter' system can be used for accurate color measurement. Then, we reformulate our filter design optimisation for making the sensor responses as close to the CIEXYZ tristimulus values as possible given the knowledge of real measured surfaces and illuminants spectra data. This data-driven method in turn is extended to incorporate constraints on the filter (smoothness and bounded transmission). Also, because how the optimisation is initialised is shown to impact on the performance of the solved-for filters, a multi-initialisation optimisation is developed. Experiments demonstrate that, by taking pictures through our optimised color filters, we can make cameras significantly more colorimetric.

Providing a Single Ground-Truth for Illuminant Estimation for the ColorChecker Dataset

The ColorChecker dataset is one of the most widely used image sets for evaluating and ranking illuminant estimation algorithms. However, this single set of images has at least 3 different sets of ground-truth (i.e., correct answers) associated with it. In the literature it is often asserted that one algorithm is better than another when the algorithms in question have been tuned and tested with the different ground-truths. In this short correspondence we present some of the background as to why the 3 existing ground-truths are different and go on to make a new single and recommended set of correct answers. Experiments reinforce the importance of this work in that we show that the total ordering of a set of algorithms may be reversed depending on whether we use the new or legacy ground-truth data.

Physical-based optimization for non-physical image dehazing methods

Images captured under hazy conditions (e.g. fog, air pollution) usually present faded colors and loss of contrast. To improve their visibility, a process called image dehazing can be applied. Some of the most successful image dehazing algorithms are based on image processing methods but do not follow any physical image formation model, which limits their performance. In this paper, we propose a post-processing technique to alleviate this handicap by enforcing the original method to be consistent with a popular physical model for image formation under haze. Our results improve upon those of the original methods qualitatively and according to several metrics, and they have also been validated via psychophysical experiments. These results are particularly striking in terms of avoiding over-saturation and reducing color artifacts, which are the most common shortcomings faced by image dehazing methods.

Colour and illumination in computer vision

In computer vision, illumination is considered to be a problem that needs to be 'solved'. The colour cast due to illumination is removed to support colour-based image recognition and stable tracking (in and out of shadows), among other tasks. In this paper, I review historical and current algorithms for illumination estimation. In the classical approach, the illuminant colour is estimated by an ever more sophisticated analysis of simple image summary statistics often followed by a bias correction step. Bias correction is a function applied to the estimates made by a given illumination estimation algorithm to correct consistent errors in the estimations. Most recently, the full power, and much higher complexity, of deep learning has been deployed (where, effectively, the definition of the image statistics of interest and the type of analysis carried out are found as part of an overall optimization). In this paper, I challenge the orthodoxy of deep learning, i.e. that it is the best approach for illuminant estimation. We instead focus on the final bias correction stage found in many simple illumination estimation algorithms. There are two key insights in our method. First, we argue that the bias must be corrected in an exposure invariant way. Second, we show that this bias correction amounts to 'solving for a homography'. Homography-based illuminant estimation is shown to deliver leading illumination estimation performance (at a very small fraction of the complexity of deep learning methods).

The Reproduction Angular Error for Evaluating the Performance of Illuminant Estimation Algorithms

The angle between the RGBs of the measured illuminant and estimated illuminant colors-the recovery angular error-has been used to evaluate the performance of the illuminant estimation algorithms. However we noticed that this metric is not in line with how the illuminant estimates are used. Normally, the illuminant estimates are `divided out' from the image to, hopefully, provide image colors that are not confounded by the color of the light. However, even though the same reproduction results the same scene might have a large range of recovery errors. In this work the scale of the problem with the recovery error is quantified. Next we propose a new metric for evaluating illuminant estimation algorithms, called the reproduction angular error, which is defined as the angle between the RGB of a white surface when the actual and estimated illuminations are `divided out'. Our new metric ties algorithm performance to how the illuminant estimates are used. For a given algorithm, adopting the new reproduction angular error leads to different optimal parameters. Further the ranked list of best to worst algorithms changes when the reproduction angular is used. The importance of using an appropriate performance metric is established.

Estimating individual cone fundamentals from their color-matching functions

Estimation of individual spectral cone fundamentals from color-matching functions is a classical and longstanding problem in color science. In this paper we propose a novel method to carry out this estimation based on a linear optimization technique, employing an assumption of a priori knowledge of the retinal absorptance functions. The result is an estimation of the combined lenticular and macular filtration for an individual, along with the nine coefficients in the linear combination that relates their color-matching functions to their estimated spectral-cone fundamentals. We test the method on the individual Stiles and Burch color-matching functions and derive cone-fundamental estimations for different viewing fields and matching experiment repetition. We obtain cone-fundamental estimations that are remarkably similar to those available in the literature. This suggests that the method yields results that are close to the true fundamentals.

Spectral edge: gradient-preserving spectral mapping for image fusion

This paper describes a novel approach to image fusion for color display. Our goal is to generate an output image whose gradient matches that of the input as closely as possible. We achieve this using a constrained contrast mapping paradigm in the gradient domain, where the structure tensor of a high-dimensional gradient representation is mapped exactly to that of a low-dimensional gradient field which is then reintegrated to form an output. Constraints on output colors are provided by an initial RGB rendering. Initially, we motivate our solution with a simple "ansatz" (educated guess) for projecting higher-D contrast onto color gradients, which we expand to a more rigorous theorem to incorporate color constraints. The solution to these constrained optimizations is closed-form, allowing for simple and hence fast and efficient algorithms. The approach can map any N-D image data to any M-D output and can be used in a variety of applications using the same basic algorithm. In this paper, we focus on the problem of mapping N-D inputs to 3D color outputs. We present results in five applications: hyperspectral remote sensing, fusion of color and near-infrared or clear-filter images, multilighting imaging, dark flash, and color visualization of magnetic resonance imaging diffusion-tensor imaging.

Color correction using root-polynomial regression

Cameras record three color responses (RGB) which are device dependent. Camera coordinates are mapped to a standard color space, such as XYZ-useful for color measurement-by a mapping function, e.g., the simple 3×3 linear transform (usually derived through regression). This mapping, which we will refer to as linear color correction (LCC), has been demonstrated to work well in the number of studies. However, it can map [Formula: see text] to XYZs with high error. The advantage of the LCC is that it is independent of camera exposure. An alternative and potentially more powerful method for color correction is polynomial color correction (PCC). Here, the R, G, and B values at a pixel are extended by the polynomial terms. For a given calibration training set PCC can significantly reduce the colorimetric error. However, the PCC fit depends on exposure, i.e., as exposure changes the vector of polynomial components is altered in a nonlinear way which results in hue and saturation shifts. This paper proposes a new polynomial-type regression loosely related to the idea of fractional polynomials which we call root-PCC (RPCC). Our idea is to take each term in a polynomial expansion and take its k th root of each k -degree term. It is easy to show terms defined in this way scale with exposure. RPCC is a simple (low complexity) extension of LCC. The experiments presented in this paper demonstrate that RPCC enhances color correction performance on real and synthetic data.

Chromatic illumination discrimination ability reveals that human colour constancy is optimised for blue daylight illuminations

The phenomenon of colour constancy in human visual perception keeps surface colours constant, despite changes in their reflected light due to changing illumination. Although colour constancy has evolved under a constrained subset of illuminations, it is unknown whether its underlying mechanisms, thought to involve multiple components from retina to cortex, are optimised for particular environmental variations. Here we demonstrate a new method for investigating colour constancy using illumination matching in real scenes which, unlike previous methods using surface matching and simulated scenes, allows testing of multiple, real illuminations. We use real scenes consisting of solid familiar or unfamiliar objects against uniform or variegated backgrounds and compare discrimination performance for typical illuminations from the daylight chromaticity locus (approximately blue-yellow) and atypical spectra from an orthogonal locus (approximately red-green, at correlated colour temperature 6700 K), all produced in real time by a 10-channel LED illuminator. We find that discrimination of illumination changes is poorer along the daylight locus than the atypical locus, and is poorest particularly for bluer illumination changes, demonstrating conversely that surface colour constancy is best for blue daylight illuminations. Illumination discrimination is also enhanced, and therefore colour constancy diminished, for uniform backgrounds, irrespective of the object type. These results are not explained by statistical properties of the scene signal changes at the retinal level. We conclude that high-level mechanisms of colour constancy are biased for the blue daylight illuminations and variegated backgrounds to which the human visual system has typically been exposed.

Reducing integrability error of color tensor gradients for image fusion

To overcome the difficulties in applying gradient-based operators to color images, Di Zenzo introduced the color tensor, an operator that provides a gradient field for multichannel images. An elegant application for this operator was developed in the domain of multichannel image visualization: Socolinsky and Wolff proposed to reintegrate Di Zenzo's gradient by solving a Poisson equation, yielding a greyscale representation of the multispectral contrast of the input image. Di Zenzo's gradients are, however, generally not integrable and some approximation must be introduced. Thus, the resulting image can suffer from artifacts such as the smearing of edges. In this paper, we focus on the integrability of Di Zenzo's gradients. We show that the integrability of the obtained field can be improved dramatically through a simple desaturation of the color image (as in the HSV color space). This result can be readily extended to multispectral images by defining an analogue to saturation. We present several results explaining what happens to color tensors as the saturation changes. Significantly we show that small changes of the saturation in the linear image space can result in large improvements in the integrability of tensor gradients calculated in logarithmic color space. This result is important for two reasons. 1) Log-differences are more perceptually meaningful. 2) In log-space we can operate with retinex algorithms, which are well known techniques for contrast enhancement. We propose that they can be used to "put back" any contrast that might be lost in the desaturation step and, more importantly, they can enhance contrast at the same time as reintegrating the gradient field because of their relation to partial differential equations. Finally, we evaluate our method psychophysically. Compared with other commonly used image fusion methods, experiments show that our data fusion using the Di Zenzo color tensor after desaturating the image and where a simple contrast boost is applied is strongly preferred.

Spectral sharpening by spherical sampling

There are many works in color that assume illumination change can be modeled by multiplying sensor responses by individual scaling factors. The early research in this area is sometimes grouped under the heading "von Kries adaptation": the scaling factors are applied to the cone responses. In more recent studies, both in psychophysics and in computational analysis, it has been proposed that scaling factors should be applied to linear combinations of the cones that have narrower support: they should be applied to the so-called "sharp sensors." In this paper, we generalize the computational approach to spectral sharpening in three important ways. First, we introduce spherical sampling as a tool that allows us to enumerate in a principled way all linear combinations of the cones. This allows us to, second, find the optimal sharp sensors that minimize a variety of error measures including CIE Delta E (previous work on spectral sharpening minimized RMS) and color ratio stability. Lastly, we extend the spherical sampling paradigm to the multispectral case. Here the objective is to model the interaction of light and surface in terms of color signal spectra. Spherical sampling is shown to improve on the state of the art.

Lookup-table-based gradient field reconstruction

In computer vision, there are many applications, where it is advantageous to process an image in the gradient domain and then reintegrate the gradient field: important examples include shadow removal, lightness calculation, and data fusion. A serious problem with this approach is that the reconstruction step often introduces artefacts-commonly, smoothed and smeared edges-to the recovered image. This is a result of the inherent ill-posedness of reintegrating a nonintegrable field. Artefacts can be diminished but not removed, by using complex to highly complex reintegration techniques. Here, we present a remarkably simple (and on the face of it naive) algorithm for reconstructing gradient fields. Suppose we start with a multichannel original, and from it derive a (possibly one of many) 1-D gradient field; for many applications, the derived gradient field will be nonintegrable. Here, we propose a lookup-table-based map relating the multichannel original to a reconstructed scalar output image, whose gradient best matches the target gradient field. The idea, at base, is that if we learn how to map the gradients of the multichannel original onto the desired output gradient, and then using the lookup table (LUT) constraint, we effectively derive the mapping from the multichannel input to the desired, reintegrated, image output. While this map could take a variety of forms, here we derive the best map from the multichannel gradient as a (nonlinear) function of the input to each of the target scalar gradients. In this framework, reconstruction is a simple equation-solving exercise of low dimensionality. One obvious application of our method is to the image-fusion problem, e.g., the problem of converting a color or higher-D image into grayscale. We will show, through extensive experiments and complementary theoretical arguments, that our straightforward method preserves the target contrast as well as do complex previous reintegration methods, but without artefacts, and with a substantially cheaper computational cost. Finally, we demonstrate the generality of the method by applying it to gradient field reconstruction in an additional area, the shading recovery problem.

Constrained pseudo-Brownian motion and its application to image enhancement

Brownian motion is a random process that finds application in many fields, and its relation to certain color perception phenomena has recently been observed. On this ground, Marini and Rizzi developed a retinex algorithm based on Brownian motion paths. However, while their approach has several advantages and delivers interesting results, it has a high computational complexity. We propose an efficient algorithm that generates pseudo-Brownian paths with a very important constraint: we can guarantee a lower bound to the number of visits to each pixel, as well as its average. Despite these constraints, we show that the paths generated have certain statistical similarities to random walk and Brownian motion. Finally, we present a retinex implementation that exploits the paths generated with our algorithm, and we compare some images it generates with those obtained with the McCann99 and Frankle and McCann's algorithms (two multiscale retinex implementations that have a low computational complexity). We find that our approach causes fewer artifacts and tends to require a smaller number of pixel comparisons to achieve similar results, thus compensating for the slightly higher computational complexity.

Analytic solution for separating spectra into illumination and surface reflectance components

The measured light spectrum is the result of an illuminant interacting with a surface. The illuminant spectral power distribution multiplies the surface spectral reflectance function to form a color signal--the light spectrum that gives rise to our perception. Disambiguation of the two factors, illuminant and surface, is difficult without prior knowledge. Previously [IEEE Trans. Pattern Anal. Mach. Intell.12, 966 (1990); J. Opt. Soc. Am. A21, 1825 (2004)], one approach to this problem applied a finite-dimensional basis function model to recover the separate illuminant and surface reflectance components that make up the color signal, using principal component bases for lights and for reflectances. We introduce the idea of making use of finite-dimensional models of logarithms of spectra for this problem. Recognizing that multiplications turn into additions in such a formulation, we can replace the original iterative method with a direct, analytic algorithm with no iteration, resulting in a speedup of several orders of magnitude. Moreover, in the new, logarithm-based approach, it is straightforward to further design new basis functions, for both illuminant and reflectance simultaneously, such that the initial basis function coefficients derived from the input color signal are optimally mapped onto separate coefficients that produce spectra that more closely approximate the illuminant and the surface reflectance for any given dimensionality. This is accomplished by using an extra bias correction step that maps the analytically determined basis function coefficients onto the optimal coefficient set, separately for lights and surfaces, for the training set. The analytic equation plus the bias correction is then used for unknown input color signals.

Metamer-set-based approach to estimating surface reflectance from camera RGB

We present an approach to estimating the reflectance of a surface given its camera response. In this approach we first solve the general form of this problem: we calculate the set of all possible surface reflectances, called the metamer set, and then choose a member from this set. Three possibilities in choosing a single reflectance are described here. First, we assume that all reflectances are equally likely and minimize worst-case error. Second, we adopt the assumption that reflectances follow a normal probability distribution and maximize this probability. Finally, we assume that reflectances are smooth and maximize this property. The results of our experiments show that there is significant benefit from the proposed approach in terms of the accuracy of the estimation compared with that of standard estimation methods. Moreover, the present approach introduces a notion of robustness of estimates in the form of error bars.

Reevaluation of color constancy algorithm performance

The relative performance of color constancy algorithms is evaluated. We highlight some problems with previous algorithm evaluation and define more appropriate testing procedures. We discuss how best to measure algorithm accuracy on a single image as well as suitable methods for summarizing errors over a set of images. We also discuss how the relative performance of two or more algorithms should best be compared, and we define an experimental framework for testing algorithms. We reevaluate the performance of six color constancy algorithms using the procedures that we set out and show that this leads to a significant change in the conclusions that we draw about relative algorithm performance as compared with those from previous work.

On the removal of shadows from images

This paper is concerned with the derivation of a progression of shadow-free image representations. First, we show that adopting certain assumptions about lights and cameras leads to a 1D, gray-scale image representation which is illuminant invariant at each image pixel. We show that as a consequence, images represented in this form are shadow-free. We then extend this 1D representation to an equivalent 2D, chromaticity representation. We show that in this 2D representation, it is possible to relight all the image pixels in the same way, effectively deriving a 2D image representation which is additionally shadow-free. Finally, we show how to recover a 3D, full color shadow-free image representation by first (with the help of the 2D representation) identifying shadow edges. We then remove shadow edges from the edge-map of the original image by edge in-painting and we propose a method to reintegrate this thresholded edge map, thus deriving the sought-after 3D shadow-free image.

Metamer sets

If two different surfaces look the same when viewed under a particular light source, then they are called metamers. We show mathematically how one can solve for the whole set of physically realizable natural surface reflectances that relate to the same tristimulus, the metamer set. Our analysis is based on very general linear models of reflectances, coupled with constraints that reflectances should adhere to (e.g., positivity and boundedness). We show that we can recover metamer sets for linear models of an arbitrary high dimension. To illustrate our new algorithm, we provide an example of calculating the metamer set and its manifestation as a mismatch region. Given a single XYZ observed under illuminant D65, we can examine the set of XYZs that would be possible under illuminant A.

Multispectral processing without spectra

It is often the case that multiplications of whole spectra, component by component, must be carried out,for example when light reflects from or is transmitted through materials. This leads to particularly taxing calculations, especially in spectrally based ray tracing or radiosity in graphics, making a full-spectrum method prohibitively expensive. Nevertheless, using full spectra is attractive because of the many important phenomena that can be modeled only by using all the physics at hand. We apply to the task of spectral multiplication a method previously used in modeling RGB-based light propagation. We show that we can often multiply spectra without carrying out spectral multiplication. In previous work [J. Opt. Soc. Am. A 11, 1553 (1994)] we developed a method called spectral sharpening, which took camera RGBs to a special sharp basis that was designed to render illuminant change simple to model. Specifically, in the new basis, one can effectively model illuminant change by using a diagonal matrix rather than the 3 x 3 linear transform that results from a three-component finite-dimensional model [G. Healey and D. Slater, J. Opt. Soc. Am. A 11, 3003 (1994)]. We apply this idea of sharpening to the set of principal components vectors derived from a representative set of spectra that might reasonably be encountered in a given application. With respect to the sharp spectral basis, we show that spectral multiplications can be modeled as the multiplication of the basis coefficients. These new product coefficients applied to the sharp basis serve to accurately reconstruct the spectral product. Although the method is quite general, we show how to use spectral modeling by taking advantage of metameric surfaces, ones that match under one light but not another, for tasks such as volume rendering. The use of metamers allows a user to pick out or merge different volume structures in real time simply by changing the lighting.

Color constancy at a pixel

In computational terms we can solve the color constancy problem if device red, green, and blue sensor responses, or RGB's, for surfaces seen under an unknown illuminant can be mapped to corresponding RGB's under a known reference light. In recent years almost all authors have argued that this three-dimensional problem is too hard. It is argued that because a bright light striking a dark surface results in the same physical spectra as those of a dim light incident on a light surface, the magnitude of RGB's cannot be recovered. Consequently, modern color constancy algorithms attempt only to recover image chromaticities under the reference light: They solve a two-dimensional problem. While significant progress has been made toward achieving chromaticity constancy, recent work has shown that the most advanced algorithms are unable to render chromaticity stable enough so that it can be used as a cue for object recognition [B. V. Funt, K. Bernard, and L. Martin, in Proceedings of the Fifth European Conference on Computer Vision (European Vision Society, Springer-Verlag, Berlin, 1998), Vol. II, p. 445.] We take this reductionist approach a little further and look at the one-dimensional color constancy problem. We ask, Is there a single color coordinate, a function of image chromaticities, for which the color constancy problem can be solved? Our answer is an emphatic yes. We show that there exists a single invariant color coordinate, a function of R, G, and B, that depends only on surface reflectance. Two corollaries follow. First, given an RGB image of a scene viewed under any illuminant, we can trivially synthesize the same gray-scale image (we simply code the invariant coordinate as a gray scale). Second, this result implies that we can solve the one-dimensional color constancy problem at a pixel (in scenes with no color diversity whatsoever). We present experiments that show that invariant gray-scale histograms are a stable feature for object recognition. Indexing on invariant distributions supports almost perfect recognition for a dataset of 11 objects viewed under five colored lights. In contrast, object recognition based on chromaticity histograms (post-color constancy preprocessing) delivers much poorer recognition.

Spectral sharpening with positivity

Spectral sharpening is a method for developing camera or other optical-device sensor functions that are more narrowband than those in hardware, by means of a linear transform of sensor functions. The utility of such a transform is that many computer vision and color-correction algorithms perform better in a sharpened space, and thus such a space can be used as an intermediate representation for carrying out calculations. In this paper we consider how one may sharpen sensor functions such that the transformed sensors are all positive. We show that constrained optimization can be used to produce positive sensors in two fundamentally different ways: by constraining the coefficients in the transform or by constraining the functions directly. In the former method, we prove that convexity can be used to constrain the solution exactly. In a sense, we are continuing the work of MacAdam and of Pearson and Yule, who formed positive combinations of the color-matching functions. However, the advantage of the spectral sharpening approach is that not only can we produce positive curves, but the process is "steerable" in that we can produce positive curves with as good or better properties for sharpening within a given set of sharpening intervals. At base, however, it is positive colors in the transformed space that are the prime objective. Therefore we also carry out sharpening of sensor curves governed not by positivity of the curves themselves but of colors resulting from them. Curves that result have negative lobes but generate positive colors. We find that this type of constrained sharpening generates the best results, which are almost as good as for unconstrained sharpening but without the penalty of negative colors. All methods discussed may be used with any number of sensors.

Spectral sharpening: sensor transformations for improved color constancy

We develop sensor transformations, collectively called spectral sharpening, that convert a given set of sensor sensitivity functions into a new set that will improve the performance of any color-constancy algorithm that is based on an independent adjustment of the sensor response channels. Independent adjustment of multiplicative coefficients corresponds to the application of a diagonal-matrix transform (DMT) to the sensor response vector and is a common feature of many theories of color constancy. Land's retinex and von Kries adaptation in particular. We set forth three techniques for spectral sharpening. Sensor-based sharpening focuses on the production of new sensors as linear combinations of the given ones such that each new sensor has its spectral sensitivity concentrated as much as possible within a narrow band of wavelengths. Data-based sharpening, on the other hand, extracts new sensors by optimizing the ability of a DMT to account for a given illumination change by examining the sensor response vectors obtained from a set of surfaces under two different illuminants. Finally in perfect sharpening we demonstrate that, if illumination and surface reflectance are described by two- and three-parameter finite-dimensional models, there exists a unique optimal sharpening transform. All three sharpening methods yield similar results. When sharpened cone sensitivities are used as sensors, a DMT models illumination change extremely well. We present simulation results suggesting that in general nondiagonal transforms can do only marginally better. Our sharpening results correlate well with the psychophysical evidence of spectral sharpening in the human visual system.