Camera Spectral Sensitivity and White Balance Estimation from Sky Images

Identification of exfoliated monolayer hexagonal boron nitride films with a digital color camera under white light illumination

Optical microscopy with white light illumination has been employed when obtaining exfoliated monolayer hexagonal boron nitride (1L hBN) films from a large number of randomly placed films on a substrate. However, real-time observation of 1L hBN using a color camera under white light illumination remains challenging since hBN is transparent in the visible wavelength range. The poor optical constant of 1L hBN films in microphotographs is significantly improved using a Si substrate coated with a SiNxthin-film (SiNx/Si). When observing hBN thin films on SiNx/Si using a color digital camera in an optical microscope under white light illumination, the clarity of the captured color images depends on the thickness of the SiNxfilm (d). For real-time direct observation, thedwas optimized based on quantitative chromatic studies tailored to Bayer filters of a color image sensor. Through image simulation, it was determined that the color difference between 1L hBN and the bare substrate is maximized atd= 59 or 70 nm, which was experimentally verified. The SiNx/Si with optimizeddvalues visualized 1L hBN films without requiring significant contrast enhancement via image processing under white light illumination in real-time. Furthermore, the captured color photographs facilitate the reliable determination of the number of layers in few-layer hBN films using the contrast of the green channel of the images.

Optimized principal component analysis for camera spectral sensitivity estimation

This paper describes the use of a weighted principal component analysis (PCA) method for camera spectral sensitivity estimation. A comprehensive set of spectral sensitivities of 111 cameras was collected from four publicly available databases. It was proposed to weight the spectral sensitivities in the database according to the similarities with those of the test camera. The similarity was evaluated by the reciprocal predicted errors of camera responses. Thus, a set of dynamic principal components was generated from the weighted spectral sensitivity database and served as the basis functions to estimate spectral sensitivities. The test stimuli included self-luminous colors from a multi-channel LED system and reflective colors from a color chart. The proposed method was tested in both the simulated and practical experiments, and the results were compared with the classical PCA method, three commonly used basis function methods (Fourier, polynomial, and radial bases), and a regularization method. It was demonstrated that the proposed method significantly improved the accuracy of spectral sensitivity estimation.

Spectral and RGB analysis of the light climate and its ecological impacts using an all-sky camera system in the Arctic

The ArcLight observatory provides an hourly continuous time series of all-sky images providing light climate data (intensity, spectral composition, and photoperiod) from the Arctic (Svalbard at 79°N). Until recently, no complete annual time series of light climate relevant for biological processes has been provided from the high Arctic because of insufficient sensitivity of commercial light sensors during the Polar Night. The ArcLight set up is unique, as it provides both all-sky images and the corresponding integrated spectral irradiance in the visible part of the solar electromagnetic spectrum (E P A R ). Here we present a further development providing hourly diel-annual dynamics from 2020 of the irradiance partitioned into the red, green, and blue parts of the solar spectrum and illustrate their relation to weather conditions, and sun and moon trajectories. We show that there is variation between the RGB proportions of irradiance throughout the year, with the blue part of the spectrum showing the greatest variation, which is dependent on weather conditions (i.e., cloud cover). We further provide an example of the biological impact of these spectral variations in the light climate using in vivo Chl a-specific absorption coefficients of diatoms (mean of six low light acclimated northern-Arctic bloom-forming species) to model total algal light absorption (AQ t o t a l ) and the corresponding fraction of quanta used by Photosystem II (AQPSII) (O 2 production) in RGB bands and the potential impacts on the photoreceptor response, suggesting periods where repair and maintenance functions dominate activity in the absence of appreciable levels of red or green light. The method used here can be applied to light climate data and spectral response data worldwide to give localized ecological models of AQ.

Joint Camera Spectral Response Selection and Hyperspectral Image Recovery

Hyperspectral image (HSI) recovery from a single RGB image has attracted much attention, whose performance has recently been shown to be sensitive to the camera spectral response (CSR). In this paper, we present an efficient convolutional neural network (CNN) based method, which can jointly select the optimal CSR from a candidate dataset and learn a mapping to recover HSI from a single RGB image captured with this algorithmically selected camera under multi-chip or single-chip setups. Given a specific CSR, we first present a HSI recovery network, which accounts for the underlying characteristics of the HSI, including spectral nonlinear mapping and spatial similarity. Later, we append a CSR selection layer onto the recovery network, and the optimal CSR under both multi-chip and single-chip setups can thus be automatically determined from the network weights under the nonnegative sparse constraint. Experimental results on three hyperspectral datasets and two camera spectral response datasets demonstrate that our HSI recovery network outperforms state-of-the-art methods in terms of both quantitative metrics and perceptive quality, and the selection layer always returns a CSR consistent to the best one determined by exhaustive search. Finally, we show that our method can also perform well in the real capture system, and collect a hyperspectral flower dataset to evaluate the effect from HSI recovery on classification problem.

Scan-free end-to-end new approach for snapshot camera spectral sensitivity estimation

Spectral sensitivity is largely related to sensor imaging, which has drawn widespread attention in computer vision. Accurate estimation becomes increasingly urgent because manufacturers rarely disclose it. In this Letter, we present a novel, compact, inexpensive, and real-time computational system for snapshot spectral sensitivity estimation. A multi-scale camera based on the multi-scale convolutional neural network is first proposed, to the best of our knowledge, to automatically extract multiplexing features of an input image by multiscale deep learning, which is vital to solving the inverse problem in sensitivity estimation. Our network is flexible and can be designed with different convolutional kernel sizes for a given application. We build a dataset with 10,500 raw images and generate an excellent pre-trained model. Commercial cameras are adopted to test model validity; the results show that our system can achieve estimation accuracy as high as 91.35%. We provide a method for system design, propose a deep learning network, build a dataset, demonstrate training process, and present experimental results with high precision. This simple and effective method provides an accurate approach for precise estimation of spectral sensitivity and is suitable for computational applications such as pathological digital stain, virtual/augmented reality display, and high-quality image acquisition.

Compressive recovery of smartphone RGB spectral sensitivity functions

Spectral response (or sensitivity) functions of a three-color image sensor (or trichromatic camera) allow a mapping from spectral stimuli to RGB color values. Like biological photosensors, digital RGB spectral responses are device dependent and significantly vary from model to model. Thus, the information on the RGB spectral response functions of a specific device is vital in a variety of computer vision as well as mobile health (mHealth) applications. Theoretically, spectral response functions can directly be measured with sophisticated calibration equipment in a specialized laboratory setting, which is not easily accessible for most application developers. As a result, several mathematical methods have been proposed relying on standard color references. Typical optimization frameworks with constraints are often complicated, requiring a large number of colors. We report a compressive sensing framework in the frequency domain for accurately predicting RGB spectral response functions only with several primary colors. Using a scientific camera, we first validate the estimation method with direct spectral sensitivity measurements and ensure that the root mean square errors between the ground truth and recovered RGB spectral response functions are negligible. We further recover the RGB spectral response functions of smartphones and validate with an expanded color checker reference. We expect that this simple yet reliable estimation method of RGB spectral sensitivity can easily be applied for color calibration and standardization in machine vision, hyperspectral filters, and mHealth applications that capitalize on the built-in cameras of smartphones.

Spectral sensitivity estimation of trichromatic camera based on orthogonal test and window filtering

The three-channel spectral sensitivity of a trichromatic camera represents the characteristics of system color space. It is a mapping bridge from the spectral information of a scene to the response value of a camera. In this paper, we propose an estimation method for three-channel spectral sensitivity of a trichromatic camera. It includes calibration experiment by orthogonal test design and the data processing by window filtering. The calibration experiment was first designed by an orthogonal table of the 9-level and 3-factor. A rough estimation model of spectral sensitivity is established on the data pairs of the system input and output in calibration experiments. The data of rough estimation is then modulated by two window filters on frequency and spatial domain. The Luther-Ives condition and the smoothness condition are introduced to design the window, and help to achieve the optimal estimation of the system spectral sensitivity. Finally, the proposed method is verified by some comparison experiments. The results show that the estimated spectral sensitivity is basically consistent with the measured results of the monochromator experiments, the relative full-scale errors of the RGB three-channel is obviously lower than the Wiener filtering method and the Fourier band-limitedness method. The proposed method can estimate the spectral sensitivity of the trichromatic digital camera very well, which is of great significance for the colorimetric characterization and evaluation of imaging systems.

Practical Camera Sensor Spectral Response and Uncertainty Estimation

Knowledge of the spectral response of a camera is important in many applications such as illumination estimation, spectrum estimation in multi-spectral camera systems, and color consistency correction for computer vision. We present a practical method for estimating the camera sensor spectral response and uncertainty, consisting of an imaging method and an algorithm. We use only 15 images (four diffraction images and 11 images of color patches of known spectra to obtain high-resolution spectral response estimates) and obtain uncertainty estimates by training an ensemble of response estimation models. The algorithm does not assume any strict priors that would limit the possible spectral response estimates and is thus applicable to any camera sensor, at least in the visible range. The estimates have low errors for estimating color channel values from known spectra, and are consistent with previously reported spectral response estimates.

Estimation of the camera spectral sensitivity function using neural learning and architecture

In this paper, we propose a robust method to estimate the camera spectral sensitivity function using a neural-network-based model and a custom learning algorithm. A new and specially designed architecture for training our neural network model is presented to estimate the spectral sensitivity as a function of wavelength. The sensitivity function is modeled as the sum of a few Gaussian functions, and a radial basis function neural network is trained to approximate this function over the visual wavelengths. No constraints are imposed on the illumination distribution or spectral sensitivity, as similar methods usually do. Experimental results show that the proposed method produces superior results with much lower root mean square error compared to the methods using basis functions or constraint optimization approaches. Study of the reproduced colors also verifies the accuracy of our method.

Improving color constancy by discounting the variation of camera spectral sensitivity

It is an ill-posed problem to recover the true scene colors from a color biased image by discounting the effects of scene illuminant and camera spectral sensitivity (CSS) at the same time. Most color constancy (CC) models have been designed to first estimate the illuminant color, which is then removed from the color biased image to obtain an image taken under white light, without the explicit consideration of CSS effect on CC. This paper first studies the CSS effect on illuminant estimation arising in the inter-dataset-based CC (inter-CC), i.e., training a CC model on one dataset and then testing on another dataset captured by a distinct CSS. We show the clear degradation of existing CC models for inter-CC application. Then a simple way is proposed to overcome such degradation by first learning quickly a transform matrix between the two distinct CSSs (CSS-1 and CSS-2). The learned matrix is then used to convert the data (including the illuminant ground truth and the color-biased images) rendered under CSS-1 into CSS-2, and then train and apply the CC model on the color-biased images under CSS-2 without the need of burdensome acquiring of the training set under CSS-2. Extensive experiments on synthetic and real images show that our method can clearly improve the inter-CC performance for traditional CC algorithms. We suggest that, by taking the CSS effect into account, it is more likely to obtain the truly color constant images invariant to the changes of both illuminant and camera sensors.

Camera response prediction for various capture settings using the spectral sensitivity and crosstalk model

In this paper, a camera response formation model is proposed to accurately predict the responses of images captured under various exposure settings. Differing from earlier works that estimated the camera relative spectral sensitivity, our model constructs the physical spectral sensitivity curves and device-dependent parameters that convert the absolute spectral radiances of target surfaces to the camera readout responses. With this model, the camera responses to miscellaneous combinations of surfaces and illuminants could be accurately predicted. Thus, creating an "imaging simulator" by using the colorimetric and photometric research based on the cameras would be of great convenience.

Simple and effective calculations about spectral power distributions of outdoor light sources for computer vision

The spectral power distributions (SPD) of outdoor light sources are not constant over time and atmospheric conditions, which causes the appearance variation of a scene and common natural illumination phenomena, such as twilight, shadow, and haze/fog. Calculating the SPD of outdoor light sources at different time (or zenith angles) and under different atmospheric conditions is of interest to physically-based vision. In this paper, for computer vision and its applications, we propose a feasible, simple, and effective SPD calculating method based on analyzing the transmittance functions of absorption and scattering along the path of solar radiation through the atmosphere in the visible spectrum. Compared with previous SPD calculation methods, our model has less parameters and is accurate enough to be directly applied in computer vision. It can be applied in computer vision tasks including spectral inverse calculation, lighting conversion, and shadowed image processing. The experimental results of the applications demonstrate that our calculation methods have practical values in computer vision. It establishes a bridge between image and physical environmental information, e.g., time, location, and weather conditions.