An Analysis of Visual Adaptation and Contrast Perception for Tone Mapping

High dynamic range image compression based on the multi-peak S-shaped tone curve

Tone mapping methods aim to compress the high dynamic range (HDR) images so that they can be displayed on common devices. The tone curve plays a key role in many tone mapping methods, which can directly adjust the range of the HDR image. The S-shaped tone curves can produce impressive performances due to their flexibility. However, the conventional S-shaped tone curve in tone mapping methods is single and had the problem of excessive compressing of the dense grayscale areas, resulting in the loss of details in this area, and insufficient compressing of the sparse grayscale areas, resulting in low contrast of tone mapped image. This paper proposes a multi-peak S-shaped (MPS) tone curve to address these problems. Specifically, the grayscale interval of the HDR image is divided according to the significant peak and valley distribution of the grayscale histogram, and each interval is tone mapped by an S-shaped tone curve. We further propose an adaptive S-shaped tone curve based on the luminance adaptation mechanism of the human visual system, which can effectively reduce the compression in the dense grayscale areas and increase the compression in the sparse grayscale areas, preserving details while improving the contrast of tone mapped images. Experiments show that our MPS tone curve replaces the single S-shaped tone curve in relevant methods for better performance and outperforms the state-of-the-art tone mapping methods.

Photographic Reproduction and Enhancement Using HVS-Based Modified Histogram Equalization

Photographic reproduction and enhancement is challenging because it requires the preservation of all the visual information during the compression of the dynamic range of the input image. This paper presents a cascaded-architecture-type reproduction method that can simultaneously enhance local details and retain the naturalness of original global contrast. In the pre-processing stage, in addition to using a multiscale detail injection scheme to enhance the local details, the Stevens effect is considered for adapting different luminance levels and normally compressing the global feature. We propose a modified histogram equalization method in the reproduction stage, where individual histogram bin widths are first adjusted according to the property of overall image content. In addition, the human visual system (HVS) is considered so that a luminance-aware threshold can be used to control the maximum permissible width of each bin. Then, the global tone is modified by performing histogram equalization on the output modified histogram. Experimental results indicate that the proposed method can outperform the five state-of-the-art methods in terms of visual comparisons and several objective image quality evaluations.

Vision Models for Wide Color Gamut Imaging in Cinema

Gamut mapping is the problem of transforming the colors of image or video content so as to fully exploit the color palette of the display device where the content will be shown, while preserving the artistic intent of the original content's creator. In particular, in the cinema industry, the rapid advancement in display technologies has created a pressing need to develop automatic and fast gamut mapping algorithms. In this article, we propose a novel framework that is based on vision science models, performs both gamut reduction and gamut extension, is of low computational complexity, produces results that are free from artifacts and outperforms state-of-the-art methods according to psychophysical tests. Our experiments also highlight the limitations of existing objective metrics for the gamut mapping problem.

A Biological Retina Inspired Tone Mapping Processor for High-Speed and Energy-Efficient Image Enhancement

In this work, a biological retina inspired tone mapping processor for high-speed and energy-efficient image enhancement has been proposed. To achieve high throughput and high energy efficiency, several hardware design techniques have been proposed, including data partition based parallel processing with S-shape sliding, adjacent frame feature sharing, multi-layer convolution pipelining, and convolution filter compression with zero skipping convolution. Implemented on a Xilinx's Virtex7 FPGA, the proposed design achieves a high throughput of 189 frames per second for 1024 × 768 RGB images while consuming 819 mW. Compared with several state-of-the-art tone mapping processors, the proposed design shows higher throughput and energy efficiency. It is suitable for high-speed and energy-constrained image enhancement applications.

Fast Single-Image HDR Tone-Mapping by Avoiding Base Layer Extraction

The tone-mapping algorithm compresses the high dynamic range (HDR) information into the standard dynamic range for regular devices. An ideal tone-mapping algorithm reproduces the HDR image without losing any vital information. The usual tone-mapping algorithms mostly deal with detail layer enhancement and gradient-domain manipulation with the help of a smoothing operator. However, these approaches often have to face challenges with over enhancement, halo effects, and over-saturation effects. To address these challenges, we propose a two-step solution to perform a tone-mapping operation using contrast enhancement. Our method improves the performance of the camera response model by utilizing the improved adaptive parameter selection and weight matrix extraction. Experiments show that our method performs reasonably well for overexposed and underexposed HDR images without producing any ringing or halo effects.

Photographic High-Dynamic-Range Scalar Visualization

We propose a photographic method to show scalar values of high dynamic range (HDR) by color mapping for 2D visualization. We combine (1) tone-mapping operators that transform the data to the display range of the monitor while preserving perceptually important features, based on a systematic evaluation, and (2) simulated glares that highlight high-value regions. Simulated glares are effective for highlighting small areas (of a few pixels) that may not be visible with conventional visualizations; through a controlled perception study, we confirm that glare is preattentive. The usefulness of our overall photographic HDR visualization is validated through the feedback of expert users.

Estimation of gamma-corrected exposure time ratio in multi-exposure images for removal of moving objects

The constructed high-dynamic-range image from merging standard low-dynamic-range images with different camera exposures contains ghost-like artifacts caused by moving objects in the scene. We present a method to utilize the gamma-corrected exposure time ratio between multi-exposure images for removal of moving objects. Between each consecutive image pair in multi-exposure images, the ratio of their exposure times is computed and raised to the power gamma, and this value is used as a cue to detect the pixels corresponding to the moving objects. We propose a method to estimate this ratio from the observed image intensity values, in case the exposure time information or gamma value is unknown. Then the moving objects in multi-exposure images are removed by replacing the intensity values of the detected moving pixels with their expected background values. Experimental results show that the proposed method could remove fast-moving objects from the original multi-exposure images and construct a ghost-free high-dynamic-range image.

A Benchmark for Edge-Preserving Image Smoothing

Edge-preserving image smoothing is an important step for many low-level vision problems. Though many algorithms have been proposed, there are several difficulties hindering its further development. First, most existing algorithms cannot perform well on a wide range of image contents using a single parameter setting. Second, the performance evaluation of edge-preserving image smoothing remains subjective, and there lacks a widely accepted datasets to objectively compare the different algorithms. To address these issues and further advance the state of the art, in this work we propose a benchmark for edge-preserving image smoothing. This benchmark includes an image dataset with groundtruth image smoothing results as well as baseline algorithms that can generate competitive edge-preserving smoothing results for a wide range of image contents. The established dataset contains 500 training and testing images with a number of representative visual object categories, while the baseline methods in our benchmark are built upon representative deep convolutional network architectures, on top of which we design novel loss functions well suited for edge-preserving image smoothing. The trained deep networks run faster than most state-of-the-art smoothing algorithms with leading smoothing results both qualitatively and quantitatively. The benchmark will be made publicly accessible.

Statistics of natural images as a function of dynamic range

The statistics of real world images have been extensively investigated, but in virtually all cases using only low dynamic range image databases. The few studies that have considered high dynamic range (HDR) images have performed statistical analyses categorizing images as HDR according to their creation technique, and not to the actual dynamic range of the underlying scene. In this study we demonstrate, using a recent HDR dataset of natural images, that the statistics of the image as received at the camera sensor change dramatically with dynamic range, with particularly strong correlations with dynamic range being observed for the median, standard deviation, skewness, and kurtosis, while the one over frequency relationship for the power spectrum breaks down for images with a very high dynamic range, in practice making HDR images not scale invariant. Effects are also noted in the derivative statistics, the single pixel histograms, and the Haar wavelet analysis. However, we also show that after some basic early transforms occurring within the eye (light scatter, nonlinear photoreceptor response, center-surround modulation) the statistics of the resulting images become virtually independent from the dynamic range, which would allow them to be processed more efficiently by the human visual system.

A Weighted Histogram-Based Tone Mapping Algorithm for CT Images

Computed Tomography (CT) images have a high dynamic range, which makes visualization challenging. Histogram equalization methods either use spatially invariant weights or limited kernel size due to the complexity of pairwise contribution calculation. We present a weighted histogram equalization-based tone mapping algorithm which utilizes Fast Fourier Transform for distance-dependent contribution calculation and distance-based weights. The weights follow power-law without distance-based cut-off. The resulting images have good local contrast without noticeable artefacts. The results are compared to eight popular tone mapping operators.

Which tone-mapping operator is the best? A comparative study of perceptual quality

Tone-mapping operators (TMOs) are designed to generate perceptually similar low-dynamic-range images from high-dynamic-range ones. We studied the performance of 15 TMOs in two psychophysical experiments where observers compared the digitally generated tone-mapped images to their corresponding physical scenes. All experiments were performed in a controlled environment, and the setups were designed to emphasize different image properties: in the first experiment we evaluated the local relationships among intensity levels, and in the second one we evaluated global visual appearance among physical scenes and tone-mapped images, which were presented side by side. We ranked the TMOs according to how well they reproduced the results obtained in the physical scene. Our results show that ranking position clearly depends on the adopted evaluation criteria, which implies that, in general, these tone-mapping algorithms consider either local or global image attributes but rarely both. Regarding the question of which TMO is the best, KimKautz ["Consistent tone reproduction," in Proceedings of Computer Graphics and Imaging (2008)] and Krawczyk ["Lightness perception in tone reproduction for high dynamic range images," in Proceedings of Eurographics (2005), p. 3] obtained the better results across the different experiments. We conclude that more thorough and standardized evaluation criteria are needed to study all the characteristics of TMOs, as there is ample room for improvement in future developments.

Minimalistic optic flow sensors applied to indoor and outdoor visual guidance and odometry on a car-like robot

Here we present a novel bio-inspired optic flow (OF) sensor and its application to visual  guidance and odometry on a low-cost car-like robot called BioCarBot. The minimalistic OF sensor was robust to high-dynamic-range lighting conditions and to various visual patterns encountered thanks to its M²APIX auto-adaptive pixels and the new cross-correlation OF algorithm implemented. The low-cost car-like robot estimated its velocity and steering angle, and therefore its position and orientation, via an extended Kalman filter (EKF) using only two downward-facing OF sensors and the Ackerman steering model. Indoor and outdoor experiments were carried out in which the robot was driven in the closed-loop mode based on the velocity and steering angle estimates. The experimental results obtained show that our novel OF sensor can deliver high-frequency measurements ([Formula: see text]) in a wide OF range (1.5-[Formula: see text]) and in a 7-decade high-dynamic light level range. The OF resolution was constant and could be adjusted as required (up to [Formula: see text]), and the OF precision obtained was relatively high (standard deviation of [Formula: see text] with an average OF of [Formula: see text], under the most demanding lighting conditions). An EKF-based algorithm gave the robot's position and orientation with a relatively high accuracy (maximum errors outdoors at a very low light level: [Formula: see text] and [Formula: see text] over about [Formula: see text] and [Formula: see text]) despite the low-resolution control systems of the steering servo and the DC motor, as well as a simplified model identification and calibration. Finally, the minimalistic OF-based odometry results were compared to those obtained using measurements based on an inertial measurement unit (IMU) and a motor's speed sensor.

Tone-Mapped Mean-Shift Based Environment Map Sampling

In this paper, we present a novel approach for environment map sampling, which is an effective and pragmatic technique to reduce the computational cost of realistic rendering and get plausible rendering images. The proposed approach exploits the advantage of adaptive mean-shift image clustering with aid of tone-mapping, yielding oversegmented strata that have uniform intensities and capture shapes of light regions. The resulted strata, however, have unbalanced importance metric values for rendering, and the strata number is not user-controlled. To handle these issues, we develop an adaptive split-and-merge scheme that refines the strata and obtains a better balanced strata distribution. Compared to the state-of-the-art methods, our approach achieves comparable and even better rendering quality in terms of SSIM, RMSE and HDRVDP2 image quality metrics. Experimental results further show that our approach is more robust to the variation of viewpoint, environment rotation, and sample number.

High dynamic range image compression by optimizing tone mapped image quality index

Tone mapping operators (TMOs) aim to compress high dynamic range (HDR) images to low dynamic range (LDR) ones so as to visualize HDR images on standard displays. Most existing TMOs were demonstrated on specific examples without being thoroughly evaluated using well-designed and subject-validated image quality assessment models. A recently proposed tone mapped image quality index (TMQI) made one of the first attempts on objective quality assessment of tone mapped images. Here, we propose a substantially different approach to design TMO. Instead of using any predefined systematic computational structure for tone mapping (such as analytic image transformations and/or explicit contrast/edge enhancement), we directly navigate in the space of all images, searching for the image that optimizes an improved TMQI. In particular, we first improve the two building blocks in TMQI—structural fidelity and statistical naturalness components—leading to a TMQI-II metric. We then propose an iterative algorithm that alternatively improves the structural fidelity and statistical naturalness of the resulting image. Numerical and subjective experiments demonstrate that the proposed algorithm consistently produces better quality tone mapped images even when the initial images of the iteration are created by the most competitive TMOs. Meanwhile, these results also validate the superiority of TMQI-II over TMQI.

A bio-inspired analog silicon retina with Michaelis-Menten auto-adaptive pixels sensitive to small and large changes in light

In this paper, we present: (i) a novel analog silicon retina featuring auto-adaptive pixels that obey the Michaelis-Menten law, i.e. V=V(m) I(n)/I(n)+σ(n); (ii) a method of characterizing silicon retinas, which makes it possible to accurately assess the pixels' response to transient luminous changes in a ±3-decade range, as well as changes in the initial steady-state intensity in a 7-decade range. The novel pixel, called M(2)APix, which stands for Michaelis-Menten Auto-Adaptive Pixel, can auto-adapt in a 7-decade range and responds appropriately to step changes up to ±3 decades in size without causing any saturation of the Very Large Scale Integration (VLSI) transistors. Thanks to the intrinsic properties of the Michaelis-Menten equation, the pixel output always remains within a constant limited voltage range. The range of the Analog to Digital Converter (ADC) was therefore adjusted so as to obtain a Least Significant Bit (LSB) voltage of 2.35mV and an effective resolution of about 9 bits. The results presented here show that the M(2)APix produced a quasi-linear contrast response once it had adapted to the average luminosity. Differently to what occurs in its biological counterparts, neither the sensitivity to changes in light nor the contrast response of the M(2)APix depend on the mean luminosity (i.e. the ambient lighting conditions). Lastly, a full comparison between the M(2)APix and the Delbrück auto-adaptive pixel is provided.

From image processing to computational neuroscience: a neural model based on histogram equalization

There are many ways in which the human visual system works to reduce the inherent redundancy of the visual information in natural scenes, coding it in an efficient way. The non-linear response curves of photoreceptors and the spatial organization of the receptive fields of visual neurons both work toward this goal of efficient coding. A related, very important aspect is that of the existence of post-retinal mechanisms for contrast enhancement that compensate for the blurring produced in early stages of the visual process. And alongside mechanisms for coding and wiring efficiency, there is neural activity in the human visual cortex that correlates with the perceptual phenomenon of lightness induction. In this paper we propose a neural model that is derived from an image processing technique for histogram equalization, and that is able to deal with all the aspects just mentioned: this new model is able to predict lightness induction phenomena, and improves the efficiency of the representation by flattening both the histogram and the power spectrum of the image signal.

A modified model of the just noticeable depth difference and its application to depth sensation enhancement

The just noticeable depth difference (JNDD) describes the threshold of human perception of the difference in the depth. In flat-panel-based three-dimensional (3-D) displays, the JNDD is typically measured by changing the depth difference between displayed image objects until the difference is perceivable. However, not only the depth, but also the perceived size changes when the depth difference increases. In this paper, we present a modified JNDD measurement method that adjusts the physical size of the object such that the perceived size of the object is maintained. We then apply the proposed JNDD measurement method to depth sensation enhancement. When the depth value difference between the objects is increased to enable the viewer to perceive the depth difference, the size of the objects is adjusted to maintain the perceived size of the objects. In addition, since the size change of the objects can produce a whole region, a depth-adaptive hole-inpainting technique is proposed to compensate for the hole region with high accuracy. The experimental results demonstrate the effectiveness of the proposed method.