Light Field Imaging Based Accurate Image Specular Highlight Removal

SpecSeg Network for Specular Highlight Detection and Segmentation in Real-World Images

Specular highlights detection and removal in images is a fundamental yet non-trivial problem of interest. Most modern techniques proposed are inadequate at dealing with real-world images taken under uncontrolled conditions with the presence of complex textures, multiple objects, and bright colours, resulting in reduced accuracy and false positives. To detect specular pixels in a wide variety of real-world images independent of the number, colour, or type of illuminating source, we propose an efficient Specular Segmentation (SpecSeg) network based on the U-net architecture that is expeditious to train on nominal-sized datasets. The proposed network can detect pixels strongly affected by specular highlights with a high degree of precision, as shown by comparison with the state-of-the-art methods. The technique proposed is trained on publicly available datasets and tested using a large selection of real-world images with highly encouraging results.

Specular highlight removal based on an iterative light field method

In this paper, a novel, to the best of our knowledge, iterative approach for highlight removal is proposed using lenselet-based plenoptic cameras without multiple exposures. An unsupervised k-means clustering approach that relates unsaturated pixels to chromatic dispersion based on the intrinsic decomposition and dichromatic reflection model is proposed to recover unsaturated highlights. Meanwhile, an adaptive direction method along with a Gaussian probability distribution model is designed to recover the saturated highlights. Finally, a method that combines the specular residual ratio with information entropy is built to quantitatively evaluate the quality of highlight removal. Generally, our method not only fully removes specular highlights, but also has low spatial complexity of image acquisition, more stability, and outstanding restoration for complex scenes.

Specular Reflections Removal for Endoscopic Image Sequences With Adaptive-RPCA Decomposition

Specular reflections (i.e., highlight) always exist in endoscopic images, and they can severely disturb surgeons' observation and judgment. In an augmented reality (AR)-based surgery navigation system, the highlight may also lead to the failure of feature extraction or registration. In this paper, we propose an adaptive robust principal component analysis (Adaptive-RPCA) method to remove the specular reflections in endoscopic image sequences. It can iteratively optimize the sparse part parameter during RPCA decomposition. In this new approach, we first adaptively detect the highlight image based on pixels. With the proposed distance metric algorithm, it then automatically measures the similarity distance between the sparse result image and the detected highlight image. Finally, the low-rank and sparse results are obtained by enforcing the similarity distance between the two types of images to fall within a certain range. Our method has been verified by multiple different types of endoscopic image sequences in minimally invasive surgery (MIS). The experiments and clinical blind tests demonstrate that the new Adaptive-RPCA method can obtain the optimal sparse decomposition parameters directly and can generate robust highlight removal results. Compared with the state-of-the-art approaches, the proposed method not only achieves the better highlight removal results but also can adaptively process image sequences.