Attention Guided Low-Light Image Enhancement with a Large Scale Low-Light Simulation Dataset

Salient object detection in low-light RGB-T scene via spatial-frequency cues mining

Low-light conditions pose significant challenges to vision tasks, such as salient object detection (SOD), due to insufficient photons. Light-insensitive RGB-T SOD models mitigate the above problems to some extent, but they are limited in performance as they only focus on spatial feature fusion while ignoring the frequency discrepancy. To this end, we propose an RGB-T SOD model by mining spatial-frequency cues, called SFMNet, for low-light scenes. Our SFMNet consists of spatial-frequency feature exploration (SFFE) modules and spatial-frequency feature interaction (SFFI) modules. To be specific, the SFFE module aims to separate spatial-frequency features and adaptively extract high and low-frequency features. Moreover, the SFFI module integrates cross-modality and cross-domain information to capture effective feature representations. By deploying both modules in a top-down pathway, our method generates high-quality saliency predictions. Furthermore, we construct the first low-light RGB-T SOD dataset as a benchmark for evaluating performance. Extensive experiments demonstrate that our SFMNet can achieve higher accuracy than the existing models for low-light scenes.

Low-Light Sparse Polarization Demosaicing Network (LLSPD-Net): Polarization Image Demosaicing Based on Stokes Vector Completion in Low-Light Environment

Polarization imaging has achieved a wide range of applications in military and civilian fields such as camouflage detection and autonomous driving. However, when the imaging environment involves a low-light condition, the number of photons is low and the photon transmittance of the conventional Division-of-Focal-Plane (DoFP) structure is small. Therefore, the traditional demosaicing methods are often used to deal with the serious noise and distortion generated by polarization demosaicing in low-light environment. Based on the aforementioned issues, this paper proposes a model called Low-Light Sparse Polarization Demosaicing Network (LLSPD-Net) for simulating a sparse polarization sensor acquisition of polarization images in low-light environments. The model consists of two parts: an intensity image enhancement network and a Stokes vector complementation network. In this work, the intensity image enhancement network is used to enhance low-light images and obtain high-quality RGB images, while the Stokes vector is used to complement the network. We discard the traditional idea of polarization intensity image interpolation and instead design a polarization demosaicing method with Stokes vector complementation. By using the enhanced intensity image as a guide, the completion of the Stokes vector is achieved. In addition, to train our network, we collected a dataset of paired color polarization images that includes both low-light and regular-light conditions. A comparison with state-of-the-art methods on both self-constructed and publicly available datasets reveals that our model outperforms traditional low-light image enhancement demosaicing methods in both qualitative and quantitative experiments.

Content-illumination coupling guided low-light image enhancement network

Current low-light enhancement algorithms fail to suppress noise when enhancing brightness, and may introduces structural distortion and color distortion caused by halos or artifacts. This paper proposes a content-illumination coupling guided low-light image enhancement network (CICGNet), it develops a truss topology based on Retinex as backbone to decompose low-light image component in an end-to-end way. The preservation of content features and the enhancement of illumination features are carried out along with depth and width direction of the truss topology. Each submodule uses the same resolution input and output to avoid the introduction of noise. Illumination component prevents misestimation of global and local illumination by using pre- and post-activation features at different depth levels, this way could avoid possible halos and artifacts. The network progressively enhances the illumination component and maintains the content component stage-by-stage. The proposed algorithm demonstrates better performance compared with advanced attention-based low-light enhancement algorithms and state-of-the-art image restoration algorithms. We also perform extensive ablation studies and demonstrate the impact of low-light enhancement algorithm on the downstream task of computer vision. Code is available at: https://github.com/Ruini94/CICGNet .

Super resolution label-free dark-field microscopy by deep learning

Dark-field microscopy (DFM) is a powerful label-free and high-contrast imaging technique due to its ability to reveal features of transparent specimens with inhomogeneities. However, owing to the Abbe's diffraction limit, fine structures at sub-wavelength scale are difficult to resolve. In this work, we report a single image super resolution DFM scheme using a convolutional neural network (CNN). A U-net based CNN is trained with a dataset which is numerically simulated based on the forward physical model of the DFM. The forward physical model described by the parameters of the imaging setup connects the object ground truths and dark field images. With the trained network, we demonstrate super resolution dark field imaging of various test samples with twice resolution improvement. Our technique illustrates a promising deep learning approach to double the resolution of DFM without any hardware modification.

DRLIE: Flexible Low-Light Image Enhancement via Disentangled Representations

Low-light image enhancement (LIME) aims to convert images with unsatisfied lighting into desired ones. Different from existing methods that manipulate illumination in uncontrollable manners, we propose a flexible framework to take user-specified guide images as references to improve the practicability. To achieve the goal, this article models an image as the combination of two components, that is, content and exposure attribute, from an information decoupling perspective. Specifically, we first adopt a content encoder and an attribute encoder to disentangle the two components. Then, we combine the scene content information of the low-light image with the exposure attribute of the guide image to reconstruct the enhanced image through a generator. Extensive experiments on public datasets demonstrate the superiority of our approach over state-of-the-art alternatives. Particularly, the proposed method allows users to enhance images according to their preferences, by providing specific guide images. Our source code and the pretrained model are available at https://github.com/Linfeng-Tang/DRLIE.

Hierarchical Optimization-Derived Learning

In recent years, by utilizing optimization techniques to formulate the propagation of deep model, a variety of so-called Optimization-Derived Learning (ODL) approaches have been proposed to address diverse learning and vision tasks. Although having achieved relatively satisfying practical performance, there still exist fundamental issues in existing ODL methods. In particular, current ODL methods tend to consider model constructing and learning as two separate phases, and thus fail to formulate their underlying coupling and depending relationship. In this work, we first establish a new framework, named Hierarchical ODL (HODL), to simultaneously investigate the intrinsic behaviors of optimization-derived model construction and its corresponding learning process. Then we rigorously prove the joint convergence of these two sub-tasks, from the perspectives of both approximation quality and stationary analysis. To our best knowledge, this is the first theoretical guarantee for these two coupled ODL components: optimization and learning. We further demonstrate the flexibility of our framework by applying HODL to challenging learning tasks, which have not been properly addressed by existing ODL methods. Finally, we conduct extensive experiments on both synthetic data and real applications in vision and other learning tasks to verify the theoretical properties and practical performance of HODL in various application scenarios.

Image De-Raining Transformer

Existing deep learning based de-raining approaches have resorted to the convolutional architectures. However, the intrinsic limitations of convolution, including local receptive fields and independence of input content, hinder the model's ability to capture long-range and complicated rainy artifacts. To overcome these limitations, we propose an effective and efficient transformer-based architecture for the image de-raining. First, we introduce general priors of vision tasks, i.e., locality and hierarchy, into the network architecture so that our model can achieve excellent de-raining performance without costly pre-training. Second, since the geometric appearance of rainy artifacts is complicated and of significant variance in space, it is essential for de-raining models to extract both local and non-local features. Therefore, we design the complementary window-based transformer and spatial transformer to enhance locality while capturing long-range dependencies. Besides, to compensate for the positional blindness of self-attention, we establish a separate representative space for modeling positional relationship, and design a new relative position enhanced multi-head self-attention. In this way, our model enjoys powerful abilities to capture dependencies from both content and position, so as to achieve better image content recovery while removing rainy artifacts. Experiments substantiate that our approach attains more appealing results than state-of-the-art methods quantitatively and qualitatively.

Learning With Nested Scene Modeling and Cooperative Architecture Search for Low-Light Vision

Images captured from low-light scenes often suffer from severe degradations, including low visibility, color casts, intensive noises, etc. These factors not only degrade image qualities, but also affect the performance of downstream Low-Light Vision (LLV) applications. A variety of deep networks have been proposed to enhance the visual quality of low-light images. However, they mostly rely on significant architecture engineering and often suffer from the high computational burden. More importantly, it still lacks an efficient paradigm to uniformly handle various tasks in the LLV scenarios. To partially address the above issues, we establish Retinex-inspired Unrolling with Architecture Search (RUAS), a general learning framework, that can address low-light enhancement task, and has the flexibility to handle other challenging downstream vision tasks. Specifically, we first establish a nested optimization formulation, together with an unrolling strategy, to explore underlying principles of a series of LLV tasks. Furthermore, we design a differentiable strategy to cooperatively search specific scene and task architectures for RUAS. Last but not least, we demonstrate how to apply RUAS for both low- and high-level LLV applications (e.g., enhancement, detection and segmentation). Extensive experiments verify the flexibility, effectiveness, and efficiency of RUAS.

Multi-Branch and Progressive Network for Low-Light Image Enhancement

Low-light images incur several complicated degradation factors such as poor brightness, low contrast, color degradation, and noise. Most previous deep learning-based approaches, however, only learn the mapping relationship of single channel between the input low-light images and the expected normal-light images, which is insufficient enough to deal with low-light images captured under uncertain imaging environment. Moreover, too deeper network architecture is not conducive to recover low-light images due to extremely low values in pixels. To surmount aforementioned issues, in this paper we propose a novel multi-branch and progressive network (MBPNet) for low-light image enhancement. To be more specific, the proposed MBPNet is comprised of four different branches which build the mapping relationship at different scales. The followed fusion is performed on the outputs obtained from four different branches for the final enhanced image. Furthermore, to better handle the difficulty of delivering structural information of low-light images with low values in pixels, a progressive enhancement strategy is applied in the proposed method, where four convolutional long short-term memory networks (LSTM) are embedded in four branches and an recurrent network architecture is developed to iteratively perform the enhancement process. In addition, a joint loss function consisting of the pixel loss, the multi-scale perceptual loss, the adversarial loss, the gradient loss, and the color loss is framed to optimize the model parameters. To evaluate the effectiveness of proposed MBPNet, three popularly used benchmark databases are used for both quantitative and qualitative assessments. The experimental results confirm that the proposed MBPNet obviously outperforms other state-of-the-art approaches in terms of quantitative and qualitative results. The code will be available at https://github.com/kbzhang0505/MBPNet.

Dental image enhancement network for early diagnosis of oral dental disease

Intelligent robotics and expert system applications in dentistry suffer from identification and detection problems due to the non-uniform brightness and low contrast in the captured images. Moreover, during the diagnostic process, exposure of sensitive facial parts to ionizing radiations (e.g., X-Rays) has several disadvantages and provides a limited angle for the view of vision. Capturing high-quality medical images with advanced digital devices is challenging, and processing these images distorts the contrast and visual quality. It curtails the performance of potential intelligent and expert systems and disincentives the early diagnosis of oral and dental diseases. The traditional enhancement methods are designed for specific conditions, and network-based methods rely on large-scale datasets with limited adaptability towards varying conditions. This paper proposed a novel and adaptive dental image enhancement strategy based on a small dataset and proposed a paired branch Denticle-Edification network (Ded-Net). The input dental images are decomposed into reflection and illumination in a multilayer Denticle network (De-Net). The subsequent enhancement operations are performed to remove the hidden degradation of reflection and illumination. The adaptive illumination consistency is maintained through the Edification network (Ed-Net). The network is regularized following the decomposition congruity of the input data and provides user-specific freedom of adaptability towards desired contrast levels. The experimental results demonstrate that the proposed method improves visibility and contrast and preserves the edges and boundaries of the low-contrast input images. It proves that the proposed method is suitable for intelligent and expert system applications for future dental imaging.

Robust contrast enhancement method using a retinex model with adaptive brightness for detection applications

Low light image enhancement with adaptive brightness, color and contrast preservation in degraded visual conditions (e.g., extreme dark background, lowlight, back-light, mist. etc.) is becoming more challenging for machine cognition applications than anticipated. A realistic image enhancement framework should preserve brightness and contrast in robust scenarios. The extant direct enhancement methods amplify objectionable structure and texture artifacts, whereas network-based enhancement approaches are based on paired or large-scale training datasets, raising fundamental concerns about their real-world applicability. This paper presents a new framework to get deep into darkness in degraded visual conditions following the fundamental of retinex-based image decomposition. We separate the reflection and illumination components to perform independent weighted enhancement operations on each component to preserve the visual details with a balance of brightness and contrast. A comprehensive weighting strategy is proposed to constrain image decomposition while disrupting the irregularities of high frequency reflection and illumination to improve the contrast. At the same time, we propose to guide the illumination component with a high-frequency component for structure and texture preservation in degraded visual conditions. Unlike existing approaches, the proposed method works regardless of the training data type (i.e., low light, normal light, or normal and low light pairs). A deep into darkness network (D2D-Net) is proposed to maintain the visual balance of smoothness without compromising the image quality. We conduct extensive experiments to demonstrate the superiority of the proposed enhancement. We test the performance of our method for object detection tasks in extremely dark scenarios. Experimental results demonstrate that our method maintains the balance of visual smoothness, making it more viable for future interactive visual applications.

End-to-End Retinex-Based Illumination Attention Low-Light Enhancement Network for Autonomous Driving at Night

Low-light image enhancement is a preprocessing work for many recognition and tracking tasks for autonomous driving at night. It needs to handle various factors simultaneously including uneven lighting, low contrast, and artifacts. We propose a novel end-to-end Retinex-based illumination attention low-light enhancement network. Specifically, our proposed method adopts multibranch architecture to extract rich features for different depth levels. Meanwhile, we consider the features from different scales in built-in illumination attention module. We encode reflectance features and illumination features into latent space based on Retinex in each submodule, which could cater for highly ill-posed image decomposition tasks. It aims to enhance the desired illumination features under different receptive fields. Subsequently, we propose a memory gate mechanism to learn adaptively long-term and short-term memory. Their weight could control how many high-level and low-level features should be reserved. This method could improve the image quality from both different feature scales and feature levels. Comprehensive experiments on BDD10K and cityscapes datasets demonstrate that our proposed method outperforms various types of methods in terms of visual quality and quantitative metrics. We also show that our proposed method has certain antinoise capability and generalizes well without fine-tuning when dealing with unseen images. Meanwhile, our restoration performance is comparable to that of advanced computationally intensive models.1

Successive Low-Light Image Enhancement Using an Image-Adaptive Mask

Low-light images are obtained in dark environments or in environments where there is insufficient light. Because of this, low-light images have low intensity values and dimmed features, making it difficult to directly apply computer vision or image recognition software to them. Therefore, to use computer vision processing on low-light images, an image improvement procedure is needed. There have been many studies on how to enhance low-light images. However, some of the existing methods create artifact and distortion effects in the resulting images. To improve low-light images, their contrast should be stretched naturally according to their features. This paper proposes the use of a low-light image enhancement method utilizing an image-adaptive mask that is composed of an image-adaptive ellipse. As a result, the low-light regions of the image are stretched and the bright regions are enhanced in a way that appears natural by an image-adaptive mask. Moreover, images that have been enhanced using the proposed method are color balanced, as this method has a color compensation effect due to the use of an image-adaptive mask. As a result, the improved image can better reflect the image’s subject, such as a sunset, and appears natural. However, when low-light images are stretched, the noise elements are also enhanced, causing part of the enhanced image to look dim and hazy. To tackle this issue, this paper proposes the use of guided image filtering based on using triple terms for the image-adaptive value. Images enhanced by the proposed method look natural and are objectively superior to those enhanced via other state-of-the-art methods.

GAN-Based Video Denoising with Attention Mechanism for Field-Applicable Pig Detection System

Infrared cameras allow non-invasive and 24 h continuous monitoring. Thus, they are widely used in automatic pig monitoring, which is essential to maintain the profitability and sustainability of intensive pig farms. However, in practice, impurities such as insect secretions continuously pollute camera lenses. This causes problems with IR reflections, which can seriously affect pig detection performance. In this study, we propose a noise-robust, real-time pig detection system that can improve accuracy in pig farms where infrared cameras suffer from the IR reflection problem. The system consists of a data collector to collect infrared images, a preprocessor to transform noisy images into clean images, and a detector to detect pigs. The preprocessor embeds a multi-scale spatial attention module in U-net and generative adversarial network (GAN) models, enabling the model to pay more attention to the noisy area. The GAN model was trained on paired sets of clean data and data with simulated noise. It can operate in a real-time and end-to-end manner. Experimental results show that the proposed preprocessor was able to significantly improve the average precision of pig detection from 0.766 to 0.906, with an additional execution time of only 4.8 ms on a PC environment.

M. DR. Low-Light Image Enhancement Based on Generative Adversarial Network

Image enhancement is considered to be one of the complex tasks in image processing. When the images are captured under dim light, the quality of the images degrades due to low visibility degenerating the vision-based algorithms' performance that is built for very good quality images with better visibility. After the emergence of a deep neural network number of methods has been put forward to improve images captured under low light. But, the results shown by existing low-light enhancement methods are not satisfactory because of the lack of effective network structures. A low-light image enhancement technique (LIMET) with a fine-tuned conditional generative adversarial network is presented in this paper. The proposed approach employs two discriminators to acquire a semantic meaning that imposes the obtained results to be realistic and natural. Finally, the proposed approach is evaluated with benchmark datasets. The experimental results highlight that the presented approach attains state-of-the-performance when compared to existing methods. The models' performance is assessed using Visual Information Fidelitysse, which assesses the generated image's quality over the degraded input. VIF obtained for different datasets using the proposed approach are 0.709123 for LIME dataset, 0.849982 for DICM dataset, 0.619342 for MEF dataset.

Low-Light Image Enhancement Based on Multi-Path Interaction

Due to the non-uniform illumination conditions, images captured by sensors often suffer from uneven brightness, low contrast and noise. In order to improve the quality of the image, in this paper, a multi-path interaction network is proposed to enhance the R, G, B channels, and then the three channels are combined into the color image and further adjusted in detail. In the multi-path interaction network, the feature maps in several encoding–decoding subnetworks are used to exchange information across paths, while a high-resolution path is retained to enrich the feature representation. Meanwhile, in order to avoid the possible unnatural results caused by the separation of the R, G, B channels, the output of the multi-path interaction network is corrected in detail to obtain the final enhancement results. Experimental results show that the proposed method can effectively improve the visual quality of low-light images, and the performance is better than the state-of-the-art methods.

Multi-Feature Guided Low-Light Image Enhancement

Due to the characteristics of low signal-to-noise ratio and low contrast, low-light images will have problems such as color distortion, low visibility, and accompanying noise, which will cause the accuracy of the target detection problem to drop or even miss the detection target. However, recalibrating the dataset for this type of image will face problems such as increased cost or reduced model robustness. To solve this kind of problem, we propose a low-light image enhancement model based on deep learning. In this paper, the feature extraction is guided by the illumination map and noise map, and then the neural network is trained to predict the local affine model coefficients in the bilateral space. Through these methods, our network can effectively denoise and enhance images. We have conducted extensive experiments on the LOL datasets, and the results show that, compared with traditional image enhancement algorithms, the model is superior to traditional methods in image quality and speed.