Deep learning polarimetric three-dimensional integral imaging object recognition in adverse environmental conditions

Polarimetric Imaging for Robot Perception: A Review

In recent years, the integration of polarimetric imaging into robotic perception systems has increased significantly, driven by the accessibility of affordable polarimetric sensors. This technology complements traditional color imaging by capturing and analyzing the polarization characteristics of light. This additional information provides robots with valuable insights into object shape, material composition, and other properties, ultimately enabling more robust manipulation tasks. This review aims to provide a comprehensive analysis of the principles behind polarimetric imaging and its diverse applications within the field of robotic perception. By exploiting the polarization state of light, polarimetric imaging offers promising solutions to three key challenges in robot vision: Surface segmentation; depth estimation through polarization patterns; and 3D reconstruction using polarimetric data. This review emphasizes the practical value of polarimetric imaging in robotics by demonstrating its effectiveness in addressing real-world challenges. We then explore potential applications of this technology not only within the core robotics field but also in related areas. Through a comparative analysis, our goal is to elucidate the strengths and limitations of polarimetric imaging techniques. This analysis will contribute to a deeper understanding of its broad applicability across various domains within and beyond robotics.

Noise-aware infrared polarization image fusion based on salient prior with attention-guided filtering network

Infrared polarization image fusion integrates intensity and polarization information, producing a fused image that enhances visibility and captures crucial details. However, in complex environments, polarization imaging is susceptible to noise interference. Existing fusion methods typically use the infrared intensity (S0) and degree of linear polarization (DoLP) images for fusion but fail to consider the noise interference, leading to reduced performance. To cope with this problem, we propose a fusion method based on polarization salient prior, which extends DoLP by angle of polarization (AoP) and introduces polarization distance (PD) to obtain salient target features. Moreover, according to the distribution difference between S0 and DoLP features, we construct a fusion network based on attention-guided filtering, utilizing cross-attention to generate filter kernels for fusion. The quantitative and qualitative experimental results validate the effectiveness of our approach. Compared with other fusion methods, our method can effectively suppress noise interference and preserve salient target features.

Polarizer-free measurement of the full Stokes vector using a fiber-coupled superconducting nanowire single photon detector with a polarization extinction ratio of ∼2

The characterization and manipulation of polarization state at single photon level are of great importance in research fields such as quantum information processing and quantum key distribution, where photons are normally delivered using single mode optical fibers. To date, the demonstrated polarimetry measurement techniques based on a superconducting nanowire single photon detector (SNSPD) require the SNSPD to be either highly sensitive or highly insensitive to the photon's polarization state, therefore placing an unavoidable challenge on the SNSPD's design and fabrication processes. In this article, we present the development of an alternative polarimetry measurement technique, of which the stringent requirement on the SNSPD's polarization sensitivity is removed. We validate the proposed technique by a rigorous theoretical analysis and comparisons of the experimental results obtained using a fiber-coupled SNSPD with a polarization extinction ratio of ∼2 to that obtained using other well-established known methods. Based on the full Stokes data measured by the proposed technique, we also demonstrate that at the single photon level (∼ -100 dBm), the polarization state of the photon delivered to the superconducting nanowire facet plane can be controlled at will using a further developed algorithm. Note that other than the fiber-coupled SNSPD, the only component involved is a quarter-wave plate (no external polarizer is necessary), which when aligned well has a paid insertion loss less than 0.5 dB.

Single-shot Mueller-matrix imaging of zebrafish tissues: In vivo analysis of developmental and pathological features

Zebrafish is a well-established animal model for developmental and disease studies. Its optical transparency at early developmental stages allows in vivo tissues visualization. Interaction of polarized light with these tissues provides information on their structure and properties. This approach is effective for muscle tissue analysis due to its birefringence. To enable real-time Mueller-matrix characterization of unanesthetized fish, we assembled a microscope for single-shot Mueller-matrix imaging. First, we performed a continuous observation of 48 species within the period of 2 to 96 hpf and measured temporal dependencies of the polarization features in different tissues. These measurements show that hatching was accompanied by a sharp change in the angle and degree of linearly polarized light after interaction with muscles. Second, we analyzed nine species with skeletal disorders and demonstrated that the spatial distribution of light depolarization features clearly indicated them. Obtained results demonstrated that real-time Mueller-matrix imaging is a powerful tool for label-free monitoring zebrafish embryos.

Urban objects classification using Mueller matrix polarimetry and machine learning

Detecting and recognizing different kinds of urban objects is an important problem, in particular, in autonomous driving. In this context, we studied the potential of Mueller matrix polarimetry for classifying a set of relevant real-world objects: vehicles, pedestrians, traffic signs, pavements, vegetation and tree trunks. We created a database with their experimental Mueller matrices measured at 1550 nm and trained two machine learning classifiers, support vector machine and artificial neural network, to classify new samples. The overall accuracy of over 95% achieved with this approach, with either models, reveals the potential of polarimetry, specially combined with other remote sensing techniques, to enhance object recognition.

An end-to-end laser-induced damage change detection approach for optical elements via siamese network and multi-layer perceptrons

With the presence of complex background noise, parasitic light, and dust attachment, it is still a challenging issue to perform high-precision laser-induced damage change detection of optical elements in the captured optical images. For resolving this problem, this paper presents an end-to-end damage change detection model based on siamese network and multi-layer perceptrons (SiamMLP). Firstly, representative features of bi-temporal damage images are efficiently extracted by the cascaded multi-layer perceptron modules in the siamese network. After that, the extracted features are concatenated and then classified into changed and unchanged classes. Due to its concise architecture and strong feature representation ability, the proposed method obtains excellent damage change detection results efficiently and effectively. To address the unbalanced distribution of hard and easy samples, a novel metric called hard metric is introduced in this paper for quantitatively evaluating the classification difficulty degree of the samples. The hard metric assigns a classification difficulty for each individual sample to precisely adjust the loss assigned to the sample. In the training stage, a novel hard loss is presented to train the proposed model. Cooperating with the hard metric, the hard loss can up-weight the loss of hard samples and down-weight the loss of easy samples, which results in a more powerful online hard sample mining ability of the proposed model. The experimental results on two real datasets validate the effectiveness and superiority of the proposed method.

Learning feature fusion for target detection based on polarimetric imaging

We propose a polarimetric imaging processing method based on feature fusion and apply it to the task of target detection. Four images with distinct polarization orientations were used as one parallel input, and they were fused into a single feature map with richer feature information. We designed a learning feature fusion method using convolutional neural networks (CNNs). The fusion strategy was derived from training. Meanwhile, we generated a dataset involving one original image, four polarization orientation images, ground truth masks, and bounding boxes. The effectiveness of our method was compared to that of conventional deep learning methods. Experimental results revealed that our method gets a 0.80 mean average precision (mAP) and a 0.09 miss rate (MR), which are both better than the conventional deep learning method.

Three-dimensional polarimetric image restoration in low light with deep residual learning and integral imaging

Polarimetric imaging can become challenging in degraded environments such as low light illumination conditions or in partial occlusions. In this paper, we propose the denoising convolutional neural network (DnCNN) model with three-dimensional (3D) integral imaging to enhance the reconstructed image quality of polarimetric imaging in degraded environments such as low light and partial occlusions. The DnCNN is trained based on the physical model of the image capture in degraded environments to enhance the visualization of polarimetric imaging where simulated low light polarimetric images are used in the training process. The DnCNN model is experimentally tested on real polarimetric images captured in real low light environments and in partial occlusion. The performance of DnCNN model is compared with that of total variation denoising. Experimental results demonstrate that DnCNN performs better than total variation denoising for polarimetric integral imaging in terms of signal-to-noise ratio and structural similarity index measure in low light environments as well as low light environments under partial occlusions. To the best of our knowledge, this is the first report of polarimetric 3D object visualization and restoration in low light environments and occlusions using DnCNN with integral imaging. The proposed approach is also useful for 3D image restoration in conventional (non-polarimetric) integral imaging in a degraded environment.