Fourier Spectral Filter Array for Optimal Multispectral Imaging

Unsupervised Spectral Demosaicing With Lightweight Spectral Attention Networks

This paper presents a deep learning-based spectral demosaicing technique trained in an unsupervised manner. Many existing deep learning-based techniques relying on supervised learning with synthetic images, often underperform on real-world images, especially as the number of spectral bands increases. This paper presents a comprehensive unsupervised spectral demosaicing (USD) framework based on the characteristics of spectral mosaic images. This framework encompasses a training method, model structure, transformation strategy, and a well-fitted model selection strategy. To enable the network to dynamically model spectral correlation while maintaining a compact parameter space, we reduce the complexity and parameters of the spectral attention module. This is achieved by dividing the spectral attention tensor into spectral attention matrices in the spatial dimension and spectral attention vector in the channel dimension. This paper also presents Mosaic 25 , a real 25-band hyperspectral mosaic image dataset featuring various objects, illuminations, and materials for benchmarking purposes. Extensive experiments on both synthetic and real-world datasets demonstrate that the proposed method outperforms conventional unsupervised methods in terms of spatial distortion suppression, spectral fidelity, robustness, and computational cost. Our code and dataset are publicly available at https://github.com/polwork/Unsupervised-Spectral-Demosaicing.

Unsupervised design for broadband multispectral and polarization filter array patterns

Imaging multiple wavelength and polarization components is problematic due to the complexity of equipment and the increase in the number of imaging shots, so imaging using filter arrays with various patterns has been widely reported from elemental research to practical applications. Most of them use bandpass filters with different center wavelengths for each pixel. Recently, however, filter arrays with multimodal transmission characteristics have been proposed using photonic crystals or Fabry-Perot filters. In any of these methods, the design of the filter array arrangement pattern is important to improve the quality of the captured image, as well as the improvement of the demosaicking algorithm. One way to design a filter array pattern is to minimize the mean squared error (MSE) between the ideal image and the demosaicked image. However, the more multidimensional the imaging components, the more difficult it becomes to collect training data. In such cases, it is necessary to empirically determine candidate transmission characteristics and patterns of filter arrays. In this study, we propose a method for evaluating filter array patterns without using any training data in the design of filter arrays for multispectral and polarization imaging. The proposed method estimates the MSE by approximating the autocorrelation matrix without using image data by expressing the imaging model as a linear forward problem and the demosaicking as a linear inverse problem. Since this method can be applied not only to ideal bandpass filter arrangements, but also to multispectral filter arrays with multimodal spectral transmission characteristics and even multispectral polarization filter arrays with different extinction ratios at different wavelengths, we will show that image quality can be improved over empirical arrangements by evaluating these patterns and by testing examples of optimal designs using genetic algorithms.

Handheld snapshot multi-spectral camera at tens-of-megapixel resolution

Multi-spectral imaging is a fundamental tool characterizing the constituent energy of scene radiation. However, current multi-spectral video cameras cannot scale up beyond megapixel resolution due to optical constraints and the complexity of the reconstruction algorithms. To circumvent the above issues, we propose a tens-of-megapixel handheld multi-spectral videography approach (THETA), with a proof-of-concept camera achieving 65-megapixel videography of 12 wavebands within visible light range. The high performance is brought by multiple designs: We propose an imaging scheme to fabricate a thin mask for encoding spatio-spectral data using a conventional film camera. Afterwards, a fiber optic plate is introduced for building a compact prototype supporting pixel-wise encoding with a large space-bandwidth product. Finally, a deep-network-based algorithm is adopted for large-scale multi-spectral data decoding, with the coding pattern specially designed to facilitate efficient coarse-to-fine model training. Experimentally, we demonstrate THETA's advantageous and wide applications in outdoor imaging of large macroscopic scenes.

Opti-MSFA: a toolbox for generalized design and optimization of multispectral filter arrays

Multispectral imaging captures spatial information across a set of discrete spectral channels and is widely utilized across diverse applications such as remote sensing, industrial inspection, and biomedical imaging. Multispectral filter arrays (MSFAs) are filter mosaics integrated atop image sensors that facilitate cost-effective, compact, snapshot multispectral imaging. MSFAs are pre-configured based on application-where filter channels are selected corresponding to targeted absorption spectra-making the design of optimal MSFAs vital for a given application. Despite the availability of many design and optimization approaches for spectral channel selection and spatial arrangement, major limitations remain. There are few robust approaches for joint spectral-spatial optimization, techniques are typically only applicable to limited datasets and most critically, are not available for general use and improvement by the wider community. Here, we reconcile current MSFA design techniques and present Opti-MSFA: a Python-based open-access toolbox for the centralized design and optimization of MSFAs. Opti-MSFA incorporates established spectral-spatial optimization algorithms, such as gradient descent and simulated annealing, multispectral-RGB image reconstruction, and is applicable to user-defined input of spatial-spectral datasets or imagery. We demonstrate the utility of the toolbox by comparing against other published MSFAs using the standard hyperspectral datasets Samson and Jasper Ridge, and further show application on experimentally acquired fluorescence imaging data. In conjunction with end-user input and collaboration, we foresee the continued development of Opti-MSFA for the benefit of the wider research community.

Opti-MSFA: a toolbox for generalized design and optimization of multispectral filter arrays

Multispectral imaging captures spatial information across a set of discrete spectral channels and is widely utilized across diverse applications such as remote sensing, industrial inspection, and biomedical imaging. Multispectral filter arrays (MSFAs) are filter mosaics integrated atop image sensors that facilitate cost-effective, compact, snapshot multispectral imaging. MSFAs are pre-configured based on application-where filter channels are selected corresponding to targeted absorption spectra-making the design of optimal MSFAs vital for a given application. Despite the availability of many design and optimization approaches for spectral channel selection and spatial arrangement, major limitations remain. There are few robust approaches for joint spectral-spatial optimization, techniques are typically only applicable to limited datasets and most critically, are not available for general use and improvement by the wider community. Here, we reconcile current MSFA design techniques and present Opti-MSFA: a Python-based open-access toolbox for the centralized design and optimization of MSFAs. Opti-MSFA incorporates established spectral-spatial optimization algorithms, such as gradient descent and simulated annealing, multispectral-RGB image reconstruction, and is applicable to user-defined input of spatial-spectral datasets or imagery. We demonstrate the utility of the toolbox by comparing against other published MSFAs using the standard hyperspectral datasets Samson and Jasper Ridge, and further show application on experimentally acquired fluorescence imaging data. In conjunction with end-user input and collaboration, we foresee the continued development of Opti-MSFA for the benefit of the wider research community.

Camouflaged Target Detection Based on Snapshot Multispectral Imaging

The spectral information contained in the hyperspectral images (HSI) distinguishes the intrinsic properties of a target from the background, which is widely used in remote sensing. However, the low imaging speed and high data redundancy caused by the high spectral resolution of imaging spectrometers limit their application in scenarios with the real-time requirement. In this work, we achieve the precise detection of camouflaged targets based on snapshot multispectral imaging technology and band selection methods in urban-related scenes. Specifically, the camouflaged target detection algorithm combines the constrained energy minimization (CEM) algorithm and the improved maximum between-class variance (OTSU) algorithm (t-OTSU), which is proposed to obtain the initial target detection results and adaptively segment the target region. Moreover, an object region extraction (ORE) algorithm is proposed to obtain a complete target contour that improves the target detection capability of multispectral images (MSI). The experimental results show that the proposed algorithm has the ability to detect different camouflaged targets by using only four bands. The detection accuracy is above 99%, and the false alarm rate is below 0.2%. The research achieves the effective detection of camouflaged targets and has the potential to provide a new means for real-time multispectral sensing in complex scenes.

Generalized MSFA Engineering With Structural and Adaptive Nonlocal Demosaicing

The emerging multispectral-filter-array (MSFA) cameras require generalized demosaicing for MSFA engineering. The existing interpolation, compressive sensing and deep learning based methods suffer from either limited reconstruction accuracy or poor generalization. In this work, we report a generalized demosaicing method with structural and adaptive nonlocal optimization, enabling boosted reconstruction accuracy for different MSFAs. The advantages lie in the following three aspects. First, the nonlocal low-rank optimization is applied and extended to the multiple spatial-spectral-temporal dimensions to exploit more crucial details. Second, the block matching accuracy is promoted by employing a novel structural similarity metric instead of the conventional Euclidean distance. Third, the running efficiency is boosted by an adaptive iteration strategy. We built a prototype system to capture raw mosaic images under different MSFAs, and used the technique as an off-the-shelf tool to demonstrate MSFA engineering. The experiments show that the binary tree (BT) based filter array produces higher accuracy than the random and regular ones for different number of channels.

General demosaicking for multispectral polarization filter arrays using total generalized variation and weighted tensor nuclear norm minimization

We focus on a demosaicking method for recovering multispectral polarization images (MSPIs) from a single image captured by a multispectral polarization filter array (MSPFA). Since the image captured by the MSPFA can be represented by a linear model, an algorithm to solve the inverse problem can be designed to enable general-purpose demosaicking regardless of the transmission characteristics and patterns of the MSPFA. Thus, we propose a method for demosaicking MSPIs by solving an inverse problem that introduces the decorrelated vectorial total generalized variation (D-VTGV) and weighted tensor nuclear norm (WTNN) regularization functions. D-VTGV evaluates the edge-preserving property in the spatial direction while preserving the correlation between bands and polarization angles, while WTNN exploits the correlation and low-rank property in nonlocal regions of the image to perform proper texture restoration and denoising. The experimental results show that the proposed method can restore images well for both the ideal MSPFA and an MSPFA manufactured from photonic crystals.

Alignment-free filter array: Snapshot multispectral polarization imaging based on a Voronoi-like random photonic crystal filter

We develop a photonic crystal filter with a new structure and propose a method to realize a snapshot multispectral polarization camera by mounting the filter on a monochrome imager with no requirement for a specific alignment. The developed filter is based on the Voronoi structure, which forms multilayered photonic crystals with random wave-like structures in each of the Voronoi cells. Because the transmission characteristics of the multilayered photonic crystal can be controlled simply by changing the microstructure, there is no need to change the manufacturing process and materials for each Voronoi cell. Furthermore, the Voronoi cell is randomly distributed so that the filter can be junctioned with the imager at arbitrary positions and angles without the need to position the filter during mounting, although it requires measurement of the camera characteristics and an image restoration process after filter mounting. In this experiment, we evaluated to reconstruct spectra as well as linearly polarized components and RGB images in the visible wavelength range from a single exposure image.

Snapshot multispectral imaging using a pixel-wise polarization color image sensor

This study proposes a new imaging technique for snapshot multispectral imaging in which a multispectral image was captured using an imaging lens that combines a set of multiple spectral filters and polarization filters, as well as a pixel-wise color polarization image sensor. The author produced a prototype nine-band multispectral camera system that covered from visible to near-infrared regions and was very compact. The camera's spectral performance was evaluated using experiments; moreover, the camera was used to detect the freshness of food and the activity of wild plants and was mounted on a vehicle to obtain a multispectral video while driving.

Demosaicking Using a Spatial Reference Image for an Anti-Aliasing Multispectral Filter Array

Multispectral imaging with a multispectral filter array (MSFA) facilitates snapshot imaging; however, a demosaicking process is required to estimate a fully defined multispectral image based on undersampled sensor data. Undersampling induces aliasing and adverse artifacts in the reconstructed image. To solve this problem, Jia et al. proposed the Fourier spectral filter array (FSFA), which can reduce aliasing. In this paper, we analyze the FSFA and a more generalized anti-aliasing MSFA, and we identify the property that makes MSFAs anti-aliasing. Furthermore, we propose a novel demosaicking method that is a hybrid of frequency-decomposition-based and compressive-sensing-based demosaicking. Anti-aliasing MSFAs enable demosaicking to comprehend the precise spatial structures of an image. The image assists our proposed method in precisely reconstructing images using compressive sensing. Our experimental results demonstrated that the proposed method performs better than the existing demosaicking methods, especially in terms of spatial reconstruction.

Multispectral video acquisition using spectral sweep camera

This paper proposes a novel multispectral video acquisition method for dynamic scenes by using the Spectral-Sweep camera. To fully utilize the redundancies of multispectral videos in the spatial, temporal and spectral dimensions, we propose a Complex Optical Flow (COF) method that could extract the spatial and spectral signal variations between adjacent spectral-sweep frames. A complex L ₁-norm constrained optimization algorithm is proposed to compute the COF maps, with which we recover the entire multispectral video by temporally propagating the captured spectral-sweep frames under the guidance of reconstructed COF maps. We demonstrate the promising accuracy of reconstructing full spatial and temporal sensor resolution multispectral videos with our method both quantitatively and qualitatively. Compared with state-of-the-art multispectral imagers, our computational multispectral imaging system can significantly reduce the hardware complexities, while achieves comparable or even better performance.

Optimized Multi-Spectral Filter Arrays for Spectral Reconstruction

Multispectral filter array (MSFA)-based imaging is a compact, practical technique for snapshot spectral image capturing and reconstruction. The imaging and reconstruction quality is highly influenced by the spectral sensitivities and spatial arrangement of channels on MSFAs, and the used reconstruction method. In order to design a MSFA with high imaging capacity, we propose a sparse representation based approach to optimize spectral sensitivities and spatial arrangement of MSFAs. The proposed approach first overall models the various errors associated with spectral reconstruction, and then uses a global heuristic searching method to optimize MSFAs via minimizing the estimated error of MSFAs. Our MSFA optimization method can select filters from off-the-shelf candidate filter sets while assigning the selected filters to the designed MSFA. Experimental results on three datasets show that the proposed method is more efficient, flexible, and can design MSFAs with lower spectral construction errors when compared with existing state-of-the-art methods. The MSFAs designed by our method show better performance than others even using different spectral reconstruction methods.

Snapshot multispectral polarization imaging using a photonic crystal filter array

A new filter array and a demosaicking method for snapshot multispectral polarization imaging are proposed in this paper. The proposed filter array is a thin-film wavy multilayer structure regarded as a photonic crystal that can be fabricated using the autocloning method. The multispectral polarization filter array is developed by altering the wave structure of the photonic crystal at each pixel. In addition, we propose a demosaicking method for multispectral polarization images by considering snapshot imaging as a linear model. In the experiments, we evaluated the recovered spectrum error in some color charts and showed various demosaicked images such as multispectral polarization images, specific-band degree of linear polarization images, polarized RGB images, and non-polarized RGB images.

Optimized Multi-Spectral Filter Array Based Imaging of Natural Scenes

Multi-spectral imaging using a camera with more than three channels is an efficient method to acquire and reconstruct spectral data and is used extensively in tasks like object recognition, relighted rendering, and color constancy. Recently developed methods are used to only guide content-dependent filter selection where the set of spectral reflectances to be recovered are known a priori. We present the first content-independent spectral imaging pipeline that allows optimal selection of multiple channels. We also present algorithms for optimal placement of the channels in the color filter array yielding an efficient demosaicing order resulting in accurate spectral recovery of natural reflectance functions. These reflectance functions have the property that their power spectrum statistically exhibits a power-law behavior. Using this property, we propose power-law based error descriptors that are minimized to optimize the imaging pipeline. We extensively verify our models and optimizations using large sets of commercially available wide-band filters to demonstrate the greater accuracy and efficiency of our multi-spectral imaging pipeline over existing methods.

High Dynamic Range Spectral Imaging Pipeline For Multispectral Filter Array Cameras

Spectral filter arrays imaging exhibits a strong similarity with color filter arrays. This permits us to embed this technology in practical vision systems with little adaptation of the existing solutions. In this communication, we define an imaging pipeline that permits high dynamic range (HDR)-spectral imaging, which is extended from color filter arrays. We propose an implementation of this pipeline on a prototype sensor and evaluate the quality of our implementation results on real data with objective metrics and visual examples. We demonstrate that we reduce noise, and, in particular we solve the problem of noise generated by the lack of energy balance. Data are provided to the community in an image database for further research.

Focal plane filter array engineering I: rectangular lattices

Focal planes arrays (FPA) measure values proportional to an integrated irradiance with little sensitivity to wavelength or polarization in the optical wavelength range. The measurement of spectral properties is often achieved via a spatially varying color filter array. Recently spatially varying polarization filter arrays have been used to extract polarization information. Although measurement of color and polarization utilize separate physical methods, the underlying design and engineering methodology is linked. In this communication we derive a formalism which can be used to design any type of periodic filter array on a rectangular lattice. A complete system description can be obtained from the number of unit cells, the pixel shape, and the unit cell geometry. This formalism can be used to engineer the channel structure for any type of periodic tiling of a rectangular lattice for any type of optical filter array yielding irradiance measurements.