The design and evaluation of a generic method for generating mosaicked multispectral filter arrays

A Multi-Shot Approach for Spatial Resolution Improvement of Multispectral Images from an MSFA Sensor

Multispectral imaging technology has advanced significantly in recent years, allowing single-sensor cameras with multispectral filter arrays to be used in new scene acquisition applications. Our camera, developed as part of the European CAVIAR project, uses an eight-band MSFA to produce mosaic images that can be decomposed into eight sparse images. These sparse images contain only pixels with similar spectral properties and null pixels. A demosaicing process is then applied to obtain fully defined images. However, this process faces several challenges in rendering fine details, abrupt transitions, and textured regions due to the large number of null pixels in the sparse images. Therefore, we propose a sparse image composition method to overcome these challenges by reducing the number of null pixels in the sparse images. To achieve this, we increase the number of snapshots by simultaneously introducing a spatial displacement of the sensor by one to three pixels on the horizontal and/or vertical axes. The set of snapshots acquired provides a multitude of mosaics representing the same scene with a redistribution of pixels. The sparse images from the different mosaics are added together to get new composite sparse images in which the number of null pixels is reduced. A bilinear demosaicing approach is applied to the composite sparse images to obtain fully defined images. Experimental results on images projected onto the response of our MSFA filter show that our composition method significantly improves image spatial resolution and minimizes reconstruction errors while preserving spectral fidelity.

InSPECtor: an end-to-end design framework for compressive pixelated hyperspectral instruments

Classic designs of hyperspectral instrumentation densely sample the spatial and spectral information of the scene of interest. Data may be compressed after the acquisition. In this paper, we introduce a framework for the design of an optimized, micropatterned snapshot hyperspectral imager that acquires an optimized subset of the spatial and spectral information in the scene. The data is thereby already compressed at the sensor level but can be restored to the full hyperspectral data cube by the jointly optimized reconstructor. This framework is implemented with TensorFlow and makes use of its automatic differentiation for the joint optimization of the layout of the micropatterned filter array as well as the reconstructor. We explore the achievable compression ratio for different numbers of filter passbands, number of scanning frames, and filter layouts using data collected by the Hyperscout instrument. We show resulting instrument designs that take snapshot measurements without losing significant information while reducing the data volume, acquisition time, or detector space by a factor of 40 as compared to classic, dense sampling. The joint optimization of a compressive hyperspectral imager design and the accompanying reconstructor provides an avenue to substantially reduce the data volume from hyperspectral imagers.

Unsupervised design for broadband multispectral and polarization filter array patterns

Imaging multiple wavelength and polarization components is problematic due to the complexity of equipment and the increase in the number of imaging shots, so imaging using filter arrays with various patterns has been widely reported from elemental research to practical applications. Most of them use bandpass filters with different center wavelengths for each pixel. Recently, however, filter arrays with multimodal transmission characteristics have been proposed using photonic crystals or Fabry-Perot filters. In any of these methods, the design of the filter array arrangement pattern is important to improve the quality of the captured image, as well as the improvement of the demosaicking algorithm. One way to design a filter array pattern is to minimize the mean squared error (MSE) between the ideal image and the demosaicked image. However, the more multidimensional the imaging components, the more difficult it becomes to collect training data. In such cases, it is necessary to empirically determine candidate transmission characteristics and patterns of filter arrays. In this study, we propose a method for evaluating filter array patterns without using any training data in the design of filter arrays for multispectral and polarization imaging. The proposed method estimates the MSE by approximating the autocorrelation matrix without using image data by expressing the imaging model as a linear forward problem and the demosaicking as a linear inverse problem. Since this method can be applied not only to ideal bandpass filter arrangements, but also to multispectral filter arrays with multimodal spectral transmission characteristics and even multispectral polarization filter arrays with different extinction ratios at different wavelengths, we will show that image quality can be improved over empirical arrangements by evaluating these patterns and by testing examples of optimal designs using genetic algorithms.

Design of a Dual-Mode Multispectral Filter Array

Multispectral imaging is valuable in many vision-related fields as it provides an additional modality to observe the world. Cameras equipped with multispectral filter arrays (MSFAs) are typically impractical for everyday use due to their intractable demosaicking and chromatic reproduction processes, which restrict their applicability beyond academic research. In this work, a novel MSFA design is proposed to enable dual-mode imaging for multispectral cameras. In addition to a conventional multispectral image, the camera is also able to produce a Bayer-formed RGB image from a single shot by grouping and merging adjacent pixels in the proposed MSFA, making it suitable for scenarios where display-ready RGB images are required. Furthermore, a two-stage optimization scheme is implemented to jointly optimize objective functions for both imaging modes. The evaluation results on multiple datasets suggest that the proposed MSFA design is able to simultaneously achieve competitive spectral reconstruction accuracy compared to elaborate multispectral cameras and chromatic accuracy compared to commercial RGB cameras.

Modular snapshot multispectral-panchromatic imager (MSPI) with customized filter arrays

As one of the simplest methods to construct snapshot spectral imagers, multispectral filter array (MSFA) has been applied to commercial miniatured spectral imagers. While most of them have fixed configurations of spectral channels, lacking flexibility and replaceability. Moreover, conventional MSFA only comprises filtering channels but lacks the panchromatic channel which is essential in detecting dim and indistinct objects. Here, we propose a modular assembly method for snapshot imager which can simultaneously acquire the object's multispectral and panchromatic information based on a customized filter array. The multispectral-panchromatic filter array is batch fabricated and integrated with the imaging senor through a modular mode. Five-band spectral images and a broadband intensity image can be efficiently acquired in a single snapshot photographing. The efficacy and accuracy of the imager are experimentally verified in imaging and spectral measurements. Owing to the modular architecture, our proposed assembly method owns the advantages of compactness, simple assembling, rapid replacement, and customized designing, which overcomes the expensiveness and complexity of scientific-level snapshot spectral imaging systems.

Batch fabrication and compact integration of customized multispectral filter arrays towards snapshot imaging

Snapshot multispectral imaging (MSI) has been widely employed in the rapid visual inspection by virtues of the non-invasive detection mode and short integration time. As the critical functional elements of snapshot MSI, narrowband, customizable, and pixel-level multispectral filter arrays (MSFAs) that are compatible with imaging sensors are difficult to be efficiently manufactured. Meanwhile, monolithically integrating MSFAs into snapshot multispectral imagers still remains challenging considering the strict alignment precision. Here, we propose a cost-efficient, wafer-level, and customized approach for fabricating transmissive MSFAs based on Fabry-Perot structures, both in the pixel-level and window-tiled configuration, by utilizing the conventional lithography combined with the deposition method. The MSFA chips own a total dimension covering the area of 4.8 mm × 3.6 mm with 4 × 4 bands, possessing the capability to maintain narrow line widths (∼25 nm) across the whole visible frequencies. After the compact integration with the imaging sensor, the MSFAs are validated to be effective in filtering and target identification. Our proposed fabrication method and imaging mode show great potentials to be an alternative to MSFAs production and MSI, by reducing both complexity and cost of manufacturing, while increasing flexibility and customization of imaging system.

Multispectral imaging via nanostructured random broadband filtering

It is a challenge to acquire a snapshot image of very high resolutions in both spectral and spatial domain via a single short exposure. In this setting one cannot trade time for spectral resolution, such as via spectral bands scanning. Cameras of color filter arrays (CFA) (e.g., the Bayer mosaic) cannot obtain high spectral resolution. To overcome these difficulties, we propose a new multispectral imaging system that makes random linear broadband measurements of the spectrum via a nanostructured multispectral filter array (MSFA). These MSFA random measurements can be used by sparsity-based recovery algorithms to achieve much higher spectral resolution than conventional CFA cameras, without sacrificing spatial resolution. The key innovation is to jointly exploit both spatial and spectral sparsity properties that are inherent to spectral irradiance of natural objects. Experimental results establish the superior performance of the proposed multispectral imaging system over existing ones.

Demosaicking Using a Spatial Reference Image for an Anti-Aliasing Multispectral Filter Array

Multispectral imaging with a multispectral filter array (MSFA) facilitates snapshot imaging; however, a demosaicking process is required to estimate a fully defined multispectral image based on undersampled sensor data. Undersampling induces aliasing and adverse artifacts in the reconstructed image. To solve this problem, Jia et al. proposed the Fourier spectral filter array (FSFA), which can reduce aliasing. In this paper, we analyze the FSFA and a more generalized anti-aliasing MSFA, and we identify the property that makes MSFAs anti-aliasing. Furthermore, we propose a novel demosaicking method that is a hybrid of frequency-decomposition-based and compressive-sensing-based demosaicking. Anti-aliasing MSFAs enable demosaicking to comprehend the precise spatial structures of an image. The image assists our proposed method in precisely reconstructing images using compressive sensing. Our experimental results demonstrated that the proposed method performs better than the existing demosaicking methods, especially in terms of spatial reconstruction.

Optimized Multi-Spectral Filter Arrays for Spectral Reconstruction

Multispectral filter array (MSFA)-based imaging is a compact, practical technique for snapshot spectral image capturing and reconstruction. The imaging and reconstruction quality is highly influenced by the spectral sensitivities and spatial arrangement of channels on MSFAs, and the used reconstruction method. In order to design a MSFA with high imaging capacity, we propose a sparse representation based approach to optimize spectral sensitivities and spatial arrangement of MSFAs. The proposed approach first overall models the various errors associated with spectral reconstruction, and then uses a global heuristic searching method to optimize MSFAs via minimizing the estimated error of MSFAs. Our MSFA optimization method can select filters from off-the-shelf candidate filter sets while assigning the selected filters to the designed MSFA. Experimental results on three datasets show that the proposed method is more efficient, flexible, and can design MSFAs with lower spectral construction errors when compared with existing state-of-the-art methods. The MSFAs designed by our method show better performance than others even using different spectral reconstruction methods.

Optimized Multi-Spectral Filter Array Based Imaging of Natural Scenes

Multi-spectral imaging using a camera with more than three channels is an efficient method to acquire and reconstruct spectral data and is used extensively in tasks like object recognition, relighted rendering, and color constancy. Recently developed methods are used to only guide content-dependent filter selection where the set of spectral reflectances to be recovered are known a priori. We present the first content-independent spectral imaging pipeline that allows optimal selection of multiple channels. We also present algorithms for optimal placement of the channels in the color filter array yielding an efficient demosaicing order resulting in accurate spectral recovery of natural reflectance functions. These reflectance functions have the property that their power spectrum statistically exhibits a power-law behavior. Using this property, we propose power-law based error descriptors that are minimized to optimize the imaging pipeline. We extensively verify our models and optimizations using large sets of commercially available wide-band filters to demonstrate the greater accuracy and efficiency of our multi-spectral imaging pipeline over existing methods.

Adaptive Residual Interpolation for Color and Multispectral Image Demosaicking

Color image demosaicking for the Bayer color filter array is an essential image processing operation for acquiring high-quality color images. Recently, residual interpolation (RI)-based algorithms have demonstrated superior demosaicking performance over conventional color difference interpolation-based algorithms. In this paper, we propose adaptive residual interpolation (ARI) that improves existing RI-based algorithms by adaptively combining two RI-based algorithms and selecting a suitable iteration number at each pixel. These are performed based on a unified criterion that evaluates the validity of an RI-based algorithm. Experimental comparisons using standard color image datasets demonstrate that ARI can improve existing RI-based algorithms by more than 0.6 dB in the color peak signal-to-noise ratio and can outperform state-of-the-art algorithms based on training images. We further extend ARI for a multispectral filter array, in which more than three spectral bands are arrayed, and demonstrate that ARI can achieve state-of-the-art performance also for the task of multispectral image demosaicking.

Adaptive Multispectral Demosaicking Based on Frequency-Domain Analysis of Spectral Correlation

Color filter array (CFA) interpolation, or three-band demosaicking, is a process of interpolating the missing color samples in each band to reconstruct a full color image. In this paper, we are concerned with the challenging problem of multispectral demosaicking, where each band is significantly undersampled due to the increment in the number of bands. Specifically, we demonstrate a frequency-domain analysis of the subsampled color-difference signal and observe that the conventional assumption of highly correlated spectral bands for estimating undersampled components is not precise. Instead, such a spectral correlation assumption is image dependent and rests on the aliasing interferences among the various color-difference spectra. To address this problem, we propose an adaptive spectral-correlation-based demosaicking (ASCD) algorithm that uses a novel anti-aliasing filter to suppress these interferences, and we then integrate it with an intra-prediction scheme to generate a more accurate prediction for the reconstructed image. Our ASCD is computationally very simple, and exploits the spectral correlation property much more effectively than the existing algorithms. Experimental results conducted on two data sets for multispectral demosaicking and one data set for CFA demosaicking demonstrate that the proposed ASCD outperforms the state-of-the-art algorithms.

Universal Demosaicking of Color Filter Arrays

A large number of color filter arrays (CFAs), periodic or aperiodic, have been proposed. To reconstruct images from all different CFAs and compare their imaging quality, a universal demosaicking method is needed. This paper proposes a new universal demosaicking method based on inter-pixel chrominance capture and optimal demosaicking transformation. It skips the commonly used step to estimate the luminance component at each pixel, and thus, avoids the associated estimation error. Instead, we directly use the acquired CFA color intensity at each pixel as an input component. Two independent chrominance components are estimated at each pixel based on the inter-pixel chrominance in the window, which is captured with the difference of CFA color values between the pixel of interest and its neighbors. Two mechanisms are employed for the accurate estimation: distance-related and edge-sensing weighting to reflect the confidence levels of the inter-pixel chrominance components, and pseudoinverse-based estimation from the components in a window. Then from the acquired CFA color component and two estimated chrominance components, the three primary colors are reconstructed by a linear color transform, which is optimized for the least transform error. Our experiments show that the proposed method is much better than other published universal demosaicking methods.

Spectral Characterization of a Prototype SFA Camera for Joint Visible and NIR Acquisition

Multispectral acquisition improves machine vision since it permits capturing more information on object surface properties than color imaging. The concept of spectral filter arrays has been developed recently and allows multispectral single shot acquisition with a compact camera design. Due to filter manufacturing difficulties, there was, up to recently, no system available for a large span of spectrum, i.e., visible and Near Infra-Red acquisition. This article presents the achievement of a prototype of camera that captures seven visible and one near infra-red bands on the same sensor chip. A calibration is proposed to characterize the sensor, and images are captured. Data are provided as supplementary material for further analysis and simulations. This opens a new range of applications in security, robotics, automotive and medical fields.

Fourier Spectral Filter Array for Optimal Multispectral Imaging

Limitations to existing multispectral imaging modalities include speed, cost, range, spatial resolution, and application-specific system designs that lack versatility of the hyperspectral imaging modalities. In this paper, we propose a novel general-purpose single-shot passive multispectral imaging modality. Central to this design is a new type of spectral filter array (SFA) based not on the notion of spatially multiplexing narrowband filters, but instead aimed at enabling single-shot Fourier transform spectroscopy. We refer to this new SFA pattern as Fourier SFA, and we prove that this design solves the problem of optimally sampling the hyperspectral image data.

A practical one-shot multispectral imaging system using a single image sensor

Single-sensor imaging using the Bayer color filter array (CFA) and demosaicking is well established for current compact and low-cost color digital cameras. An extension from the CFA to a multispectral filter array (MSFA) enables us to acquire a multispectral image in one shot without increased size or cost. However, multispectral demosaicking for the MSFA has been a challenging problem because of very sparse sampling of each spectral band in the MSFA. In this paper, we propose a high-performance multispectral demosaicking algorithm, and at the same time, a novel MSFA pattern that is suitable for our proposed algorithm. Our key idea is the use of the guided filter to interpolate each spectral band. To generate an effective guide image, in our proposed MSFA pattern, we maintain the sampling density of the G -band as high as the Bayer CFA, and we array each spectral band so that an adaptive kernel can be estimated directly from raw MSFA data. Given these two advantages, we effectively generate the guide image from the most densely sampled G -band using the adaptive kernel. In the experiments, we demonstrate that our proposed algorithm with our proposed MSFA pattern outperforms existing algorithms and provides better color fidelity compared with a conventional color imaging system with the Bayer CFA. We also show some real applications using a multispectral camera prototype we built.

Multispectral filter arrays: recent advances and practical implementation

Thanks to some technical progress in interferencefilter design based on different technologies, we can finally successfully implement the concept of multispectral filter array-based sensors. This article provides the relevant state-of-the-art for multispectral imaging systems and presents the characteristics of the elements of our multispectral sensor as a case study. The spectral characteristics are based on two different spatial arrangements that distribute eight different bandpass filters in the visible and near-infrared area of the spectrum. We demonstrate that the system is viable and evaluate its performance through sensor spectral simulation.

Spectrogenic imaging: a novel approach to multispectral imaging in an uncontrolled environment

Increasing the number of imaging channels beyond the conventional three has been shown to be beneficial for a wide range of applications. However, it is mostly limited to imaging in a controlled environment, where the capture environment (illuminant) is known a priori. We propose here a novel system and methodology for multispectral imaging in an uncontrolled environment. Two images of a scene, a normal RGB and a filtered RGB are captured. The illuminant under which an image is captured is estimated using a chromagenic based algorithm, and the multispectral system is calibrated automatically using the estimated illuminant. A 6-band multispectral image of a scene is obtained from the two RGB images. The spectral reflectances of the scene are then estimated using an appropriate spectral estimation method. The proposed concept and methodology is generic one, as it is valid in whatever way we acquire the two images of a scene. A system that can acquire two images of a scene can be realized, for instance in two shots using a digital camera and a filter, or in a single shot using a stereo camera, or a custom color filter array design. Simulation experiments using a stereo camera based system confirms the effectiveness of the proposed method. This could be useful in many imaging applications and computer vision.

Single sensor that outputs narrowband multispectral images

We report the work of developing a hand-held (or miniaturized), low-cost, stand-alone, real-time-operation, narrow bandwidth multispectral imaging device for the detection of early stage pressure ulcers.

Spatio-spectral color filter array design for optimal image recovery

In digital imaging applications, data are typically obtained via a spatial subsampling procedure implemented as a color filter array-a physical construction whereby only a single color value is measured at each pixel location. Owing to the growing ubiquity of color imaging and display devices, much recent work has focused on the implications of such arrays for subsequent digital processing, including in particular the canonical demosaicking task of reconstructing a full color image from spatially subsampled and incomplete color data acquired under a particular choice of array pattern. In contrast to the majority of the demosaicking literature, we consider here the problem of color filter array design and its implications for spatial reconstruction quality. We pose this problem formally as one of simultaneously maximizing the spectral radii of luminance and chrominance channels subject to perfect reconstruction, and-after proving sub-optimality of a wide class of existing array patterns-provide a constructive method for its solution that yields robust, new panchromatic designs implementable as subtractive colors. Empirical evaluations on multiple color image test sets support our theoretical results, and indicate the potential of these patterns to increase spatial resolution for fixed sensor size, and to contribute to improved reconstruction fidelity as well as significantly reduced hardware complexity.

Binary tree-based generic demosaicking algorithm for multispectral filter arrays

In this paper, we extend the idea of using mosaicked color filter array (CFA) in color imaging, which has been widely adopted in the digital color camera industry, to the use of multispectral filter array (MSFA) in multispectral imaging. The filter array technique can help reduce the cost, achieve exact registration, and improve the robustness of the imaging system. However, the extension from CFA to MSFA is not straightforward. First, most CFAs only deal with a few bands (3 or 4) within the narrow visual spectral region, while the design of MSFA needs to handle the arrangement of multiple bands (more than 3) across a much wider spectral range. Second, most existing CFA demosaicking algorithms assume the fixed Bayer CFA and are confined to properties only existed in the color domain. Therefore, they cannot be directly applied to multispectral demosaicking. The main challenges faced in multispectral demosaicking is how to design a generic algorithm that can handle the more diversified MSFA patterns, and how to improve performance with a coarser spatial resolution and a less degree of spectral correlation. In this paper, we present a binary tree based generic demosaicking method. Two metrics are used to evaluate the generic algorithm, including the root mean-square error (RMSE) for reconstruction performance and the classification accuracy for target discrimination performance. Experimental results show that the demosaicked images present low RMSE (less than 7) and comparable classification performance as original images. These results support that MSFA technique can be applied to multispectral imaging with unique advantages.