Compressed sensing snapshot spectral imaging by a regular digital camera with an added optical diffuser

A broadband hyperspectral image sensor with high spatio-temporal resolution

Hyperspectral imaging provides high-dimensional spatial-temporal-spectral information showing intrinsic matter characteristics1-5. Here we report an on-chip computational hyperspectral imaging framework with high spatial and temporal resolution. By integrating different broadband modulation materials on the image sensor chip, the target spectral information is non-uniformly and intrinsically coupled to each pixel with high light throughput. Using intelligent reconstruction algorithms, multi-channel images can be recovered from each frame, realizing real-time hyperspectral imaging. Following this framework, we fabricated a broadband visible-near-infrared (400-1,700 nm) hyperspectral image sensor using photolithography, with an average light throughput of 74.8% and 96 wavelength channels. The demonstrated resolution is 1,024 × 1,024 pixels at 124 fps. We demonstrated its wide applications, including chlorophyll and sugar quantification for intelligent agriculture, blood oxygen and water quality monitoring for human health, textile classification and apple bruise detection for industrial automation, and remote lunar detection for astronomy. The integrated hyperspectral image sensor weighs only tens of grams and can be assembled on various resource-limited platforms or equipped with off-the-shelf optical systems. The technique transforms the challenge of high-dimensional imaging from a high-cost manufacturing and cumbersome system to one that is solvable through on-chip compression and agile computation.

Snapshot spectral imaging: from spatial-spectral mapping to metasurface-based imaging

Snapshot spectral imaging technology enables the capture of complete spectral information of objects in an extremely short period of time, offering wide-ranging applications in fields requiring dynamic observations such as environmental monitoring, medical diagnostics, and industrial inspection. In the past decades, snapshot spectral imaging has made remarkable breakthroughs with the emergence of new computational theories and optical components. From the early days of using various spatial-spectral data mapping methods, they have evolved to later attempts to encode various dimensions of light, such as amplitude, phase, and wavelength, and then computationally reconstruct them. This review focuses on a systematic presentation of the system architecture and mathematical modeling of these snapshot spectral imaging techniques. In addition, the introduction of metasurfaces expands the modulation of spatial-spectral data and brings advantages such as system size reduction, which has become a research hotspot in recent years and is regarded as the key to the next-generation snapshot spectral imaging techniques. This paper provides a systematic overview of the applications of metasurfaces in snapshot spectral imaging and provides an outlook on future directions and research priorities.

A compressive hyperspectral video imaging system using a single-pixel detector

Capturing fine spatial, spectral, and temporal information of the scene is highly desirable in many applications. However, recording data of such high dimensionality requires significant transmission bandwidth. Current computational imaging methods can partially address this challenge but are still limited in reducing input data throughput. In this paper, we report a video-rate hyperspectral imager based on a single-pixel photodetector which can achieve high-throughput hyperspectral video recording at a low bandwidth. We leverage the insight that 4-dimensional (4D) hyperspectral videos are considerably more compressible than 2D grayscale images. We propose a joint spatial-spectral capturing scheme encoding the scene into highly compressed measurements and obtaining temporal correlation at the same time. Furthermore, we propose a reconstruction method relying on a signal sparsity model in 4D space and a deep learning reconstruction approach greatly accelerating reconstruction. We demonstrate reconstruction of 128 × 128 hyperspectral images with 64 spectral bands at more than 4 frames per second offering a 900× data throughput compared to conventional imaging, which we believe is a first-of-its kind of a single-pixel-based hyperspectral imager.

Residual image recovery method based on the dual-camera design of a compressive hyperspectral imaging system

Compressive hyperspectral imaging technology can quickly detect the encoded two-dimensional measurements and reconstruct the three-dimensional hyperspectral images offline, which is of great significance for object detection and analysis. To provide more information for reconstruction and improve the reconstruction quality, some of the latest compressive hyperspectral imaging systems adopt a dual-camera design. To utilize the information from additional camera more efficiently, this paper proposes a residual image recovery method. The proposed method takes advantage of the structural similarity between the image captured by the additional camera and the hyperspectral image, combining the measurements from the additional camera and coded aperture snapshot spectral imaging (CASSI) sensor to construct an estimated hyperspectral image. Then, the component of the estimated hyperspectral image is subtracted from the measurement of the CASSI sensor to obtain the residual data. The residual data is used to reconstruct the residual hyperspectral image. Finally, the reconstructed hyperspectral image is the sum of the estimated and residual image. Compared with some state-of-the-art algorithms based on such systems, the proposed method can significantly improve the reconstruction quality of hyperspectral image.

Spectral imaging with deep learning

The goal of spectral imaging is to capture the spectral signature of a target. Traditional scanning method for spectral imaging suffers from large system volume and low image acquisition speed for large scenes. In contrast, computational spectral imaging methods have resorted to computation power for reduced system volume, but still endure long computation time for iterative spectral reconstructions. Recently, deep learning techniques are introduced into computational spectral imaging, witnessing fast reconstruction speed, great reconstruction quality, and the potential to drastically reduce the system volume. In this article, we review state-of-the-art deep-learning-empowered computational spectral imaging methods. They are further divided into amplitude-coded, phase-coded, and wavelength-coded methods, based on different light properties used for encoding. To boost future researches, we've also organized publicly available spectral datasets.

Single-shot compressive hyperspectral imaging with dispersed and undispersed light using a generally available grating

Commercially available hyperspectral cameras are useful for remote sensing, but in most cases snapshot imaging is difficult due to the need for scanning. The coded aperture snapshot spectral imager (CASSI) has been proposed to simultaneously acquire a target scene's spatial and spectral dimensional data, employing a refractive prism as a disperser. This paper proposes a CASSI-based technique using a generally available diffraction grating of a Ronchi ruling and blazed grating and its improvement using the undispersed zeroth-order light. The feasibility and performance of the proposed technique are experimentally validated, and the grating parameters are identified.

Toward Perfect Optical Diffusers: Dielectric Huygens' Metasurfaces with Critical Positional Disorder

Conventional optical diffusers, such as thick volume scatterers (Rayleigh scattering) or microstructured surface scatterers (geometric scattering), lack the potential for on-chip integration and are thus incompatible with next-generation photonic devices. Dielectric Huygens' metasurfaces, on the other hand, consist of 2D arrangements of resonant dielectric nanoparticles and therefore constitute a promising material platform for ultrathin and highly efficient photonic devices. When the nanoparticles are arranged in a random but statistically specific fashion, diffusers with exceptional properties are expected to come within reach. This work explores how dielectric Huygens' metasurfaces can implement wavelength-selective diffusers with negligible absorption losses and nearly Lambertian scattering profiles that are largely independent of the angle and polarization of incident waves. The combination of tailored positional disorder with a carefully balanced electric and magnetic response of the nanoparticles is shown to be an integral requirement for the operation as a diffuser. The proposed metasurfaces' directional scattering performance is characterized both experimentally and numerically, and their usability in wavefront-shaping applications is highlighted. Since the metasurfaces operate on the principles of Mie scattering and are embedded in a glassy environment, they may easily be incorporated in integrated photonic devices, fiber optics, or mechanically robust augmented reality displays.

DD-Net: spectral imaging from a monochromatic dispersed and diffused snapshot

We propose a snapshot spectral imaging method for the visible spectral range using a single monochromatic camera equipped with a two-dimensional (2D) binary-encoded phase diffuser placed at the pupil of the imaging lens and by resorting to deep learning (DL) algorithms for signal reconstruction. While spectral imaging was shown to be feasible using two cameras equipped with a single, one-dimensional (1D) binary diffuser and compressed sensing (CS) algorithms [Appl. Opt.59, 7853 (2020).APOPAI0003-693510.1364/AO.395541], the suggested diffuser design expands the optical response and creates optical spatial and spectral encoding along both dimensions of the image sensor. To recover the spatial and spectral information from the dispersed and diffused (DD) monochromatic snapshot, we developed novel DL algorithms, dubbed DD-Nets, which are tailored to the unique response of the optical system, which includes either a 1D or a 2D diffuser. High-quality reconstructions of the spectral cube in simulation and lab experiments are presented for system configurations consisting of a single monochromatic camera with either a 1D or a 2D diffuser. We demonstrate that the suggested system configuration with the 2D diffuser outperforms system configurations with a 1D diffuser that utilize either DL-based or CS-based algorithms for the reconstruction of the spectral cube.

100,000 frames-per-second compressive imaging with a conventional rolling-shutter camera by random point-spread-function engineering

We demonstrate an approach that allows taking videos at very high frame-rates of over 100,000 frames per second by exploiting the fast sampling rate of the standard rolling-shutter readout mechanism, common to most conventional sensors, and a compressive-sampling acquisition scheme. Our approach is directly applied to a conventional imaging system by the simple addition of a diffuser to the pupil plane that randomly encodes the entire field-of-view to each camera row, while maintaining diffraction-limited resolution. A short video is reconstructed from a single camera frame via a compressed-sensing reconstruction algorithm, exploiting the inherent sparsity of the imaged scene.

Design of binary-phase diffusers for a compressed sensing snapshot spectral imaging system with two cameras

We propose designs of pupil-domain optical diffusers for a snapshot spectral imaging system using binary-phase encoding. The suggested designs enable the creation of point-spread functions with defined optical response, having profiles that are dependent on incident wavefront wavelength. This efficient combination of dispersive and diffusive optical responses enables us to perform snapshot spectral imaging using compressed sensing algorithms while keeping a high optical throughput alongside a simple fabrication process. Experimental results are reported.

Dual-camera snapshot spectral imaging with a pupil-domain optical diffuser and compressed sensing algorithms

We propose a snapshot spectral imaging method for the visible spectral range using two digital cameras placed side-by-side: a regular red-green-blue (RGB) camera and a monochromatic camera equipped with a dispersive diffractive diffuser placed at the pupil of the imaging lens. While spectral imaging was shown to be feasible using a single monochromatic camera with a pupil diffuser [Appl. Opt.55, 432 (2016)APOPAI0003-693510.1364/AO.55.000432], adding an RGB camera provides more spatial and spectral information for stable reconstruction of the spectral cube of a scene. Results of optical experiments confirm that the combined data from the two cameras relax the complexity of the underdetermined reconstruction problem and improve the reconstructed image quality obtained using compressed sensing-based algorithms.

DeepCubeNet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks

Several hyperspectral (HS) systems based on compressive sensing (CS) theory have been presented to capture HS images with high accuracy and with a lower number of measurements than needed by conventional systems. However, the reconstruction of HS compressed measurements is time-consuming and commonly involves hyperparameter tuning per each scenario. In this paper, we introduce a Convolutional Neural Network (CNN) designed for the reconstruction of HS cubes captured with CS imagers based on spectral modulation. Our Deep Neural Network (DNN), dubbed DeepCubeNet, provides significant reduction in the reconstruction time compared to classical iterative methods. The performance of DeepCubeNet is investigated on simulated data, and we demonstrate for the first time, to the best of our knowledge, real reconstruction of CS HS measurements using DNN. We demonstrate significantly enhanced reconstruction accuracy compared to iterative CS reconstruction, as well as improvement in reconstruction time by many orders of magnitude.

Compressive ultraspectral imaging using multiscale structured illumination

We present a novel compressive spectral imaging technique that attains spatially resolved ultraspectral resolution. The technique employs a multiscale sampling technique based on the Hadamard basis for the single pixel hyperspectral imager. The proposed multiscale sampling method offers high-quality images at a low compression ratio while also facilitating a preview image at a lower resolution by using the fast Hadamard transform.

Compressive Sensing Hyperspectral Imaging by Spectral Multiplexing with Liquid Crystal

Hyperspectral (HS) imaging involves the sensing of a scene’s spectral properties, which are often redundant in nature. The redundancy of the information motivates our quest to implement Compressive Sensing (CS) theory for HS imaging. This article provides a review of the Compressive Sensing Miniature Ultra-Spectral Imaging (CS-MUSI) camera, its evolution, and its different applications. The CS-MUSI camera was designed within the CS framework and uses a liquid crystal (LC) phase retarder in order to modulate the spectral domain. The outstanding advantage of the CS-MUSI camera is that the entire HS image is captured from an order of magnitude fewer measurements of the sensor array, compared to conventional HS imaging methods.

Hyperspectral imaging system based on a single-pixel camera design for detecting differences in tissue properties

Optical spectroscopy can be used to distinguish between healthy and diseased tissue. In this study, the design and testing of a single-pixel hyperspectral imaging (HSI) system that uses autofluorescence emission from collagen (400 nm) and nicotinamide adenine dinucleotide phosphate (475 nm) along with differences in the optical reflectance spectra to differentiate between healthy and thermally damaged tissue is discussed. The changes in protein autofluorescence and reflectance due to thermal damage are studied in ex vivo porcine tissue models. Thermal lesions were created in porcine skin (n=12) and liver (n=15) samples using an IR laser. The damaged regions were clearly visible in the hyperspectral images. Sizes of the thermally damaged regions as measured via HSI are compared to sizes of these regions as measured in white-light images and via physical measurement. Good agreement between the sizes measured in the hyperspectral images, white-light imaging, and physical measurements were found. The HSI system can differentiate between healthy and damaged tissue. Possible applications of this imaging system include determination of tumor margins during surgery/biopsy and cancer diagnosis and staging.

Mapping of spectral signatures with snapshot spectral imaging

We propose a snapshot spectral imaging method that enables direct reconstruction of spatial maps for spectral signatures of given materials using a monochromatic image sensor. An image-plane array of dispersive shapers converts an aerial image of an object into a tailored mixture of spectral and spatial data that is sensed and digitally processed to reconstruct weight coefficients of the spectral signatures. The feasibility of the method is proven by computer simulations.