DeepCubeNet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks

Imaging inside highly scattering media using hybrid deep learning and analytical algorithm

Imaging through highly scattering media is a challenging problem with numerous applications in biomedical and remote-sensing fields. Existing methods that use analytical or deep learning tools are limited by simplified forward models or a requirement for prior physical knowledge, resulting in blurry images or a need for large training databases. To address these limitations, we propose a hybrid scheme called Hybrid-DOT that combines analytically derived image estimates with a deep learning network. Our analysis demonstrates that Hybrid-DOT outperforms a state-of-the-art ToF-DOT algorithm by improving the PSNR ratio by 4.6 dB and reducing the resolution by a factor of 2.5. Furthermore, when compared to a deep learning stand-alone model, Hybrid-DOT achieves a 0.8 dB increase in PSNR, 1.5 times the resolution, and a significant reduction in the required dataset size (factor of 1.6-3). The proposed model remains effective at higher depths, providing similar improvements for up to 160 mean-free paths.

JR2net: a joint non-linear representation and recovery network for compressive spectral imaging

Deep learning models are state-of-the-art in compressive spectral imaging (CSI) recovery. These methods use a deep neural network (DNN) as an image generator to learn non-linear mapping from compressed measurements to the spectral image. For instance, the deep spectral prior approach uses a convolutional autoencoder (CAE) network in the optimization algorithm to recover the spectral image by using a non-linear representation. However, the CAE training is detached from the recovery problem, which does not guarantee optimal representation of the spectral images for the CSI problem. This work proposes a joint non-linear representation and recovery network (JR2net), linking the representation and recovery task into a single optimization problem. JR2net consists of an optimization-inspired network following an alternating direction method of multipliers (ADMM) formulation that learns a non-linear low-dimensional representation and simultaneously performs the spectral image recovery, trained via the end-to-end approach. Experimental results show the superiority of the proposed method with improvements up to 2.57 dB in peak signal-to-noise ratio (PSNR) and performance around 2000 times faster than state-of-the-art methods.

Backtracking Reconstruction Network for Three-Dimensional Compressed Hyperspectral Imaging

Compressed sensing (CS) has been widely used in hyperspectral (HS) imaging to obtain hyperspectral data at a sub-Nyquist sampling rate, lifting the efficiency of data acquisition. Yet, reconstructing the acquired HS data via iterative algorithms is time consuming, which hinders the real-time application of compressed HS imaging. To alleviate this problem, this paper makes the first attempt to adopt convolutional neural networks (CNNs) to reconstruct three-dimensional compressed HS data by backtracking the entire imaging process, leading to a simple yet effective network, dubbed the backtracking reconstruction network (BTR-Net). Concretely, we leverage the divide-and-conquer method to divide the imaging process based on coded aperture tunable filter (CATF) spectral imager into steps, and build a subnetwork for each step to specialize in its reverse process. Consequently, BTR-Net introduces multiple built-in networks which performs spatial initialization, spatial enhancement, spectral initialization and spatial–spectral enhancement in an independent and sequential manner. Extensive experiments show that BTR-Net can reconstruct compressed HS data quickly and accurately, which outperforms leading iterative algorithms both quantitatively and visually, while having superior resistance to noise.

Eliminating Temporal Illumination Variations in Whisk-broom Hyperspectral Imaging

We propose a method for eliminating the temporal illumination variations in whisk-broom (point-scan) hyperspectral imaging. Whisk-broom scanning is useful for acquiring a spatial measurement using a pixel-based hyperspectral sensor. However, when it is applied to outdoor cultural heritages, temporal illumination variations become an issue due to the lengthy measurement time. As a result, the incoming illumination spectra vary across the measured image locations because different locations are measured at different times. To overcome this problem, in addition to the standard raster scan, we propose an additional perpendicular scan that traverses the raster scan. We show that this additional scan allows us to infer the illumination variations over the raster scan. Furthermore, the sparse structure in the illumination spectrum is exploited to robustly eliminate these variations. We quantitatively show that a hyperspectral image captured under sunlight is indeed affected by temporal illumination variations, that a Naïve mitigation method suffers from severe artifacts, and that the proposed method can robustly eliminate the illumination variations. Finally, we demonstrate the usefulness of the proposed method by capturing historic stained-glass windows of a French cathedral.

Spectral imaging with deep learning

The goal of spectral imaging is to capture the spectral signature of a target. Traditional scanning method for spectral imaging suffers from large system volume and low image acquisition speed for large scenes. In contrast, computational spectral imaging methods have resorted to computation power for reduced system volume, but still endure long computation time for iterative spectral reconstructions. Recently, deep learning techniques are introduced into computational spectral imaging, witnessing fast reconstruction speed, great reconstruction quality, and the potential to drastically reduce the system volume. In this article, we review state-of-the-art deep-learning-empowered computational spectral imaging methods. They are further divided into amplitude-coded, phase-coded, and wavelength-coded methods, based on different light properties used for encoding. To boost future researches, we've also organized publicly available spectral datasets.

3D denoised completion network for deep single-pixel reconstruction of hyperspectral images

Single-pixel imaging acquires an image by measuring its coefficients in a transform domain, thanks to a spatial light modulator. However, as measurements are sequential, only a few coefficients can be measured in the real-time applications. Therefore, single-pixel reconstruction is usually an underdetermined inverse problem that requires regularization to obtain an appropriate solution. Combined with a spectral detector, the concept of single-pixel imaging allows for hyperspectral imaging. While each channel can be reconstructed independently, we propose to exploit the spectral redundancy between channels to regularize the reconstruction problem. In particular, we introduce a denoised completion network that includes 3D convolution filters. Contrary to black-box approaches, our network combines the classical Tikhonov theory with the deep learning methodology, leading to an explainable network. Considering both simulated and experimental data, we demonstrate that the proposed approach yields hyperspectral images with higher quantitative metrics than the approaches developed for grayscale images.

Channeled imaging spectropolarimeter reconstruction by neural networks

Snapshot channeled imaging spectropolarimetry (SCISP), which can achieve spectral and polarization imaging without scanning (a single exposure), is a promising optical technique. As Fourier transform is used to reconstruct information, SCISP has its inherent limitations such as channel crosstalk, resolution and accuracy drop, the complex phase calibration, et al. To overcome these drawbacks, a nonlinear technique based on neural networks (NNs) is introduced to replace the role of Fourier reconstruction. Herein, abundant spectral and polarization datasets were built through specially designed generators. The established NNs can effectively learn the forward conversion procedure through minimizing a loss function, subsequently enabling a stable output containing spectral, polarization, and spatial information. The utility and reliability of the proposed technique is confirmed by experiments, which are proved to maintain high spectral and polarization accuracy.

Compressive spectral image reconstruction using deep prior and low-rank tensor representation

Compressive spectral imaging (CSI) has emerged as an alternative spectral image acquisition technology, which reduces the number of measurements at the cost of requiring a recovery process. In general, the reconstruction methods are based on handcrafted priors used as regularizers in optimization algorithms or recent deep neural networks employed as an image generator to learn a non-linear mapping from the low-dimensional compressed measurements to the image space. However, these deep learning methods need many spectral images to obtain good performance. In this work, a deep recovery framework for CSI without training data is presented. The proposed method is based on the fact that the structure of some deep neural networks and an appropriated low-dimensional structure are sufficient to impose a structure of the underlying spectral image from CSI. We analyzed the low-dimensional structure via the Tucker representation, modeled in the first net layer. The proposed scheme is obtained by minimizing the ${\ell _2}$-norm distance between the compressive measurements and the predicted measurements, and the desired recovered spectral image is formed just before the forward operator. Simulated and experimental results verify the effectiveness of the proposed method for the coded aperture snapshot spectral imaging.

DD-Net: spectral imaging from a monochromatic dispersed and diffused snapshot

We propose a snapshot spectral imaging method for the visible spectral range using a single monochromatic camera equipped with a two-dimensional (2D) binary-encoded phase diffuser placed at the pupil of the imaging lens and by resorting to deep learning (DL) algorithms for signal reconstruction. While spectral imaging was shown to be feasible using two cameras equipped with a single, one-dimensional (1D) binary diffuser and compressed sensing (CS) algorithms [Appl. Opt.59, 7853 (2020).APOPAI0003-693510.1364/AO.395541], the suggested diffuser design expands the optical response and creates optical spatial and spectral encoding along both dimensions of the image sensor. To recover the spatial and spectral information from the dispersed and diffused (DD) monochromatic snapshot, we developed novel DL algorithms, dubbed DD-Nets, which are tailored to the unique response of the optical system, which includes either a 1D or a 2D diffuser. High-quality reconstructions of the spectral cube in simulation and lab experiments are presented for system configurations consisting of a single monochromatic camera with either a 1D or a 2D diffuser. We demonstrate that the suggested system configuration with the 2D diffuser outperforms system configurations with a 1D diffuser that utilize either DL-based or CS-based algorithms for the reconstruction of the spectral cube.

Solver-informed neural networks for spectrum reconstruction of colloidal quantum dot spectrometers

Recently, the miniature spectrometer based on the optical filter array has received much attention due to its versatility. Among many open challenges, designing efficient and stable algorithms to recover the input spectrum from the raw measurements is the key to success. Of many existing spectrum reconstruction algorithms, regularization-based algorithms have emerged as practical approaches to the spectrum reconstruction problem, but the reconstruction is still challenging due to ill-posedness of the problem. To alleviate this issue, we propose a novel reconstruction method based on a solver-informed neural network (NN). This approach consists of two components: (1) an existing spectrum reconstruction solver to extract the spectral feature from the raw measurements (2) a multilayer perceptron to build a map from the input feature to the spectrum. We investigate the reconstruction performance of the proposed method on a synthetic dataset and a real dataset collected by the colloidal quantum dot (CQD) spectrometer. The results demonstrate the reconstruction accuracy and robustness of the solver-informed NN. In conclusion, the proposed reconstruction method shows excellent potential for spectral recovery of filter-based miniature spectrometers.

On the value of CTIS imagery for neural-network-based classification: a simulation perspective

The computed tomography imaging spectrometer (CTIS) is a snapshot hyperspectral imaging system. Its output is a 2D image of multiplexed spatiospectral projections of the hyperspectral cube of the scene. Traditionally, the 3D cube is reconstructed from this image before further analysis. In this paper, we show that it is possible to learn information directly from the CTIS raw output, by training a neural network to perform binary classification on such images. The use case we study is an agricultural one, as snapshot imagery is used substantially in this field: the detection of apple scab lesions on leaves. To train the network appropriately and to study several degrees of scab infection, we simulated CTIS images of scabbed leaves. This was made possible with a novel CTIS simulator, where special care was taken to preserve realistic pixel intensities compared to true images. To the best of our knowledge, this is the first application of compressed learning on a simulated CTIS system.

Snapshot multispectral endomicroscopy

Multispectral endomicroscopy provides tissue functional information in addition to structural information for accurate disease diagnosis. In this Letter, we propose a snapshot multispectral endomicroscope that employs a fiber bundle to deliver an in-body tissue spatial-spectral datastream to an external compressive spectral imager. Equipped with an end-to-end deep-learning-based reconstruction algorithm, we are able to capture tissue multispectral data in video rates and reconstruct high-resolution multispectral images with up to 24 spectral channels in near-real time.

mHealth spectroscopy of blood hemoglobin with spectral super-resolution

Although blood hemoglobin (Hgb) testing is a routine procedure in a variety of clinical situations, noninvasive, continuous, and real-time blood Hgb measurements are still challenging. Optical spectroscopy can offer noninvasive blood Hgb quantification, but requires bulky optical components that intrinsically limit the development of mobile health (mHealth) technologies. Here, we report spectral super-resolution (SSR) spectroscopy that virtually transforms the built-in camera (RGB sensor) of a smartphone into a hyperspectral imager for accurate and precise blood Hgb analyses. Statistical learning of SSR enables us to reconstruct detailed spectra from three color RGB data. Peripheral tissue imaging with a mobile application is further combined to compute exact blood Hgb content without a priori personalized calibration. Measurements over a wide range of blood Hgb values show reliable performance of SSR blood Hgb quantification. Given that SSR does not require additional hardware accessories, the mobility, simplicity, and affordability of conventional smartphones support the idea that SSR blood Hgb measurements can be used as an mHealth method.