Compressive Sensing Hyperspectral Imaging by Spectral Multiplexing with Liquid Crystal

Compressive hyperspectral microscopy for cancer detection

Significance

Hyperspectral microscopy grants the ability to characterize unique properties of tissues based on their spectral fingerprint. The ability to label and measure multiple molecular probes simultaneously provides pathologists and oncologists with a powerful tool to enhance accurate diagnostic and prognostic decisions. As the pathological workload grows, having an objective tool that provides companion diagnostics is of immense importance. Therefore, fast whole-slide spectral imaging systems are of immense importance for automated cancer prognostics that meet current and future needs.

Aim

We aim to develop a fast and accurate hyperspectral microscopy system that can be easily integrated with existing microscopes and provide flexibility for optimizing measurement time versus spectral resolution.

Approach

The method employs compressive sensing (CS) and a spectrally encoded illumination device integrated into the illumination path of a standard microscope. The spectral encoding is obtained using a compact liquid crystal cell that is operated in a fast mode. It provides time-efficient measurements of the spectral information, is modular and versatile, and can also be used for other applications that require rapid acquisition of hyperspectral images.

Results

We demonstrated the acquisition of breast cancer biopsies hyperspectral data of the whole camera area within ∼ 1    s . This means that a typical 1 × 1    cm 2 biopsy can be measured in ∼ 10    min . The hyperspectral images with 250 spectral bands are reconstructed from 47 spectrally encoded images in the spectral range of 450 to 700 nm.

Conclusions

CS hyperspectral microscopy was successfully demonstrated on a common lab microscope for measuring biopsies stained with the most common stains, such as hematoxylin and eosin. The high spectral resolution demonstrated here in a rather short time indicates the ability to use it further for coping with the highly demanding needs of pathological diagnostics, both for cancer diagnostics and prognostics.

OpenSpyrit: an ecosystem for open single-pixel hyperspectral imaging

This paper describes OpenSpyrit, an open access and open source ecosystem for reproducible research in hyperspectral single-pixel imaging, composed of SPAS (a Python single-pixel acquisition software), SPYRIT (a Python single-pixel reconstruction toolkit) and SPIHIM (a single-pixel hyperspectral image collection). The proposed OpenSpyrit ecosystem responds to the need for reproducibility and benchmarking in single-pixel imaging by providing open data and open software. The SPIHIM collection, which is the first open-access FAIR dataset for hyperspectral single-pixel imaging, currently includes 140 raw measurements acquired using SPAS and the corresponding hypercubes reconstructed using SPYRIT. The hypercubes are reconstructed by both inverse Hadamard transformation of the raw data and using the denoised completion network (DC-Net), a data-driven reconstruction algorithm. The hypercubes obtained by inverse Hadamard transformation have a native size of 64 × 64 × 2048 for a spectral resolution of 2.3 nm and a spatial resolution that is comprised between 182.4 µm and 15.2 µm depending on the digital zoom. The hypercubes obtained using the DC-Net are reconstructed at an increased resolution of 128 × 128 × 2048. The OpenSpyrit ecosystem should constitute a reference to support benchmarking for future developments in single-pixel imaging.

Deep learning for compressive sensing: a ubiquitous systems perspective

Compressive sensing (CS) is a mathematically elegant tool for reducing the sensor sampling rate, potentially bringing context-awareness to a wider range of devices. Nevertheless, practical issues with the sampling and reconstruction algorithms prevent further proliferation of CS in real world domains, especially among heterogeneous ubiquitous devices. Deep learning (DL) naturally complements CS for adapting the sampling matrix, reconstructing the signal, and learning from the compressed samples. While the CS–DL integration has received substantial research interest recently, it has not yet been thoroughly surveyed, nor has any light been shed on practical issues towards bringing the CS–DL to real world implementations in the ubiquitous computing domain. In this paper we identify main possible ways in which CS and DL can interplay, extract key ideas for making CS–DL efficient, outline major trends in the CS–DL research space, and derive guidelines for the future evolution of CS–DL within the ubiquitous computing domain.

Eliminating Temporal Illumination Variations in Whisk-broom Hyperspectral Imaging

We propose a method for eliminating the temporal illumination variations in whisk-broom (point-scan) hyperspectral imaging. Whisk-broom scanning is useful for acquiring a spatial measurement using a pixel-based hyperspectral sensor. However, when it is applied to outdoor cultural heritages, temporal illumination variations become an issue due to the lengthy measurement time. As a result, the incoming illumination spectra vary across the measured image locations because different locations are measured at different times. To overcome this problem, in addition to the standard raster scan, we propose an additional perpendicular scan that traverses the raster scan. We show that this additional scan allows us to infer the illumination variations over the raster scan. Furthermore, the sparse structure in the illumination spectrum is exploited to robustly eliminate these variations. We quantitatively show that a hyperspectral image captured under sunlight is indeed affected by temporal illumination variations, that a Naïve mitigation method suffers from severe artifacts, and that the proposed method can robustly eliminate the illumination variations. Finally, we demonstrate the usefulness of the proposed method by capturing historic stained-glass windows of a French cathedral.

Towards `Fourth Paradigm’ Spectral Sensing

Reconstruction algorithms are at the forefront of accessible and compact data collection. In this paper, we present a novel reconstruction algorithm, SpecRA, that adapts based on the relative rarity of a signal compared to previous observations. We leverage a data-driven approach to learn optimal encoder-array sensitivities for a novel filter-array spectrometer. By taking advantage of the regularities mined from diverse online repositories, we are able to exploit low-dimensional patterns for improved spectral reconstruction from as few as p=2 channels. Furthermore, the performance of SpecRA is largely independent of signal complexity. Our results illustrate the superiority of our method over conventional approaches and provide a framework towards "fourth paradigm" spectral sensing. We hope that this work can help reduce the size, weight and cost constraints of future spectrometers for specific spectral monitoring tasks in applied contexts such as in remote sensing, healthcare, and quality control.

Spectral imaging with deep learning

The goal of spectral imaging is to capture the spectral signature of a target. Traditional scanning method for spectral imaging suffers from large system volume and low image acquisition speed for large scenes. In contrast, computational spectral imaging methods have resorted to computation power for reduced system volume, but still endure long computation time for iterative spectral reconstructions. Recently, deep learning techniques are introduced into computational spectral imaging, witnessing fast reconstruction speed, great reconstruction quality, and the potential to drastically reduce the system volume. In this article, we review state-of-the-art deep-learning-empowered computational spectral imaging methods. They are further divided into amplitude-coded, phase-coded, and wavelength-coded methods, based on different light properties used for encoding. To boost future researches, we've also organized publicly available spectral datasets.

Snapshot multispectral imaging using a pixel-wise polarization color image sensor

This study proposes a new imaging technique for snapshot multispectral imaging in which a multispectral image was captured using an imaging lens that combines a set of multiple spectral filters and polarization filters, as well as a pixel-wise color polarization image sensor. The author produced a prototype nine-band multispectral camera system that covered from visible to near-infrared regions and was very compact. The camera's spectral performance was evaluated using experiments; moreover, the camera was used to detect the freshness of food and the activity of wild plants and was mounted on a vehicle to obtain a multispectral video while driving.

Dual-camera design for hyperspectral and panchromatic imaging, using a wedge shaped liquid crystal as a spectral multiplexer

In this paper, we present a new hyperspectral compact camera which is designed to have high spatial and spectral resolutions, to be vibrations tolerant, and to achieve state-of-the-art high optical throughput values compared to existing nanosatellite hyperspectral imaging payloads with space heritage. These properties make it perfect for airborne and spaceborne remote sensing tasks. The camera has both hyperspectral and panchromatic imaging capabilities, achieved by employing a wedge-shaped liquid crystal cell together with computational image processing. The hyperspectral images are acquired through passive along-track spatial scanning when no voltage is applied to the cell, and the panchromatic images are quickly acquired in a single snapshot at a high signal-to-noise ratio when the cell is voltage driven.

DeepCubeNet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks

Several hyperspectral (HS) systems based on compressive sensing (CS) theory have been presented to capture HS images with high accuracy and with a lower number of measurements than needed by conventional systems. However, the reconstruction of HS compressed measurements is time-consuming and commonly involves hyperparameter tuning per each scenario. In this paper, we introduce a Convolutional Neural Network (CNN) designed for the reconstruction of HS cubes captured with CS imagers based on spectral modulation. Our Deep Neural Network (DNN), dubbed DeepCubeNet, provides significant reduction in the reconstruction time compared to classical iterative methods. The performance of DeepCubeNet is investigated on simulated data, and we demonstrate for the first time, to the best of our knowledge, real reconstruction of CS HS measurements using DNN. We demonstrate significantly enhanced reconstruction accuracy compared to iterative CS reconstruction, as well as improvement in reconstruction time by many orders of magnitude.

The Future of Hyperspectral Imaging

The Special Issue on hyperspectral imaging (HSI), entitled "The Future of Hyperspectral Imaging", has published 12 papers. Nine papers are related to specific current research and three more are review contributions: In both cases, the request is to propose those methods or instruments so as to show the future trends of HSI. Some contributions also update specific methodological or mathematical tools. In particular, the review papers address deep learning methods for HSI analysis, while HSI data compression is reviewed by using liquid crystals spectral multiplexing as well as DMD-based Raman spectroscopy. Specific topics explored by using data obtained by HSI include alert on the sprouting of potato tubers, the investigation on the stability of painting samples, the prediction of healing diabetic foot ulcers, and age determination of blood-stained fingerprints. Papers showing advances on more general topics include video approach for HSI dynamic scenes, localization of plant diseases, new methods for the lossless compression of HSI data, the fusing of multiple multiband images, and mixed modes of laser HSI imaging for sorting and quality controls.

Fast Method for Liquid Crystal Cell Spatial Variations Estimation Based on Modeling the Spectral Transmission

Liquid crystal phase retarders are utilized by photonic devices and imaging systems for various applications, such as tunable filtering, light modulation, polarimetric imaging, remote sensing and quality inspection. Due to technical difficulties in the manufacturing process, these phase retarders may suffer from spatial non-uniformities, which degrade the performance of the systems. These non-uniformities can be characterized by measuring the spectral transmission at each voltage and each point on the liquid crystal cell, which is time consuming. In this work, we present a new fast and simple method for measuring and computationally estimating the spatial variations of a liquid crystal phase retarder with planar alignment. The method is based on measuring the spectral transmission of the phase retarder at several spatial locations and estimating it at others. The experimental results show that the method provides an accurate spatial description of the phase retarder and can be employed for calibrating relevant systems.

Compressive ultraspectral imaging using multiscale structured illumination

We present a novel compressive spectral imaging technique that attains spatially resolved ultraspectral resolution. The technique employs a multiscale sampling technique based on the Hadamard basis for the single pixel hyperspectral imager. The proposed multiscale sampling method offers high-quality images at a low compression ratio while also facilitating a preview image at a lower resolution by using the fast Hadamard transform.