A review of snapshot multidimensional optical imaging: measuring photon tags in parallel

Temporal compressive edge imaging enabled by a lensless diffuser camera

Lensless imagers based on diffusers or encoding masks enable high-dimensional imaging from a single-shot measurement and have been applied in various applications. However, to further extract image information such as edge detection, conventional post-processing filtering operations are needed after the reconstruction of the original object images in the diffuser imaging systems. Here, we present the concept of a temporal compressive edge detection method based on a lensless diffuser camera, which can directly recover a time sequence of edge images of a moving object from a single-shot measurement, without further post-processing steps. Our approach provides higher image quality during edge detection, compared with the "conventional post-processing method." We demonstrate the effectiveness of this approach by both numerical simulation and experiments. The proof-of-concept approach can be further developed with other image post-processing operations or versatile computer vision assignments toward task-oriented intelligent lensless imaging systems.

Optical design and fabrication of a multi-channel imaging spectrometer for combustion flame monitoring

We design and construct a broadband integrated multi-channel imaging spectrometer (MCIS) from visible light to near-infrared. This system can directly obtain spectral images that conform to the consistent visual habits of the human eyes through a single exposure of the detector. The genetic algorithm is used to calculate system parameters to minimize pixel waste between spectral channels, achieving nearly 100% utilization of detector pixels. The field stop suppresses stray light in the system. This device is used for imaging an optical-resolution target, an object, and a furnace to verify the basic principles of the system. The results indicate that the system can effectively utilize detectors to monitor high-temperature objects in the visible to near-infrared wavelength range.

Tutorial on compressed ultrafast photography

Significance

Compressed ultrafast photography (CUP) is currently the world's fastest single-shot imaging technique. Through the integration of compressed sensing and streak imaging, CUP can capture a transient event in a single camera exposure with imaging speeds from thousands to trillions of frames per second, at micrometer-level spatial resolutions, and in broad sensing spectral ranges.

Aim

This tutorial aims to provide a comprehensive review of CUP in its fundamental methods, system implementations, biomedical applications, and prospect.

Approach

A step-by-step guideline to CUP's forward model and representative image reconstruction algorithms is presented with sample codes and illustrations in Matlab and Python. Then, CUP's hardware implementation is described with a focus on the representative techniques, advantages, and limitations of the three key components-the spatial encoder, the temporal shearing unit, and the two-dimensional sensor. Furthermore, four representative biomedical applications enabled by CUP are discussed, followed by the prospect of CUP's technical advancement.

Conclusions

CUP has emerged as a state-of-the-art ultrafast imaging technology. Its advanced imaging ability and versatility contribute to unprecedented observations and new applications in biomedicine. CUP holds great promise in improving technical specifications and facilitating the investigation of biomedical processes.

Polarized hyperspectral microscopic imaging system for enhancing the visualization of collagen fibers and head and neck squamous cell carcinoma

Significance

Polarized hyperspectral microscopes with the capability of full Stokes vector imaging have potential for many biological and medical applications.

Aim

The aim of this study is to investigate polarized hyperspectral imaging (PHSI) for improving the visualization of collagen fibers, which is an important biomarker related to tumor development, and improving the differentiation of normal and tumor cells on pathologic slides.

Approach

We customized a polarized hyperspectral microscopic imaging system comprising an upright microscope with a motorized stage, two linear polarizers, two liquid crystal variable retarders (LCVRs), and a compact SnapScan hyperspectral camera. The polarizers and LCVRs worked in tandem with the hyperspectral camera to acquire polarized hyperspectral images, which were further used to calculate four Stokes vectors: S 0 , S 1 , S 2 , and S 3 . Synthetic RGB images of the Stokes vectors were generated for the visualization of cellular components in PHSI images. Regions of interest of collagen, normal cells, and tumor cells in the synthetic RGB images were selected, and spectral signatures of the selected components were extracted for comparison. Specifically, we qualitatively and quantitatively investigated the enhanced visualization and spectral characteristics of dense fibers and sparse fibers in normal stroma tissue, fibers accumulated within tumors, and fibers accumulated around tumors.

Results

By employing our customized polarized hyperspectral microscope, we extract the spectral signatures of Stokes vector parameters of collagen as well as of tumor and normal cells. The measurement of Stokes vector parameters increased the image contrast of collagen fibers and cells in the slides.

Conclusions

With the spatial and spectral information from the Stokes vector data cubes (S 0 , S 1 , S 2 , and S 3 ), our PHSI microscope system enhances the visualization of tumor cells and tumor microenvironment components, thus being beneficial for pathology and oncology.

Hyperspectral screen-image-synthesis meter with scattering-noise suppression

The screen image synthesis (SIS) meter was originally proposed as a high-speed measurement tool, which fused the measured data from multiple sample-rotational angles to produce a whole-field measurement result. However, it suffered from stray light noise and lacked the capability of spectrum measurement. In this study, we propose an SIS system embedded with a snapshot hyperspectral technology, which was based on a dispersion image of the sparse sampling screen (SSS). When a photo was captured, it was transformed and calibrated to hyperspectral data at a specific sample-rotational angle. After the hyperspectral data in all sample-rotational angles were captured, an SIS image-fusion process was then applied to get the whole field hyperspectral data. By applying SSS to the SIS meter, we not only create a screen image synthesis hyperspectral meter but also effectively address the issue of stray-light noise. In the experiment, we analyze its correctness by comparing the hyperspectral value with a one-dimensional spectrum goniometer (ODSG). We also show the 2D color temperature coefficient distribution and compare it with the ODSG. Experimental results also demonstrate the feasibility in terms of both spectrum distribution meter and color coefficient temperature distribution meter.

Coded aperture snapshot hyperspectral light field tomography

Multidimensional imaging has emerged as a powerful technology capable of simultaneously acquiring spatial, spectral, and depth information about a scene. However, existing approaches often rely on mechanical scanning or multi-modal sensing configurations, leading to prolonged acquisition times and increased system complexity. Coded aperture snapshot spectral imaging (CASSI) has introduced compressed sensing to recover three-dimensional (3D) spatial-spectral datacubes from single snapshot two-dimensional (2D) measurements. Despite its advantages, the reconstruction problem remains severely underdetermined due to the high compression ratio, resulting in limited spatial and spectral reconstruction quality. To overcome this challenge, we developed a novel two-stage cascaded compressed sensing scheme called coded aperture snapshot hyperspectral light field tomography (CASH-LIFT). By appropriately distributing the computation load to each stage, this method utilizes the compressibility of natural scenes in multiple domains, reducing the ill-posed nature of datacube recovery and achieving enhanced spatial resolution, suppressed aliasing artifacts, and improved spectral fidelity. Additionally, leveraging the snapshot 3D imaging capability of LIFT, our approach efficiently records a five-dimensional (5D) plenoptic function in a single snapshot.

Fabrication of a multifaceted mapping mirror using two-photon polymerization for a snapshot image mapping spectrometer

A design and fabrication technique for making high-precision and large-format multifaceted mapping mirrors is presented. The method is based on two-photon polymerization, which allows more flexibility in the mapping mirror design. The mirror fabricated in this paper consists of 36 2D tilted square pixels, instead of the continuous facet design used in diamond cutting. The paper presents a detailed discussion of the fabrication parameters and optimization process, with particular emphasis on the optimization of stitching defects by compensating for the overall tilt angle and reducing the printing field of view. The fabricated mirrors were coated with a thin layer of aluminum (93 nm) using sputter coating to enhance the reflection rate over the target wave range. The mapping mirror was characterized using a white light interferometer and a scanning electron microscope, which demonstrates its optical quality surface (with a surface roughness of 12 nm) and high-precision tilt angles (with an average of 2.03% deviation). Finally, the incorporation of one of the 3D printed mapping mirrors into an image mapping spectrometer prototype allowed for the acquisition of high-quality images of the USAF resolution target and bovine pulmonary artery endothelial cells stained with three fluorescent dyes, demonstrating the potential of this technology for practical applications.

Spatial-spectral resolution tunable snapshot imaging spectrometer: analytical design and implementation

A snapshot imaging spectrometer is a powerful tool for dynamic target tracking and real-time recognition compared with a scanning imaging spectrometer. However, all the current snapshot spectral imaging techniques suffer from a major trade-off between the spatial and spectral resolutions. In this paper, an integral field snapshot imaging spectrometer (TIF-SIS) with a continuously tunable spatial-spectral resolution and light throughput is proposed and demonstrated. The proposed TIF-SIS is formed by a fore optics, a lenslet array, and a collimated dispersive subsystem. Theoretical analyses indicate that the spatial-spectral resolution and light throughput of the system can be continuously tuned through adjusting the F number of the fore optics, the rotation angle of the lenslet array, or the focal length of the collimating lens. Analytical relationships between the spatial and spectral resolutions and the first-order parameters of the system with different geometric arrangements of the lenslet unit are obtained. An experimental TIF-SIS consisting of a self-fabricated lenslet array with a pixelated scale of 100×100 and a fill factor of 0.716 is built. The experimental results show that the spectral resolution of the system can be steadily improved from 4.17 to 0.82 nm with a data cube (N x×N y×N λ) continuously tuned from 35×35×36 to 40×40×183 in the visible wavelength range from 500 to 650 nm, which is consistent with the theoretical prediction. The proposed method for real-time tuning of the spatial-spectral resolution and light throughput opens new possibilities for broader applications, especially for recognition of things with weak spectral signature and biomedical investigations where a high light throughput and tunable resolution are needed.

Monocular metasurface camera for passive single-shot 4D imaging

It is a grand challenge for an imaging system to simultaneously obtain multi-dimensional light field information, such as depth and polarization, of a scene for the accurate perception of the physical world. However, such a task would conventionally require bulky optical components, time-domain multiplexing, and active laser illumination. Here, we experimentally demonstrate a compact monocular camera equipped with a single-layer metalens that can capture a 4D image, including 2D all-in-focus intensity, depth, and polarization of a target scene in a single shot under ambient illumination conditions. The metalens is optimized to have a conjugate pair of polarization-decoupled rotating single-helix point-spread functions that are strongly dependent on the depth of the target object. Combined with a straightforward, physically interpretable image retrieval algorithm, the camera can simultaneously perform high-accuracy depth sensing and high-fidelity polarization imaging over an extended depth of field for both static and dynamic scenes in both indoor and outdoor environments. Such a compact multi-dimensional imaging system could enable new applications in diverse areas ranging from machine vision to microscopy.

Theoretical modeling and simulation of fiber Bragg grating sensor interrogator based on linear variable filter

With the increasing frequency of aviation accidents in recent years, aircraft safety has received increasing attention. Aircraft operating condition detection is an important part of aviation safety. Fiber Bragg grating (FBG) sensors, with their excellent characteristics, enable online monitoring of aircraft operating conditions. However, the application of FBG sensors in aviation is currently limited because it is difficult for FBG sensor interrogators to meet the requirements of small size, light weight, and good vibration resistance in the aviation field. Therefore, this paper proposes a linear variable filter (LVF)-based FBG sensor interrogator to meet the requirements. An optical model of the interrogator is established. The parameters which determine the performances of the interrogator are analyzed and the design criteria are discussed. According to the requirements in the aviation field, the optical system of the interrogator is designed. The simulation results show that the LVF-based FBG sensor interrogation system has a bandwidth range of 90 nm (1505 nm-1595 nm), a resolution of 2 pm, and a capacity of 15 FBG sensors.

Augmented light field tomography through parallel spectral encoding

Snapshot recording of transient dynamics in three dimensions (3-D) is highly demanded in both fundamental and applied sciences. Yet it remains challenging for conventional high-speed cameras to address this need due to limited electronic bandwidth and reliance on mechanical scanning. The emergence of light field tomography (LIFT) provides a new solution to these long-standing problems and enables 3-D imaging at an unprecedented frame rate. However, based on sparse-view computed tomography, LIFT can accommodate only a limited number of projections, degrading the resolution in the reconstructed image. To alleviate this problem, we herein present a spectral encoding scheme to significantly increase the number of allowable projections in LIFT while maintaining its snapshot advantage. The resultant system can record 3-D dynamics at a kilohertz volumetric frame rate. Moreover, by using a multichannel compressed sensing algorithm, we improve the image quality with an enhanced spatial resolution and suppressed aliasing artifacts.

Polarized hyperspectral microscopic imaging for collagen visualization on pathologic slides of head and neck squamous cell carcinoma

We developed a polarized hyperspectral microscope to collect four types of Stokes vector data cubes (S0, S1, S2, and S3) of the pathologic slides with head and neck squamous cell carcinoma (HNSCC). Our system consists of an optical light microscope with a movable stage, two polarizers, two liquid crystal variable retarders (LCVRs), and a SnapScan hyperspectral camera. The polarizers and LCVRs work in tandem with the hyperspectral camera to acquire polarized hyperspectral images. Synthetic pseudo-RGB images are generated from the four Stokes vector data cubes based on a transformation function similar to the spectral response of human eye for the visualization of hyperspectral images. Collagen is the most abundant extracellular matrix (ECM) protein in the human body. A major focus of studying the ECM in tumor microenvironment is the role of collagen in both normal and abnormal function. Collagen tends to accumulate in and around tumors during cancer development and growth. In this study, we acquired images from normal regions containing normal cells and collagen fibers and from tumor regions containing cancerous squamous cells and collagen fibers on HNSCC pathologic slides. The preliminary results demonstrated that our customized polarized hyperspectral microscope is able to improve the visualization of collagen on HNSCC pathologic slides under different situations, including thick fibers of normal stroma, thin fibers of normal stroma, fibers of normal muscle cells, fibers accumulated in tumors, fibers accumulated around tumors. Our preliminary results also demonstrated that the customized polarized hyperspectral microscope is capable of extracting the spectral signatures of collagen based on Stokes vector parameters and can have various applications in pathology and oncology.

An ultra-small nine-color spectrometer with a two-layer biparted ten-dichroic-mirror array and an image sensor

An ultra-small (54 × 58 × 8.5 mm) and large aperture (1 × 7 mm) nine-color spectrometer-using an array of ten dichroic mirrors "biparted" as two layers-was developed and used for snapshot spectral imaging. Incident-light flux with a cross section smaller than the aperture size is split into nine color fluxes with 20-nm-width contiguous wavelength bands and central wavelengths of 530, 550, 570, 590, 610, 630, 650, 670, and 690 nm. Images of the nine color fluxes are simultaneously and efficiently measured by an image sensor. Unlike a conventional dichroic-mirror array, the developed dichroic-mirror array has a unique biparted configuration that not only increases the number of colors that can be measured simultaneously but also improves the image resolution of each color flux. The developed nine-color spectrometer was used for four-capillary-array electrophoresis. Eight dyes concurrently migrating in each capillary were simultaneously quantified by nine-color laser-induced fluorescence detection. Since the nine-color spectrometer is not only ultra-small and inexpensive but also has high light throughput and sufficient spectral resolution for most spectral-imaging applications, it has the potential to be widely used in various fields.

A Review of Pharmaceutical Robot based on Hyperspectral Technology

The quality and safety of medicinal products are related to patients’ lives and health. Therefore, quality inspection takes a key role in the pharmaceutical industry. Most of the previous solutions are based on machine vision, however, their performance is limited by the RGB sensor. The pharmaceutical visual inspection robot combined with hyperspectral imaging technology is becoming a new trend in the high-end medical quality inspection process since the hyperspectral data can provide spectral information with spatial knowledge. Yet, there is no comprehensive review about hyperspectral imaging-based medicinal products inspection. This paper focuses on the pivotal pharmaceutical applications, including counterfeit drugs detection, active component analysis of tables, and quality testing of herbal medicines and other medical materials. We discuss the technology and hardware of Raman spectroscopy and hyperspectral imaging, firstly. Furthermore, we review these technologies in pharmaceutical scenarios. Finally, the development tendency and prospect of hyperspectral imaging technology-based robots in the field of pharmaceutical quality inspection is summarized.

Hybrid non-sequential modeling of an image mapping spectrometer

An image mapping spectrometer (IMS) is a kind of snapshot imaging spectrometer characterized by containing several array components including the image mapper, prism array, and reimaging lens array. We propose a hybrid non-sequential modeling method of IMS and present the complete optical model of the system built in Zemax. This method utilizes the spatial periodicity of the array components and requires only a small number of input parameters. Moreover, we design a collimating lens of a large relative aperture, sufficient working distance, and low aberration to meet the requirements of an IMS with good optical performance and compact volume. The designed lens is quantitatively evaluated in the entire IMS model, and the results demonstrate that the lens has excellent optical performance. The evaluation on the collimating lens also demonstrates the capability of the proposed modeling method in the design and optimization of systems such as the IMS that contain multiple array components. The designed collimating lens is manufactured and assembled in the experimental setup of the IMS. The proposed modeling method is verified by experimental results.

Ruggedized, field-ready snapshot light-guide-based imaging spectrometer for environmental and remote sensing applications

A field-ready, fiber-based high spatial sampling snapshot imaging spectrometer was developed for applications such as environmental monitoring and smart farming. The system achieves video rate frame transfer and exposure times down to a few hundred microseconds in typical daylight conditions with ∼63,000 spatial points and 32 spectral channels across the 470nm to 700nm wavelength range. We designed portable, ruggedized opto-mechanics to allow for imaging from an airborne platform. To ensure successful data collection prior to flight, imaging speed and signal-to-noise ratio was characterized for imaging a variety of land covers from the air. The system was validated by performing a series of observations including: Liriope Muscari plants under a range of water-stress conditions in a controlled laboratory experiment and field observations of sorghum plants in a variety of soil conditions. Finally, we collected data from a series of engineering flights and present reassembled images and spectral sampling of rural and urban landscapes.

Automatic detection of head and neck squamous cell carcinoma on pathologic slides using polarized hyperspectral imaging and deep learning

The study is to incorporate polarized hyperspectral imaging (PHSI) with deep learning for automatic detection of head and neck squamous cell carcinoma (SCC) on hematoxylin and eosin (H&E) stained tissue slides. A polarized hyperspectral imaging microscope had been developed in our group. In this paper, we firstly collected the Stokes vector data cubes (S0, S1, S2, and S3) of histologic slides from 17 patients with SCC by the PHSI microscope, under the wavelength range from 467 nm to 750 nm. Secondly, we generated the synthetic RGB images from the original Stokes vector data cubes. Thirdly, we cropped the synthetic RGB images into image patches at the image size of 96×96 pixels, and then set up a ResNet50-based convolutional neural network (CNN) to classify the image patches of the four Stokes vector parameters (S0, S1, S2, and S3) by application of transfer learning. To test the performances of the model, each time we trained the model based on the image patches (S0, S1, S2, and S3) of 16 patients out of 17 patients, and used the trained model to calculate the testing accuracy based on the image patches of the rest 1 patient (S0, S1, S2, and S3). We repeated the process for 6 times and obtained 24 testing accuracies (S0, S1, S2, and S3) from 6 different patients out of the 17 patients. The preliminary results showed that the average testing accuracy (84.2%) on S3 outperformed the average testing accuracy (83.5%) on S0. Furthermore, 4 of 6 testing accuracies of S3 (96.0%, 87.3%, 82.8%, and 86.7%) outperformed the testing accuracies of S0 (93.3%, 85.2%, 80.2%, and 79.0%). The study demonstrated the potential of using polarized hyperspectral imaging and deep learning for automatic detection of head and neck SCC on pathologic slides.

Snapshot hyperspectral light field tomography

We present snapshot hyperspectral light field tomography (Hyper-LIFT), a highly efficient method in recording a 5D (x, y, spatial coordinates; θ, φ, angular coordinates; λ, wavelength) plenoptic function. Using a Dove prism array and a cylindrical lens array, we simultaneously acquire multi-angled 1D en face projections of the object like those in standard sparse-view computed tomography. We further disperse those projections and measure the spectra in parallel using a 2D image sensor. Within a single snapshot, the resultant system can capture a 5D data cube with 270 × 270 × 4 × 4 × 360 voxels. We demonstrated the performance of Hyper-LIFT in imaging spectral volumetric scenes.

Channeled imaging spectropolarimeter reconstruction by neural networks

Snapshot channeled imaging spectropolarimetry (SCISP), which can achieve spectral and polarization imaging without scanning (a single exposure), is a promising optical technique. As Fourier transform is used to reconstruct information, SCISP has its inherent limitations such as channel crosstalk, resolution and accuracy drop, the complex phase calibration, et al. To overcome these drawbacks, a nonlinear technique based on neural networks (NNs) is introduced to replace the role of Fourier reconstruction. Herein, abundant spectral and polarization datasets were built through specially designed generators. The established NNs can effectively learn the forward conversion procedure through minimizing a loss function, subsequently enabling a stable output containing spectral, polarization, and spatial information. The utility and reliability of the proposed technique is confirmed by experiments, which are proved to maintain high spectral and polarization accuracy.

Combination of Structured Illumination Microscopy with Hyperspectral Imaging for Cell Analysis

Existing structured illumination microscopy (SIM) allows super-resolution live-cell imaging in few color channels that provide merely morphological information but cannot acquire the sample spectrum that is strongly relevant to the underlying physicochemical property. We develop hyperspectral SIM which enables high-speed spectral super-resolution imaging in SIM for the first time. Through optically mapping the three-dimensional (x, y, and λ) datacube of the sample to the detector plane, hyperspectral SIM allows snapshot spectral imaging of the SIM raw image, detecting the sample spectrum while retaining the high-speed and super-resolution characteristics of SIM. We demonstrate hyperspectral SIM imaging and reconstruct a datacube containing 31 super-resolution images of different wavelengths from only 9 exposures, achieving a 15 nm spectral resolution. We show time-lapse hyperspectral SIM imaging that achieves an imaging speed of 2.7 s per datacube-31-fold faster than the existing wavelength scanning strategy. To demonstrate the great prospects for further combining hyperspectral SIM with various spectral analysis methods, we also perform spectral unmixing of the hyperspectral SIM result while imaging the spectrally overlapped sample.

Snapshot three-dimensional absorption imaging of microscopic specimens

Snapshot projection optical tomography (SPOT) uses a micro-lens array (MLA) to simultaneously capture the projection images of a three-dimensional (3D) specimen corresponding to different viewing directions. Compared to other light-field imaging techniques using an MLA, SPOT is dual telecentric and can block high-angle stray rays without sacrificing the light collection efficiency. Using SPOT, we recently demonstrated snapshot 3D fluorescence imaging. Here we demonstrate snapshot 3D absorption imaging of microscopic specimens. For the illumination, we focus the incoherent light from a light-emitting diode onto a pinhole, which is placed at a conjugate plane to the sample plane. SPOT allows us to capture the ray bundles passing through the specimen along different directions. The images recorded by an array of lenslets can be related to the projections of 3D absorption coefficient along the viewing directions of lenslets. Using a tomographic reconstruction algorithm, we obtain the 3D map of absorption coefficient. We apply the developed system to different types of samples, which demonstrates the optical sectioning capability. The transverse and axial resolutions measured with gold nanoparticles are 1.3 μm and 2.3 μm, respectively.

Spectral polarization camera based on ghost imaging via sparsity constraints

A spectral polarization camera based on ghost imaging via sparsity constraints (GISC) is presented. The proposed imager modulates three-dimensional spatial and spectral information of the target into two-dimensional speckle patterns using a spatial random phase modulator and then acquires the speckle patterns at four linear polarization channels through a polarized CCD. The experimental results verify the feasibility of the system structure and reconstruction algorithm. The GISC spectral polarization camera, which has a simple structure and achieves compressive sampling during the imaging acquisition process, provides a simple scheme for obtaining multi-dimensional information of the light field.

Snapshot Imaging Spectrometer Based on Pixel-Level Filter Array (PFA)

Snapshot spectral imaging technology plays an important role in many fields. However, most existing snapshot imaging spectrometers have the shortcomings of a large volume or heavy computational burden. In this paper, we present a novel snapshot imaging spectrometer based on the pixel-level filter array (PFA), which can simultaneously obtain both spectral and spatial information. The system is composed of a fore-optics, a PFA, a relay lens, and a monochromatic sensor. The incoming light first forms an intermediate image on the PFA through the fore-optics. Then, the relay lens reimages the spectral images on the PFA onto the monochromatic sensor. Through the use of the PFA, we can capture a three-dimensional (spatial coordinates and wavelength) datacube in a single exposure. Compared with existing technologies, our system possesses the advantages of a simple implementation, low cost, compact structure, and high energy efficiency by removing stacked dispersive or interferometric elements. Moreover, the characteristic of the direct imaging mode ensures the low computational burden of the system, thus shortening the imaging time. The principle and design of the system are described in detail. An experimental prototype is built and field experiments are carried out to verify the feasibility of the proposed scheme.

Enhancement of CASSI by a zero-order image employing a single detector

Coded aperture snapshot spectral imaging (CASSI) makes it possible to recover 3D hyperspectral data from a single 2D image. However, the reconstruction problem is severely underdetermined, and efforts to improve the compression ratio typically make the imaging system more complex and cause a significant loss of incoming light intensity. In this paper, we propose a novel approach to CASSI that enables capturing both a spectrally sheared and an integrated image of a scene with a single camera. We performed hyperspectral imaging of three different testing scenes in the spectral range of 500-900 nm. We demonstrate the prominent effect of using the nondiffracted image on the reconstruction of data from our camera. The use of the spectrally integrated image improves the reconstruction quality, and we observed an approximately fivefold reduction in reconstruction time.

Estimation of skin microcirculatory hemoglobin oxygen saturation and red blood cell tissue fraction using a multispectral snapshot imaging system: a validation study

SIGNIFICANCE

Hemoglobin oxygen saturation and red blood cell (RBC) tissue fraction are important parameters when assessing microvascular status. Functional information can be attained using temporally resolved measurements performed during stimulus-response protocols. Pointwise assessments can currently be conducted with probe-based systems. However, snapshot multispectral imaging (MSI) can be used for spatial-temporal measurements.

AIM

To validate if hemoglobin oxygen saturation and RBC tissue fraction can be quantified using a snapshot MSI system and an inverse Monte Carlo algorithm.

APPROACH

Skin tissue measurements from the MSI system were compared to those from a validated probe-based system during arterial and venous occlusion provocation on 24 subjects in the wavelength interval 450 to 650 nm, to evaluate a wide range of hemoglobin oxygen saturation and RBC tissue fraction levels.

RESULTS

Arterial occlusion results show a mean linear regression R2  =  0.958 for hemoglobin oxygen saturation. Comparing relative RBC tissue fraction during venous occlusion results in R2  =  0.925. The MSI system shows larger dynamic changes than the reference system, which might be explained by a deeper sampling including more capacitance vessels.

CONCLUSIONS

The snapshot MSI system estimates hemoglobin oxygen saturation and RBC tissue fraction in skin microcirculation showing a high correlation (R2  >  0.9 in most subjects) with those measured by the reference method.

Automatic detection of head and neck squamous cell carcinoma on pathologic slides using polarized hyperspectral imaging and machine learning

The aim of this study is to incorporate polarized hyperspectral imaging (PHSI) with machine learning for automatic detection of head and neck squamous cell carcinoma (SCC) on hematoxylin and eosin (H&E) stained tissue slides. A polarized hyperspectral imaging microscope had been developed in our group. In this paper, we imaged 20 H&E stained tissue slides from 10 patients with SCC of the larynx by the PHSI microscope. Several machine learning algorithms, including support vector machine (SVM), random forest, Gaussian naive Bayes, and logistic regression, were applied to the collected image data for the automatic detection of SCC on the H&E stained tissue slides. The performance of these methods was compared among the collected PHSI data, the pseudo-RGB images generated from the PHSI data, and the PHSI data after applying the principal component analysis (PCA) transformation. The results suggest that SVM is a superior classifier for the classification task based on the PHSI data cubes compared to the other three classifiers. The incorporate of four Stokes vector parameters improved the classification accuracy. Finally, the PCA transformed image data did not improve the accuracy as it might lose some important information from the original PHSI data. The preliminary results show that polarized hyperspectral imaging can have many potential applications in digital pathology.

Snapshot spectral polarimetric light field imaging using a single detector

In this Letter, we investigate a snapshot spectral-polarimetric-volumetric imaging (SSPVI) system using a single detector. Through compressed acquisition and reconstruction, SSPVI can achieve spectral imaging (x,y,λ), polarization imaging (x,y,ψ,χ), and light field imaging (x,y,θ,φ) simultaneously. The newly discovered performance is showcased by attaining the spectral-polarimetric-volumetric video and different laboratory accuracy experiments. These never-seen-before capacities of the camera open new prospects for many applications, such as biological analysis, object recognition, and remote sensing.

Snapshot multidimensional photography through active optical mapping

Multidimensional photography can capture optical fields beyond the capability of conventional image sensors that measure only two-dimensional (2D) spatial distribution of light. By mapping a high-dimensional datacube of incident light onto a 2D image sensor, multidimensional photography resolves the scene along with other information dimensions, such as wavelength and time. However, the application of current multidimensional imagers is fundamentally restricted by their static optical architectures and measurement schemes—the mapping relation between the light datacube voxels and image sensor pixels is fixed. To overcome this limitation, we propose tunable multidimensional photography through active optical mapping. A high-resolution spatial light modulator, referred to as an active optical mapper, permutes and maps the light datacube voxels onto sensor pixels in an arbitrary and programmed manner. The resultant system can readily adapt the acquisition scheme to the scene, thereby maximising the measurement flexibility. Through active optical mapping, we demonstrate our approach in two niche implementations: hyperspectral imaging and ultrafast imaging.

Multidimensional photography has traditionally been restricted by their static optical architectures and measurement schemes. Here, the authors present a tunable multidimensional photography approach employing active optical mapping, which allows them to adapt the acquisition schemes to the scene.

Punching holes in light: recent progress in single-shot coded-aperture optical imaging

Single-shot coded-aperture optical imaging physically captures a code-aperture-modulated optical signal in one exposure and then recovers the scene via computational image reconstruction. Recent years have witnessed dazzling advances in various modalities in this hybrid imaging scheme in concomitant technical improvement and widespread applications in physical, chemical and biological sciences. This review comprehensively surveys state-of-the-art single-shot coded-aperture optical imaging. Based on the detected photon tags, this field is divided into six categories: planar imaging, depth imaging, light-field imaging, temporal imaging, spectral imaging, and polarization imaging. In each category, we start with a general description of the available techniques and design principles, then provide two representative examples of active-encoding and passive-encoding approaches, with a particular emphasis on their methodology and applications as well as their advantages and challenges. Finally, we envision prospects for further technical advancement in this field.

Single-shot stereo-polarimetric compressed ultrafast photography for light-speed observation of high-dimensional optical transients with picosecond resolution

Simultaneous and efficient ultrafast recording of multiple photon tags contributes to high-dimensional optical imaging and characterization in numerous fields. Existing high-dimensional optical imaging techniques that record space and polarization cannot detect the photon's time of arrival owing to the limited speeds of the state-of-the-art electronic sensors. Here, we overcome this long-standing limitation by implementing stereo-polarimetric compressed ultrafast photography (SP-CUP) to record light-speed high-dimensional events in a single exposure. Synergizing compressed sensing and streak imaging with stereoscopy and polarimetry, SP-CUP enables video-recording of five photon tags (x, y, z: space; t: time of arrival; and ψ: angle of linear polarization) at 100 billion frames per second with a picosecond temporal resolution. We applied SP-CUP to the spatiotemporal characterization of linear polarization dynamics in early-stage plasma emission from laser-induced breakdown. This system also allowed three-dimensional ultrafast imaging of the linear polarization properties of a single ultrashort laser pulse propagating in a scattering medium.

Development of a fast calibration method for image mapping spectrometry

An image mapping spectrometer (IMS) is a snapshot hyperspectral imager that simultaneously captures both the spatial (x, y) and spectral (λ) information of incoming light. The IMS maps a three-dimensional (3D) datacube (x, y, λ) to a two-dimensional (2D) detector array (x, y) for parallel measurement. To reconstruct the original 3D datacube, one must construct a lookup table that connects voxels in the datacube and pixels in the raw image. Previous calibration methods suffer from either low speed or poor image quality. We herein present a slit-scan calibration method that can significantly reduce the calibration time while maintaining high accuracy. Moreover, we quantitatively analyzed the major artifact in the IMS, the striped image, and developed three numerical methods to correct for it.

High light efficiency snapshot spectral imaging via spatial multiplexing and spectral mixing

Computational photography has been striving to capture the spectral information of the dynamic world in the last few decades. However, due to the curse of dimensionality between the 3D spectral images and the 2D imaging sensors, light-blocking components, e.g., bandpass filters or coded apertures, have to be used in spectral imaging systems to project the 3D signal into 2D measurements selectively. Accordingly, computational reconstruction is integrated to recover the underlying 3D signal. Thus, these systems suffer from low light efficiency and high computational burden, both of which cannot afford dynamic spectral imaging. In this paper, we propose a novel snapshot spectral imaging system that can dynamically capture the spectral images. The system is composed of a lens array, a notch filter array, and a monochrome camera in principle. Incoming light beams from the scene are spatially multiplexed by the lens array, spectrally mixed by the notch filter array, and captured by the monochrome camera. The two distinct characteristics, i.e., spatial multiplexing and spectral mixing, guarantee the advantages of low computational burden and high light efficiency, respectively. We further build a prototype system according to the imaging principles. The system can record two kinds of images in a single snapshot: bandstop multispectral images and a panchromatic image, which are used jointly to recover the bandpass multispectral images at few computational costs. Moreover, the proposed system is friendly with spectral super-resolution, for which we develop a theoretical demonstration. Both simulations and experiments are conducted to verify the effectiveness of the proposed system.

Snapshot projection optical tomography

We present a new plenoptic microscopy configuration for 3D snapshot imaging, which is dual telecentric and can directly record true projection images corresponding with different viewing angles. It also allows blocking high-angle stray rays without sacrificing the light collection efficiency. This configuration named as snapshot projection optical tomography (SPOT) arranges an objective lens and a microlens array (MLA) in a 4-f telecentric configuration and places an aperture stop at the back focal plane of a relay lens. We develop a forward imaging model for SPOT, which can also be applied to existing light field microscopy techniques using an MLA as tube lens. Using the developed system, we demonstrate snapshot 3D imaging of various fluorescent beads and a biological cell, which confirms the capability of SPOT for imaging specimens with an extended fluorophore distribution as well as isolated fluorochromes. The transverse and vertical resolutions are measured to be 0.8 μm and 1.6 μm, respectively.

A versatile, low-cost, snapshot multidimensional imaging approach based on structured light

The behaviour and function of dynamic samples can be investigated using optical imaging approaches with high temporal resolution and multidimensional acquisition. Snapshot techniques have been developed in order to meet these demands, however they are often designed to study a specific parameter, such as spectral properties, limiting their applicability. Here we present and demonstrate a frequency recognition algorithm for multiple exposures (FRAME) snapshot imaging approach, which can be reconfigured to capture polarization, temporal, depth-of-focus and spectral information by simply changing the filters used. FRAME is implemented by splitting the emitted light from a sample into four channels, filtering the light and then applying a unique spatial modulation encoding before recombining all the channels. The multiplexed information is collected in a single exposure using a single detector and extracted in post processing of the Fourier transform of the collected image, where each channel image is located in a distinct region of the Fourier domain. The approach allows for individual intensity control in each channel, has easily interchangeable filters and can be used in conjunction with, in principle, all 2D detectors, making it a low cost and versatile snapshot multidimensional imaging technique.

Plenoptic Face Presentation Attack Detection

The vulnerability of current face recognition systems to presentation attacks significantly limits their application in biometrics. Herein, we present a passive presentation attack detection method based on a complete plenoptic imaging system which can derive the complete plenoptic function of light rays using a single detector. Moreover, we constructed a multi-dimensional face database with 50 subjects and seven different types of presentation attacks. We experimentally demonstrated that our approach outperforms the state-of-the-art methods on all types of presentation attacks.

Dual-camera design for hyperspectral and panchromatic imaging, using a wedge shaped liquid crystal as a spectral multiplexer

In this paper, we present a new hyperspectral compact camera which is designed to have high spatial and spectral resolutions, to be vibrations tolerant, and to achieve state-of-the-art high optical throughput values compared to existing nanosatellite hyperspectral imaging payloads with space heritage. These properties make it perfect for airborne and spaceborne remote sensing tasks. The camera has both hyperspectral and panchromatic imaging capabilities, achieved by employing a wedge-shaped liquid crystal cell together with computational image processing. The hyperspectral images are acquired through passive along-track spatial scanning when no voltage is applied to the cell, and the panchromatic images are quickly acquired in a single snapshot at a high signal-to-noise ratio when the cell is voltage driven.

Snapshot hyperspectral light field imaging using image mapping spectrometry

In this Letter, we present a snapshot hyperspectral light field imaging system using a single camera. By integrating an unfocused light field camera with a snapshot hyperspectral imager, the image mapping spectrometer, we captured a five-dimensional (5D) ($x,y,u,v,\lambda $x,y,u,v,λ) ($x,y,$x,y, spatial coordinates; $u,v,$u,v, emittance angles; $\lambda ,$λ, wavelength) datacube in a single camera exposure. The corresponding volumetric image ($x,y,z$x,y,z) at each wavelength is then computed through a scale-depth space transform. We demonstrated the snapshot advantage of our system by imaging the spectral-volumetric scenes in real time.

Electro-optic imaging enables efficient wide-field fluorescence lifetime microscopy

Nanosecond temporal resolution enables new methods for wide-field imaging like time-of-flight, gated detection, and fluorescence lifetime. The optical efficiency of existing approaches, however, presents challenges for low-light applications common to fluorescence microscopy and single-molecule imaging. We demonstrate the use of Pockels cells for wide-field image gating with nanosecond temporal resolution and high photon collection efficiency. Two temporal frames are obtained by combining a Pockels cell with a pair of polarizing beam-splitters. We show multi-label fluorescence lifetime imaging microscopy (FLIM), single-molecule lifetime spectroscopy, and fast single-frame FLIM at the camera frame rate with 103-105 times higher throughput than single photon counting. Finally, we demonstrate a space-to-time image multiplexer using a re-imaging optical cavity with a tilted mirror to extend the Pockels cell technique to multiple temporal frames. These methods enable nanosecond imaging with standard optical systems and sensors, opening a new temporal dimension for wide-field low-light microscopy.

High-quality panchromatic image acquisition method for snapshot hyperspectral imaging Fourier transform spectrometer

The acquisition of high-quality panchromatic images is vital to the multi-spectral images pan sharpening, especially to snapshot imaging spectrometers with a low spatial resolution. As an aperture-division snapshot imaging spectrometer, a snapshot hyperspectral imaging Fourier transform spectrometer has the characteristic that images of all the sub-apertures share almost the same spatial information with a small shift. With these sub-images, super-resolution is possible. In this paper, a high-quality panchromatic image acquisition method is proposed. A pre-trained deep learning network is utilized without enlarging the instrument size. The training dataset is obtained experimentally, and the network is designed to realize the contrast enhancement and super-resolution simultaneously. The experimental results demonstrate that the proposed method performs well in high-quality panchromatic image acquisition.

Intraoperative multispectral and hyperspectral label-free imaging: A systematic review of in vivo clinical studies

Multispectral and hyperspectral imaging (HSI) are emerging optical imaging techniques with the potential to transform the way surgery is performed but it is not clear whether current systems are capable of delivering real-time tissue characterization and surgical guidance. We conducted a systematic review of surgical in vivo label-free multispectral and HSI systems that have been assessed intraoperatively in adult patients, published over a 10-year period to May 2018. We analysed 14 studies including 8 different HSI systems. Current in-vivo HSI systems generate an intraoperative tissue oxygenation map or enable tumour detection. Intraoperative tissue oxygenation measurements may help to predict those patients at risk of postoperative complications and in-vivo intraoperative tissue characterization may be performed with high specificity and sensitivity. All systems utilized a line-scanning or wavelength-scanning method but the spectral range and number of spectral bands employed varied significantly between studies and according to the system's clinical aim. The time to acquire a hyperspectral cube dataset ranged between 5 and 30 seconds. No safety concerns were reported in any studies. A small number of studies have demonstrated the capabilities of intraoperative in-vivo label-free HSI but further work is needed to fully integrate it into the current surgical workflow.

Light-guide snapshot imaging spectrometer for remote sensing applications

A fiber-based snapshot imaging spectrometer was developed with a maximum of 31853 (~188 x 170) spatial sampling and 61 spectral channels in the 450nm-750nm range. A compact, custom-fabricated fiber bundle was used to sample the object image at the input and create void spaces between rows at the output for dispersion. The bundle was built using multicore 6x6 fiber block ribbons. To avoid overlap between the cores in the direction of dispersion, we selected a subset of cores using two alternative approaches; a lenslet array and a photomask. To calibrate the >30000 spatial samples of the system, a rapid spatial calibration method was developed based on phase-shifting interferometry (PSI). System crosstalk and spectral resolution were also characterized. Preliminary hyperspectral imaging results of the Rice University campus landscape, obtained with the spectrometer, are presented to demonstrate the system's spectral imaging capability for distant scenes. The spectrum of different plant species with different health conditions, obtained with the spectrometer, was in accordance with reference instrument measurements. We also imaged Houston traffic to demonstrate the system's snapshot hyperspectral imaging capability. Potential applications of the system include terrestrial monitoring, land use, air pollution, water resources, and lightning spectroscopy. The fiber-based system design potentially allows tuning between spatial and spectral sampling to meet specific imaging requirements.

High-Speed Hyperspectral Video Acquisition By Combining Nyquist and Compressive Sampling

We propose a novel hybrid imaging system to acquire 4D high-speed hyperspectral (HSHS) videos with high spatial and spectral resolution. The proposed system consists of two branches: one branch performs Nyquist sampling in the temporal dimension while integrating the whole spectrum, resulting in a high-frame-rate panchromatic video; the other branch performs compressive sampling in the spectral dimension with longer exposures, resulting in a low-frame-rate hyperspectral video. Owing to the high light throughput and complementary sampling, these two branches jointly provide reliable measurements for recovering the underlying HSHS video. Moreover, the panchromatic video can be used to learn an over-complete 3D dictionary to represent each band-wise video sparsely, thanks to the inherent structural similarity in the spectral dimension. Based on the joint measurements and the self-adaptive dictionary, we further propose a simultaneous spectral sparse (3S) model to reinforce the structural similarity across different bands and develop an efficient computational reconstruction algorithm to recover the HSHS video. Both simulation and hardware experiments validate the effectiveness of the proposed approach. To the best of our knowledge, this is the first time that hyperspectral videos can be acquired at a frame rate up to 100fps with commodity optical elements and under ordinary indoor illumination.

Hyperspectral imaging in color vision research: tutorial

This tutorial offers an introduction to terrestrial and close-range hyperspectral imaging and some of its uses in human color vision research. The main types of hyperspectral cameras are described together with procedures for image acquisition, postprocessing, and calibration for either radiance or reflectance data. Image transformations are defined for colorimetric representations, color rendering, and cone receptor and postreceptor coding. Several example applications are also presented. These include calculating the color properties of scenes, such as gamut volume and metamerism, and analyzing the utility of color in observer tasks, such as identifying surfaces under illuminant changes. The effects of noise and uncertainty are considered in both image acquisition and color vision applications.

Compact snapshot optically replicating and remapping imaging spectrometer (ORRIS) using a focal plane continuous variable filter

In this Letter, a novel snapshot spectral imaging technique, optically replicating and remapping imaging spectrometer, is presented. It is based on the combination of shifting subimages by a specially designed lenslet array (LA) and filtering subimages by a focal plane continuous variable filter (CVF). The 3D datacube is recovered by just using a simple image remapping process. The use of the LA and the focal plane CVF makes the system compact and low in cost. A handheld proof-of-principle prototype has been built and demonstrated; it covers a wavelength range of 380-860 nm with 80 spectral channels with a spatial resolution of 400×400  pixels.

Channeled compressive imaging spectropolarimeter

A compressive imaging spectropolarimeter is proposed in this paper, capable of simultaneously acquiring full polarization, spatial and spectral information of the object scene. The spectral and polarization information is modulated through a combination of high-order retarders, a dispersion prism and a polarizer filter wheel. Using a random coded aperture, compressive sensing is applied to eliminate the channel crosstalk and resolution limitation of traditional channeled spectropolarimeters. The forward sensing model and inverse problem are developed. Computer simulation results are reported, followed by experimental demonstrations.

A Low-Rate Video Approach to Hyperspectral Imaging of Dynamic Scenes

The increased sensitivity of modern hyperspectral line-scanning systems has led to the development of imaging systems that can acquire each line of hyperspectral pixels at very high data rates (in the 200–400 Hz range). These data acquisition rates present an opportunity to acquire full hyperspectral scenes at rapid rates, enabling the use of traditional push-broom imaging systems as low-rate video hyperspectral imaging systems. This paper provides an overview of the design of an integrated system that produces low-rate video hyperspectral image sequences by merging a hyperspectral line scanner, operating in the visible and near infra-red, with a high-speed pan-tilt system and an integrated IMU-GPS that provides system pointing. The integrated unit is operated from atop a telescopic mast, which also allows imaging of the same surface area or objects from multiple view zenith directions, useful for bi-directional reflectance data acquisition and analysis. The telescopic mast platform also enables stereo hyperspectral image acquisition, and therefore, the ability to construct a digital elevation model of the surface. Imaging near the shoreline in a coastal setting, we provide an example of hyperspectral imagery time series acquired during a field experiment in July 2017 with our integrated system, which produced hyperspectral image sequences with 371 spectral bands, spatial dimensions of 1600 × 212, and 16 bits per pixel, every 0.67 s. A second example times series acquired during a rooftop experiment conducted on the Rochester Institute of Technology campus in August 2017 illustrates a second application, moving vehicle imaging, with 371 spectral bands, 16 bit dynamic range, and 1600 × 300 spatial dimensions every second.

HyperReconNet: Joint Coded Aperture Optimization and Image Reconstruction for Compressive Hyperspectral Imaging

Coded aperture snapshot spectral imaging (CASSI) system encodes the 3D hyperspectral image (HSI) within a single 2D compressive image and then reconstructs the underlying HSI by employing an inverse optimization algorithm, which equips with the distinct advantage of snapshot but usually results in low reconstruction accuracy. To improve the accuracy, existing methods attempt to design either alternative coded apertures or advanced reconstruction methods, but cannot connect these two aspects via a unified framework, which limits the accuracy improvement. In this paper, we propose a convolution neural network (CNN) based endto- end method to boost the accuracy by jointly optimizing the coded aperture and the reconstruction method. On the one hand, based on the nature of CASSI forward model, we design a repeated pattern for the coded aperture, whose entities are learned by acting as the network weights. On the other hand, we conduct the reconstruction through simultaneously exploiting intrinsic properties within HSI - the extensive correlations across the spatial and the spectral dimensions. By leveraging the power of deep learning, the coded aperture design and the image reconstruction are connected and optimized via a unified framework. Experimental results show that our method outperforms the state-of-the-art methods under both comprehensive quantitative metrics and perceptive quality.

Complete plenoptic imaging using a single detector

Multi-dimensional imaging is a powerful technique for many applications, such as biological analysis, remote sensing, and object recognition. Most existing multi-dimensional imaging systems rely on scanning or camera array, which make the system bulky and unstable. To some extent, these problems can be mitigated by employing compressed sensing algorithms. However, they are computationally expensive and highly rely on the ill-posed assumption that the information is sparse in a given domain. Here, we propose a snapshot spectral-volumetric imaging (SSVI) system by introducing the paradigm of light-field imaging into Fourier transform imaging spectroscopy. We demonstrate that SSVI can reconstruct a complete plenoptic function, P(x,y,z,θ,φ,λ,t), of the incoming light rays using a single detector. Compared with other multidimensional imagers, SSVI features prominent advantages in compactness, robustness, and low cost.

Single-shot ultrafast optical imaging

Single-shot ultrafast optical imaging can capture two-dimensional transient scenes in the optical spectral range at ≥100 million frames per second. This rapidly evolving field surpasses conventional pump-probe methods by possessing the real-time imaging capability, which is indispensable for recording non-repeatable and difficult-to-reproduce events and for understanding physical, chemical, and biological mechanisms. In this mini-review, we survey comprehensively the state-of-the-art single-shot ultrafast optical imaging. Based on the illumination requirement, we categorized the field into active-detection and passive-detection domains. Depending on the specific image acquisition and reconstruction strategies, these two categories are further divided into a total of six sub-categories. Under each sub-category, we describe operating principles, present representative cutting-edge techniques with a particular emphasis on their methodology and applications, and discuss their advantages and challenges. Finally, we envision prospects of technical advancement in this field.