Large-image-format computed tomography imaging spectrometer for fluorescence microscopy

HSGAN: Hyperspectral Reconstruction From RGB Images With Generative Adversarial Network

Hyperspectral (HS) reconstruction from RGB images denotes the recovery of whole-scene HS information, which has attracted much attention recently. State-of-the-art approaches often adopt convolutional neural networks to learn the mapping for HS reconstruction from RGB images. However, they often do not achieve high HS reconstruction performance across different scenes consistently. In addition, their performance in recovering HS images from clean and real-world noisy RGB images is not consistent. To improve the HS reconstruction accuracy and robustness across different scenes and from different input images, we present an effective HSGAN framework with a two-stage adversarial training strategy. The generator is a four-level top-down architecture that extracts and combines features on multiple scales. To generalize well to real-world noisy images, we further propose a spatial-spectral attention block (SSAB) to learn both spatial-wise and channel-wise relations. We conduct the HS reconstruction experiments from both clean and real-world noisy RGB images on five well-known HS datasets. The results demonstrate that HSGAN achieves superior performance to existing methods. Please visit https://github.com/zhaoyuzhi/HSGAN to try our codes.

CTIS spectral image reconstruction technology based on slit-scanning architecture

The computed tomography imaging spectrometer (CTIS) is a snapshot imaging spectrometer, excelling in dynamic detection tasks. It can capture two-dimensional spatial information and spectrally compressed information of a target within a single exposure time. However, traditional CTIS image reconstruction algorithms suffer from missing-cone problem, which reduces the accuracy of spectral reconstruction. In recent years, deep learning has been applied to CTIS spectral image reconstruction, significantly improving spectral reconstruction accuracy compared to traditional algorithms. However, due to the missing-cone problem, it is difficult to accurately recover the truth of spectral data cube in the real scene. Currently, most CTIS neural network reconstruction models are trained using simulated datasets of spectral data cubes and diffractive images. Because these data differ significantly from real data under actual application conditions, the established models may not be effectively applicable to real-world scenes. Therefore, we propose a new CTIS system based on slit-scanning architecture utilizing an adjustable slit aperture to obtain the real spectral data cube of the target while maintaining the simplicity of the CTIS structure. By limiting the field of view (FOV) through the slit, the area of diffraction overlap can be reduced, thereby enhancing the accuracy of CTIS spectral reconstruction using the expectation-maximization (EM) algorithm. This architecture allows us to obtain accurate spectral cubes that match the CTIS diffractive image of real-world scenes, providing a real dataset for training the reconstruction network. A prototype has been built to demonstrate the feasibility of our proposed solution. Furthermore, we also constructed a residual network based on multi-scale and attention mechanism. This network is trained using a combination of simulated and real spectral imaging data. Compared to the reconstruction performance of the EM algorithm and convolutional neural networks, our approach demonstrates superior spectral reconstruction accuracy, validating the importance of real spectral data in CTIS spectral reconstruction tasks.

Snapshot spectral imaging: from spatial-spectral mapping to metasurface-based imaging

Snapshot spectral imaging technology enables the capture of complete spectral information of objects in an extremely short period of time, offering wide-ranging applications in fields requiring dynamic observations such as environmental monitoring, medical diagnostics, and industrial inspection. In the past decades, snapshot spectral imaging has made remarkable breakthroughs with the emergence of new computational theories and optical components. From the early days of using various spatial-spectral data mapping methods, they have evolved to later attempts to encode various dimensions of light, such as amplitude, phase, and wavelength, and then computationally reconstruct them. This review focuses on a systematic presentation of the system architecture and mathematical modeling of these snapshot spectral imaging techniques. In addition, the introduction of metasurfaces expands the modulation of spatial-spectral data and brings advantages such as system size reduction, which has become a research hotspot in recent years and is regarded as the key to the next-generation snapshot spectral imaging techniques. This paper provides a systematic overview of the applications of metasurfaces in snapshot spectral imaging and provides an outlook on future directions and research priorities.

Coded aperture snapshot hyperspectral light field tomography

Multidimensional imaging has emerged as a powerful technology capable of simultaneously acquiring spatial, spectral, and depth information about a scene. However, existing approaches often rely on mechanical scanning or multi-modal sensing configurations, leading to prolonged acquisition times and increased system complexity. Coded aperture snapshot spectral imaging (CASSI) has introduced compressed sensing to recover three-dimensional (3D) spatial-spectral datacubes from single snapshot two-dimensional (2D) measurements. Despite its advantages, the reconstruction problem remains severely underdetermined due to the high compression ratio, resulting in limited spatial and spectral reconstruction quality. To overcome this challenge, we developed a novel two-stage cascaded compressed sensing scheme called coded aperture snapshot hyperspectral light field tomography (CASH-LIFT). By appropriately distributing the computation load to each stage, this method utilizes the compressibility of natural scenes in multiple domains, reducing the ill-posed nature of datacube recovery and achieving enhanced spatial resolution, suppressed aliasing artifacts, and improved spectral fidelity. Additionally, leveraging the snapshot 3D imaging capability of LIFT, our approach efficiently records a five-dimensional (5D) plenoptic function in a single snapshot.

Handheld snapshot multi-spectral camera at tens-of-megapixel resolution

Multi-spectral imaging is a fundamental tool characterizing the constituent energy of scene radiation. However, current multi-spectral video cameras cannot scale up beyond megapixel resolution due to optical constraints and the complexity of the reconstruction algorithms. To circumvent the above issues, we propose a tens-of-megapixel handheld multi-spectral videography approach (THETA), with a proof-of-concept camera achieving 65-megapixel videography of 12 wavebands within visible light range. The high performance is brought by multiple designs: We propose an imaging scheme to fabricate a thin mask for encoding spatio-spectral data using a conventional film camera. Afterwards, a fiber optic plate is introduced for building a compact prototype supporting pixel-wise encoding with a large space-bandwidth product. Finally, a deep-network-based algorithm is adopted for large-scale multi-spectral data decoding, with the coding pattern specially designed to facilitate efficient coarse-to-fine model training. Experimentally, we demonstrate THETA's advantageous and wide applications in outdoor imaging of large macroscopic scenes.

Generalized central slice theorem perspective on Fourier-transform spectral imaging at a sub-Nyquist sampling rate

Fourier-transform spectral imaging captures frequency-resolved images with high spectral resolution, broad spectral range, high photon flux, and low stray light. In this technique, spectral information is resolved by taking Fourier transformation of the interference signals of two copies of the incident light at different time delays. The time delay should be scanned at a high sampling rate beyond the Nyquist limit to avoid aliasing, at the price of low measurement efficiency and stringent requirements on motion control for time delay scan. Here we propose, what we believe to be, a new perspective on Fourier-transform spectral imaging based on a generalized central slice theorem analogous to computerized tomography, using an angularly dispersive optics decouples measurements of the spectral envelope and the central frequency. Thus, as the central frequency is directly determined by the angular dispersion, the smooth spectral-spatial intensity envelope is reconstructed from interferograms measured at a sub-Nyquist time delay sampling rate. This perspective enables high-efficiency hyperspectral imaging and even spatiotemporal optical field characterization of femtosecond laser pulses without a loss of spectral and spatial resolutions.

Compact and ultracompact spectral imagers: technology and applications in biomedical imaging

Significance

Spectral imaging, which includes hyperspectral and multispectral imaging, can provide images in numerous wavelength bands within and beyond the visible light spectrum. Emerging technologies that enable compact, portable spectral imaging cameras can facilitate new applications in biomedical imaging.

Aim

With this review paper, researchers will (1) understand the technological trends of upcoming spectral cameras, (2) understand new specific applications that portable spectral imaging unlocked, and (3) evaluate proper spectral imaging systems for their specific applications.

Approach

We performed a comprehensive literature review in three databases (Scopus, PubMed, and Web of Science). We included only fully realized systems with definable dimensions. To best accommodate many different definitions of "compact," we included a table of dimensions and weights for systems that met our definition.

Results

There is a wide variety of contributions from industry, academic, and hobbyist spaces. A variety of new engineering approaches, such as Fabry-Perot interferometers, spectrally resolved detector array (mosaic array), microelectro-mechanical systems, 3D printing, light-emitting diodes, and smartphones, were used in the construction of compact spectral imaging cameras. In bioimaging applications, these compact devices were used for in vivo and ex vivo diagnosis and surgical settings.

Conclusions

Compact and ultracompact spectral imagers are the future of spectral imaging systems. Researchers in the bioimaging fields are building systems that are low-cost, fast in acquisition time, and mobile enough to be handheld.

CTIS-GAN: computed tomography imaging spectrometry based on a generative adversarial network

Computed tomography imaging spectrometry (CTIS) is a snapshot hyperspectral imaging technique that can obtain a three-dimensional (2D spatial + 1D spectral) data cube of the scene captured within a single exposure. The CTIS inversion problem is typically highly ill-posed and is usually solved by time-consuming iterative algorithms. This work aims to take the full advantage of the recent advances in deep-learning algorithms to dramatically reduce the computational cost. For this purpose, a generative adversarial network is developed and integrated with self-attention, which cleverly exploits the clearly utilizable features of zero-order diffraction of CTIS. The proposed network is able to reconstruct a CTIS data cube (containing 31 spectral bands) in milliseconds with a higher quality than traditional methods and the state-of-the-art (SOTA). Simulation studies based on real image data sets confirmed the robustness and efficiency of the method. In numerical experiments with 1000 samples, the average reconstruction time for a single data cube was ∼16m s. The robustness of the method against noise is also confirmed by numerical experiments with different levels of Gaussian noise. The CTIS generative adversarial network framework can be easily extended to solve CTIS problems with larger spatial and spectral dimensions, or migrated to other compressed spectral imaging modalities.

Assessment of a computed tomography imaging spectrometer using an optimized expectation-maximization algorithm

We designed and built a homemade computed tomography imaging spectrometer (CTIS) of 250×250pixels of spatial resolution and 2 nm spectral resolution. The optical design considers a CTIS optical array coupled to a digital reflex camera. We reconstructed the intensity spectra of a fluorescent source, the diffuse reflectance of a ColorChecker, and samples of Capsicum annuum of three different colors, using the expectation-maximization sequential algorithm, optimized utilizing an array of indices to reduce the reconstruction time. The results obtained with a ColorChecker indicate a high positive correlation of 0.9745 with an average residual difference of 1.31% concerning the spectra obtained with a commercial integrating sphere spectrometer. The feasibility of the proposed CTIS system shows how to detect and evaluate the physiological changes resulting from the decomposition of the green fruit of the Capsicum annuum in a range from 500 to 650 nm.

Coded Hyperspectral Image Reconstruction Using Deep External and Internal Learning

To solve the low spatial and/or temporal resolution problem which the conventional hyperspectral cameras often suffer from, coded hyperspectral imaging systems have attracted more attention recently. Recovering a hyperspectral image (HSI) from its corresponding coded image is an ill-posed inverse problem, and learning accurate prior of HSI is essential to solve this inverse problem. In this paper, we present an effective convolutional neural network (CNN) based method for coded HSI reconstruction, which learns the deep prior from the external dataset as well as the internal information of input coded image with spatial-spectral constraint. Specifically, we first develop a CNN-based channel attention reconstruction network to effectively exploit the spatial-spectral correlation of the HSI. Then, the reconstruction network is learned by leveraging an arbitrary external hyperspectral dataset to exploit the general spatial-spectral correlation under adversarial loss. Finally, we customize the network by internal learning with spatial-spectral constraint and total variation regularization for each coded image, which can make use of the internal imaging model to learn specific prior for current desirable image and effectively avoids overfitting. Experimental results using both synthetic data and real images show that our method outperforms the state-of-the-art methods on several popular coded hyperspectral imaging systems under both comprehensive quantitative metrics and perceptive quality.

Joint Camera Spectral Response Selection and Hyperspectral Image Recovery

Hyperspectral image (HSI) recovery from a single RGB image has attracted much attention, whose performance has recently been shown to be sensitive to the camera spectral response (CSR). In this paper, we present an efficient convolutional neural network (CNN) based method, which can jointly select the optimal CSR from a candidate dataset and learn a mapping to recover HSI from a single RGB image captured with this algorithmically selected camera under multi-chip or single-chip setups. Given a specific CSR, we first present a HSI recovery network, which accounts for the underlying characteristics of the HSI, including spectral nonlinear mapping and spatial similarity. Later, we append a CSR selection layer onto the recovery network, and the optimal CSR under both multi-chip and single-chip setups can thus be automatically determined from the network weights under the nonnegative sparse constraint. Experimental results on three hyperspectral datasets and two camera spectral response datasets demonstrate that our HSI recovery network outperforms state-of-the-art methods in terms of both quantitative metrics and perceptive quality, and the selection layer always returns a CSR consistent to the best one determined by exhaustive search. Finally, we show that our method can also perform well in the real capture system, and collect a hyperspectral flower dataset to evaluate the effect from HSI recovery on classification problem.

Snapshot hyperspectral light field tomography

We present snapshot hyperspectral light field tomography (Hyper-LIFT), a highly efficient method in recording a 5D (x, y, spatial coordinates; θ, φ, angular coordinates; λ, wavelength) plenoptic function. Using a Dove prism array and a cylindrical lens array, we simultaneously acquire multi-angled 1D en face projections of the object like those in standard sparse-view computed tomography. We further disperse those projections and measure the spectra in parallel using a 2D image sensor. Within a single snapshot, the resultant system can capture a 5D data cube with 270 × 270 × 4 × 4 × 360 voxels. We demonstrated the performance of Hyper-LIFT in imaging spectral volumetric scenes.

Out-of-Phase Imaging after Optical Modulation (OPIOM) for Multiplexed Fluorescence Imaging Under Adverse Optical Conditions

Fluorescence imaging has become a powerful tool for observations in biology. Yet it has also encountered limitations to overcome optical interferences of ambient light, autofluorescence, and spectrally interfering fluorophores. In this account, we first examine the current approaches which address these limitations. Then we more specifically report on Out-of-Phase Imaging after Optical Modulation (OPIOM), which has proved attractive for highly selective multiplexed fluorescence imaging even under adverse optical conditions. After exposing the OPIOM principle, we detail the protocols for successful OPIOM implementation.

Fast Hyperspectral Image Recovery of Dual-Camera Compressive Hyperspectral Imaging via Non-Iterative Subspace-Based Fusion

Coded aperture snapshot spectral imaging (CASSI) is a promising technique for capturing three-dimensional hyperspectral images (HSIs), in which algorithms are used to perform the inverse problem of HSI reconstruction from a single coded two-dimensional (2D) measurement. Due to the ill-posed nature of this problem, various regularizers have been exploited to reconstruct 3D data from 2D measurements. Unfortunately, the accuracy and computational complexity are unsatisfactory. One feasible solution is to utilize additional information such as the RGB measurement in CASSI. Considering the combined CASSI and RGB measurements, in this paper, we propose a fusion model for HSI reconstruction. Specifically, we investigate the low-dimensional spectral subspace property of HSIs composed of a spectral basis and spatial coefficients. In particular, the RGB measurement is utilized to estimate the coefficients, while the CASSI measurement is adopted to provide the spectral basis. We further propose a patch processing strategy to enhance the spectral low-rank property of HSIs. The optimization of the proposed model requires neither iteration nor the spectral sensing matrix of the RGB detector. Extensive experiments on both simulated and real HSI datasets demonstrate that our proposed method not only outperforms previous state-of-the-art (iterative algorithms) methods in quality but also speeds up the reconstruction by more than 5000 times.

Snapshot Imaging Spectrometer Based on Pixel-Level Filter Array (PFA)

Snapshot spectral imaging technology plays an important role in many fields. However, most existing snapshot imaging spectrometers have the shortcomings of a large volume or heavy computational burden. In this paper, we present a novel snapshot imaging spectrometer based on the pixel-level filter array (PFA), which can simultaneously obtain both spectral and spatial information. The system is composed of a fore-optics, a PFA, a relay lens, and a monochromatic sensor. The incoming light first forms an intermediate image on the PFA through the fore-optics. Then, the relay lens reimages the spectral images on the PFA onto the monochromatic sensor. Through the use of the PFA, we can capture a three-dimensional (spatial coordinates and wavelength) datacube in a single exposure. Compared with existing technologies, our system possesses the advantages of a simple implementation, low cost, compact structure, and high energy efficiency by removing stacked dispersive or interferometric elements. Moreover, the characteristic of the direct imaging mode ensures the low computational burden of the system, thus shortening the imaging time. The principle and design of the system are described in detail. An experimental prototype is built and field experiments are carried out to verify the feasibility of the proposed scheme.

On the value of CTIS imagery for neural-network-based classification: a simulation perspective

The computed tomography imaging spectrometer (CTIS) is a snapshot hyperspectral imaging system. Its output is a 2D image of multiplexed spatiospectral projections of the hyperspectral cube of the scene. Traditionally, the 3D cube is reconstructed from this image before further analysis. In this paper, we show that it is possible to learn information directly from the CTIS raw output, by training a neural network to perform binary classification on such images. The use case we study is an agricultural one, as snapshot imagery is used substantially in this field: the detection of apple scab lesions on leaves. To train the network appropriately and to study several degrees of scab infection, we simulated CTIS images of scabbed leaves. This was made possible with a novel CTIS simulator, where special care was taken to preserve realistic pixel intensities compared to true images. To the best of our knowledge, this is the first application of compressed learning on a simulated CTIS system.

1-D imaging of rotation-vibration non-equilibrium from pure rotational ultrafast coherent anti-Stokes Raman scattering

We present one-dimensional (1-D) imaging of rotation-vibration non-equilibrium measured by two-beam pure rotational hybrid femtosecond/picosecond coherent anti-Stokes Raman scattering (fs/ps CARS). Simultaneous measurements of the spatial distribution of molecular rotation-vibration non-equilibrium are critical for understanding molecular energy transfer in low temperature plasmas and hypersonic flows. However, non-equilibrium CARS thermometry until now was limited to point measurements. The red shift of rotational energy levels by vibrational excitation was used to determine the rotational and vibrational temperatures from 1-D images of the pure rotational spectrum. Vibrational temperatures up to 5500 K were detected in a CH₄/N₂ nanosecond-pulsed pin-to-pin plasma within 2 mm near the cathode. This approach enables study of non-equilibrium systems with 40 µm spatial resolution.

Development of a fast calibration method for image mapping spectrometry

An image mapping spectrometer (IMS) is a snapshot hyperspectral imager that simultaneously captures both the spatial (x, y) and spectral (λ) information of incoming light. The IMS maps a three-dimensional (3D) datacube (x, y, λ) to a two-dimensional (2D) detector array (x, y) for parallel measurement. To reconstruct the original 3D datacube, one must construct a lookup table that connects voxels in the datacube and pixels in the raw image. Previous calibration methods suffer from either low speed or poor image quality. We herein present a slit-scan calibration method that can significantly reduce the calibration time while maintaining high accuracy. Moreover, we quantitatively analyzed the major artifact in the IMS, the striped image, and developed three numerical methods to correct for it.

Spectral reflectance recovery using optimal illuminations

The spectral reflectance of objects provides intrinsic information on material properties that have been proven beneficial in a diverse range of applications, e.g., remote sensing, agriculture and diagnostic medicine, to name a few. Existing methods for the spectral reflectance recovery from RGB or monochromatic images either ignore the effect from the illumination or implement/optimize the illumination under the linear representation assumption of the spectral reflectance. In this paper, we present a simple and efficient convolutional neural network (CNN)-based spectral reflectance recovery method with optimal illuminations. Specifically, we design illumination optimization layer to optimally multiplex illumination spectra in a given dataset or to design the optimal one under physical restrictions. Meanwhile, we develop the nonlinear representation for spectral reflectance in a data-driven way and jointly optimize illuminations under this representation in a CNN-based end-to-end architecture. Experimental results on both synthetic and real data show that our method outperforms the state-of-the-arts and verifies the advantages of deeply optimal illumination and nonlinear representation of the spectral reflectance.

Light-guide snapshot imaging spectrometer for remote sensing applications

A fiber-based snapshot imaging spectrometer was developed with a maximum of 31853 (~188 x 170) spatial sampling and 61 spectral channels in the 450nm-750nm range. A compact, custom-fabricated fiber bundle was used to sample the object image at the input and create void spaces between rows at the output for dispersion. The bundle was built using multicore 6x6 fiber block ribbons. To avoid overlap between the cores in the direction of dispersion, we selected a subset of cores using two alternative approaches; a lenslet array and a photomask. To calibrate the >30000 spatial samples of the system, a rapid spatial calibration method was developed based on phase-shifting interferometry (PSI). System crosstalk and spectral resolution were also characterized. Preliminary hyperspectral imaging results of the Rice University campus landscape, obtained with the spectrometer, are presented to demonstrate the system's spectral imaging capability for distant scenes. The spectrum of different plant species with different health conditions, obtained with the spectrometer, was in accordance with reference instrument measurements. We also imaged Houston traffic to demonstrate the system's snapshot hyperspectral imaging capability. Potential applications of the system include terrestrial monitoring, land use, air pollution, water resources, and lightning spectroscopy. The fiber-based system design potentially allows tuning between spatial and spectral sampling to meet specific imaging requirements.

Multicolor fluorescent imaging by space-constrained computational spectral imaging

Spectral imaging is a powerful technique used to simultaneously study multiple fluorophore labels with overlapping emissions. Here, we present a computational spectral imaging method, which uses sample spatial fluorescence information as a reconstruction constraint. Our method addresses both the under-sampling issue of compressive spectral imaging and the low throughput issue of scanning spectral imaging. With simulated and experimental data, we have demonstrated the reconstruction precision of our method in two and three-color imaging. We have experimentally validated this method for differentiating cellular structures labeled with two red-colored fluorescent proteins, tdTomato and mCherry, which have highly overlapping emission spectra. Our method has the advantage of totally free wavelength choice and can also be combined with conventional filter-based sequential multi-color imaging to further improve multiplexing capability.

HyperReconNet: Joint Coded Aperture Optimization and Image Reconstruction for Compressive Hyperspectral Imaging

Coded aperture snapshot spectral imaging (CASSI) system encodes the 3D hyperspectral image (HSI) within a single 2D compressive image and then reconstructs the underlying HSI by employing an inverse optimization algorithm, which equips with the distinct advantage of snapshot but usually results in low reconstruction accuracy. To improve the accuracy, existing methods attempt to design either alternative coded apertures or advanced reconstruction methods, but cannot connect these two aspects via a unified framework, which limits the accuracy improvement. In this paper, we propose a convolution neural network (CNN) based endto- end method to boost the accuracy by jointly optimizing the coded aperture and the reconstruction method. On the one hand, based on the nature of CASSI forward model, we design a repeated pattern for the coded aperture, whose entities are learned by acting as the network weights. On the other hand, we conduct the reconstruction through simultaneously exploiting intrinsic properties within HSI - the extensive correlations across the spatial and the spectral dimensions. By leveraging the power of deep learning, the coded aperture design and the image reconstruction are connected and optimized via a unified framework. Experimental results show that our method outperforms the state-of-the-art methods under both comprehensive quantitative metrics and perceptive quality.

Spatially heterodyned snapshot imaging spectrometer

Snapshot hyperspectral imaging Fourier transform (SHIFT) spectrometers are a promising technology in optical detection and target identification. For any imaging spectrometer, spatial, spectral, and temporal resolution, along with form factor, power consumption, and computational complexity are often the design considerations for a desired application. Motivated by the need for high spectral resolution systems, capable of real-time implementation, we demonstrate improvements to the spectral resolution and computation trade-space. In this paper, we discuss the implementation of spatial heterodyning, using polarization gratings, to improve the spectral resolution trade space of a SHIFT spectrometer. Additionally, we employ neural networks to reduce the computational complexity required for data reduction, as appropriate for real-time imaging applications. Ultimately, with this method we demonstrate an 87% decrease in processing steps when compared to Fourier techniques. Additionally, we show an 80% reduction in spectral reconstruction error and a 30% increase in spatial fidelity when compared to linear operator techniques.

High resolution snapshot imaging spectrometer using a fusion algorithm based on grouping principal component analysis

We reported a high resolution snapshot imaging spectrometer (HR-SIS) and a fusion algorithm based on the properties of the HR-SIS. The system consists of an imaging branch and a spectral branch. The imaging branch captures a high spatial resolution panchromatic image with 680 × 680 pixels, while the spectral branch acquires a low spatial resolution spectral image with spectral resolution of 250 cm^-1. By using a fusion algorithm base on grouping principal component analysis, the spectral image is highly improved in spatial resolution. Experimental results demonstrated that the performance of the proposed algorithm is competitive with other state-of-the-art algorithms. The computing time for a single frame is less than 1 min with an Intel Core i5-4200H CPU, which can be further reduced by utilizing a graphics processing unit (GPU).

A review of snapshot multidimensional optical imaging: measuring photon tags in parallel

Multidimensional optical imaging has seen remarkable growth in the past decade. Rather than measuring only the two-dimensional spatial distribution of light, as in conventional photography, multidimensional optical imaging captures light in up to nine dimensions, providing unprecedented information about incident photons' spatial coordinates, emittance angles, wavelength, time, and polarization. Multidimensional optical imaging can be accomplished either by scanning or parallel acquisition. Compared with scanning-based imagers, parallel acquisition-also dubbed snapshot imaging-has a prominent advantage in maximizing optical throughput, particularly when measuring a datacube of high dimensions. Here, we first categorize snapshot multidimensional imagers based on their acquisition and image reconstruction strategies, then highlight the snapshot advantage in the context of optical throughput, and finally we discuss their state-of-the-art implementations and applications.

Optical hyperspectral imaging in microscopy and spectroscopy - a review of data acquisition

Rather than simply acting as a photographic camera capturing two-dimensional (x, y) intensity images or a spectrometer acquiring spectra (λ), a hyperspectral imager measures entire three-dimensional (x, y, λ) datacubes for multivariate analysis, providing structural, molecular, and functional information about biological cells or tissue with unprecedented detail. Such data also gives clinical insights for disease diagnosis and treatment. We summarize the principles underpinning this technology, highlight its practical implementation, and discuss its recent applications at microscopic to macroscopic scales. Datacube acquisition strategies in hyperspectral imaging x, y, spatial coordinates; λ, wavelength.

Dual-camera design for coded aperture snapshot spectral imaging

Coded aperture snapshot spectral imaging (CASSI) provides an efficient mechanism for recovering 3D spectral data from a single 2D measurement. However, since the reconstruction problem is severely underdetermined, the quality of recovered spectral data is usually limited. In this paper we propose a novel dual-camera design to improve the performance of CASSI while maintaining its snapshot advantage. Specifically, a beam splitter is placed in front of the objective lens of CASSI, which allows the same scene to be simultaneously captured by a grayscale camera. This uncoded grayscale measurement, in conjunction with the coded CASSI measurement, greatly eases the reconstruction problem and yields high-quality 3D spectral data. Both simulation and experimental results demonstrate the effectiveness of the proposed method.

Diagnostic Imaging in Flames with Instantaneous Planar Coherent Raman Spectroscopy

Spatial mapping of temperature and molecular species concentrations is vitally important in studies of gaseous chemically reacting flows. Temperature marks the evolution of heat release and energy transfer, while species concentration gradients provide critical information on mixing and chemical reaction. Coherent anti-Stokes Raman spectroscopy (CARS) was pioneered in measurements of such processes almost 40 years ago and is authoritative in terms of the accuracy and precision it may provide. While a reacting flow is fully characterized in three-dimensional space, a limitation of CARS has been its applicability as a point-wise measurement technique, motivating advancement toward CARS imaging, and attempts have been made considering one-dimensional probing. Here, we report development of two-dimensional CARS, with the first diagnostics of a planar field in a combusting flow within a single laser pulse, resulting in measured isotherms ranging from 450 K up to typical hydrocarbon flame temperatures of about 2000 K with chemical mapping of O2 and N2.

Dual-coded compressive hyperspectral imaging

This Letter presents a new snapshot approach to hyperspectral imaging via dual-optical coding and compressive computational reconstruction. We demonstrate that two high-speed spatial light modulators, located conjugate to the image and spectral plane, respectively, can code the hyperspectral datacube into a single sensor image such that the high-resolution signal can be recovered in postprocessing. We show various applications by designing different optical modulation functions, including programmable spatially varying color filtering, multiplexed hyperspectral imaging, and high-resolution compressive hyperspectral imaging.

Snapshot 3D optical coherence tomography system using image mapping spectrometry

A snapshot 3-Dimensional Optical Coherence Tomography system was developed using Image Mapping Spectrometry. This system can give depth information (Z) at different spatial positions (XY) within one camera integration time to potentially reduce motion artifact and enhance throughput. The current (x,y,λ) datacube of (85×356×117) provides a 3D visualization of sample with 400 μm depth and 13.4 μm in transverse resolution. Axial resolution of 16.0 μm can also be achieved in this proof-of-concept system. We present an analysis of the theoretical constraints which will guide development of future systems with increased imaging depth and improved axial and lateral resolutions.

Spatial and spectral performance of a chromotomosynthetic hyperspectral imaging system

The spatial and spectral resolutions achievable by a prototype rotating prism chromotomosynthetic imaging (CTI) system operating in the visible spectrum are described. The instrument creates hyperspectral imagery by collecting a set of 2D images with each spectrally projected at a different rotation angle of the prism. Mathematical reconstruction techniques that have been well tested in the field of medical physics are used to reconstruct the data to produce the 3D hyperspectral image. The instrument operates with a 100 mm focusing lens in the spectral range of 400-900 nm with a field of view of 71.6 mrad and angular resolution of 0.8-1.6 μrad. The spectral resolution is 0.6 nm at the shortest wavelengths, degrading to over 10 nm at the longest wavelengths. Measurements using a point-like target show that performance is limited by chromatic aberration. The system model is slightly inaccurate due to poor estimation of detector spatial resolution, this is corrected based on results improving model performance. As with traditional dispersion technology, calibration of the transformed wavelength axis is required, though with this technology calibration improves both spectral and spatial resolution. While this prototype does not operate at high speeds, components exist which will allow for CTI systems to generate hyperspectral video imagery at rates greater than 100 Hz.

Finite conjugate embedded relay lens hyperspectral imaging system (ERL-HIS)

We present a novel embedded relay lens hyperspectral imaging system (ERL-HIS) with high spectral resolution (nominal spectral resolution of 2.8 nm) and spatial resolution (30 μm×8 μm) that transfers the scanning plane to an additional imaging plane through the internal relay lens so as to alleviate all outside moving parts for the scanning mechanism used in the traditional HIS, where image scanning is achieved by the relative movement between the object and hyperspectrometer. The ERL-HIS also enables high-speed scanning and can attach to a variety of optical modules for versatile applications. Here, we also demonstrate an application of the proposed ERL-HIS attached to a microscopic system for observing autofluorescent images of sliced cancer tissue samples.

High-throughput detection of immobilized plasmonic nanoparticles by a hyperspectral imaging system based on Fourier transform spectrometry

To facilitate the application of plasmonic nanoparticles (PNPs) in high-throughput detection, we develop a hyperspectral imaging system (HSIS) combining dark-filed microscopy and imaging Fourier transform spectrometry to measure scattering spectra from immobilized PNPs. The current setup has acquisition time of 5 seconds and spectral resolution of 21.4 nm at 532.1 nm. We demonstrate the applicability of the HSIS in conjunction with spectral data analysis to quantify multiple types of PNPs and detect small changes in localized surface plasmon resonance wavelengths of PNPs due to changes in the environmental refractive index.

Multiframe image estimation for coded aperture snapshot spectral imagers

A coded aperture snapshot spectral imager (CASSI) estimates the three-dimensional spatiospectral data cube from a snapshot two-dimensional coded projection, assuming that the scene is spatially and spectrally sparse. For less spectrally sparse scenes, we show that the use of multiple nondegenerate snapshots can make data cube recovery less ill-posed, yielding improved spatial and spectral reconstruction fidelity. Additionally, data acquisition can be easily scaled to meet the time/resolution requirements of the scene with little modification or extension of the original CASSI hardware. A multiframe reconstruction of a 640 × 480 × 53 voxel datacube with 450-650 nm white-light illumination of a scene reveals substantial improvement in the reconstruction fidelity, with limited increase in acquisition and reconstruction time.

Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy

A snapshot Image Mapping Spectrometer (IMS) with high sampling density is developed for hyperspectral microscopy, measuring a datacube of dimensions 285 x 285 x 60 (x, y, lambda). The spatial resolution is approximately 0.45 microm with a FOV of 100 x 100 microm(2). The measured spectrum is from 450 nm to 650 nm and is sampled by 60 spectral channels with average sampling interval approximately 3.3 nm. The channel's spectral resolution is approximately 8nm. The spectral imaging results demonstrate the potential of the IMS for real-time cellular fluorescence imaging.

Optical Design for a Spatial-Spectral Volume Holographic Imaging System

Spatial Spectral Holographic imaging system (S(2)-VHIS) is a promising alternative to confocal microscopy due to its capabilities to simultaneously image several sample depths with high resolution. However, the field of view of previously presented S(2)-VHIS prototypes has been restricted to less than 200μm. This paper presents experimental results of an improved S(2)-VHIS design which have a field of view of ~1mm while maintaining high resolution and dynamic range.

Development of image mappers for hyperspectral biomedical imaging applications

A new design and fabrication method is presented for creating large-format (>100 mirror facets) image mappers for a snapshot hyperspectral biomedical imaging system called an image mapping spectrometer (IMS). To verify this approach a 250 facet image mapper with 25 multiple-tilt angles is designed for a compact IMS that groups the 25 subpupils in a 5 x 5 matrix residing within a single collecting objective's pupil. The image mapper is fabricated by precision diamond raster fly cutting using surface-shaped tools. The individual mirror facets have minimal edge eating, tilt errors of <1 mrad, and an average roughness of 5.4 nm.

Compact Image Slicing Spectrometer (ISS) for hyperspectral fluorescence microscopy

An image slicing spectrometer (ISS) for microscopy applications is presented. Its principle is based on the redirecting of image zones by specially organized thin mirrors within a custom fabricated component termed an image slicer. The demonstrated prototype can simultaneously acquire a 140 nm spectral range within its 2D field of view from a single image. The spectral resolution of the system is 5.6 nm. The FOV and spatial resolution of the ISS depend on the selected microscope objective and for the results presented is 45 x 45 microm(2) and 0.45 microm respectively. This proof-of-concept system can be easily improved in the future for higher (both spectral and spatial) resolution imaging. The system requires no scanning and minimal post data processing. In addition, the reflective nature of the image slicer and use of prisms for spectral dispersion make the system light efficient. Both of the above features are highly valuable for real time fluorescent-spectral imaging in biological and diagnostic applications.

A heuristic technique for CTIS image reconstruction

An iterative method is presented for computed tomography imaging spectrometer (CTIS) image reconstruction in the presence of both photon noise in the image and postdetection Gaussian system noise. The new algorithm, which assumes the transfer matrix of the system has a particular structure, is evaluated experimentally with the result that it is significantly better, for larger problems, than both the multiplicative algebraic reconstruction technique (MART) and the mixed-expectation image-reconstruction technique (MERT) with respect to accuracy and computation time.

Snapshot hyperspectral imaging in ophthalmology

Retinal imaging spectroscopy can provide functional maps using chromophore spectra. For example, oxygen saturation maps show ischemic areas from diabetes and venous occlusions. Obtaining retinal spatial-spectral data has been difficult due to saccades and long data acquisition times (>5 s). We present a snapshot imaging spectrometer with far-reaching applicability that acquires a complete spatial-spectral image cube in approximately 3 ms from 450 to 700 nm with 50 bands, eliminating motion artifacts and pixel misregistration. Current retinal spectral imaging approaches are incapable of true snapshot operation over a wide spectral range with a large number of spectral bands. Coupled to a fundus camera, the instrument returns true color retinal images for comparison to standard fundus images and for image validation while the patient is still dilated. Oxygen saturation maps were obtained with a three-wavelength algorithm: for healthy subjects arteries were approximately 95% and veins 30 to 35% less. The instrument is now undergoing clinical trials.

Linear operator theory of channeled spectropolarimetry

Channeled spectropolarimetry is a snapshot method of measuring the spectral and polarization content of light. Wave-number domain amplitude modulation is employed to encode all four Stokes component spectra into a single optical power spectrum. We model the channeled spectropolarimeter as a linear operator, which facilitates treatment of nonideal effects and provides a convenient framework for simulations, calibration, and reconstruction. The operator's singular value decomposition is treated with analytic and computational approaches. This analysis highlights the importance of the choice of object space in constraining and imparting prior knowledge to linear reconstructions of data from underdetermined systems.

Linear calibration and reconstruction techniques for channeled spectropolarimetry

Channeled spectropolarimetry is a novel method of measuring the spectral and polarization content of light. It employs amplitude modulation to encode all four Stokes component spectra into a single optical power spectrum. We describe a practical approach to system calibration and object reconstruction, which is able to account for important non-ideal effects. These include dispersion in retarder materials and limited spectral resolution in the incorporated spectrometer. The spectropolarimeter is modeled as a linear operator, represented in practice by a matrix. The matrix is estimated in the calibration, and pseudoinverted subject to a constraint on object space for reconstructions. Experimental results are shown and compared with reference measurements. An example is given of the technique's application to the characterization of time-varying, stress-induced birefringence.

Phase grating design for a dual-band snapshot imaging spectrometer

Infrared spectral features have proved useful in the identification of threat objects. Dual-band focal-plane arrays (FPAs) have been developed in which each pixel consists of superimposed midwave and long-wave photodetectors [Dyer and Tidrow, Conference on Infrared Detectors and Focal Plane Arrays (SPIE, Bellingham, Wash., 1999), pp. 434-440]. Combining dual-band FPAs with imaging spectrometers capable of interband hyperspectral resolution greatly improves spatial target discrimination. The computed-tomography imaging spectrometer (CTIS) [Descour and Dereniak, Appl. Opt. 34, 4817-4826 (1995)] has proved effective in producing hyperspectral images in a single spectral region. Coupling the CTIS with a dual-band detector can produce two hyperspectral data cubes simultaneously. We describe the design of two-dimensional, surface-relief, computer-generated hologram dispersers that permit image information in these two bands simultaneously.