Depth-resolved image mapping spectrometer (IMS) with structured illumination

Kilohertz frame rate snapshot hyperspectral imaging of metal reactive materials

We demonstrate a kilohertz frame rate snapshot hyperspectral imaging system suitable for high-speed imaging, which we name snapshot hyperspectral imager for emission and reactions (SHEAR). This system splits the sensor of a single high-speed camera to simultaneously capture a conventional image and a spectrally sheared response of the scene under study. Given the small, point-source-like nature of burning metal micro-particles, the spectral response of the species is captured without the need for a slit, as is needed in conventional imaging spectrometers. We pair robust image registration techniques with sparse reconstruction algorithms to computationally disentangle overlapping spectra associated with many burning particles over the course of a combustion experiment. As a proof-of-concept experiment, representative physical vapor deposited Al:Zr composite particles are ignited, and their burn evolution is recorded at a frame rate of 2 kHz using this method. We demonstrate operation over two distinct wavelength ranges spanning hundreds of nanometers in wavelength and with sub-nanometer resolution. We are able to track hundreds of individual Al:Zr particles in a single high-speed video, providing ample statistics of burn time, temperature, and AlO emission timing in a high-throughput method. The demonstrated technology is high-throughput, flexible in wavelength, inexpensive, and relatively easy to implement, and provides a much needed tool for in situ composite metal fuel diagnostics.

Punching holes in light: recent progress in single-shot coded-aperture optical imaging

Single-shot coded-aperture optical imaging physically captures a code-aperture-modulated optical signal in one exposure and then recovers the scene via computational image reconstruction. Recent years have witnessed dazzling advances in various modalities in this hybrid imaging scheme in concomitant technical improvement and widespread applications in physical, chemical and biological sciences. This review comprehensively surveys state-of-the-art single-shot coded-aperture optical imaging. Based on the detected photon tags, this field is divided into six categories: planar imaging, depth imaging, light-field imaging, temporal imaging, spectral imaging, and polarization imaging. In each category, we start with a general description of the available techniques and design principles, then provide two representative examples of active-encoding and passive-encoding approaches, with a particular emphasis on their methodology and applications as well as their advantages and challenges. Finally, we envision prospects for further technical advancement in this field.

Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review [Invited]

Hyperspectral imaging (HSI) and multispectral imaging (MSI) technologies have the potential to transform the fields of digital and computational pathology. Traditional digitized histopathological slides are imaged with RGB imaging. Utilizing HSI/MSI, spectral information across wavelengths within and beyond the visual range can complement spatial information for the creation of computer-aided diagnostic tools for both stained and unstained histological specimens. In this systematic review, we summarize the methods and uses of HSI/MSI for staining and color correction, immunohistochemistry, autofluorescence, and histopathological diagnostic research. Studies include hematology, breast cancer, head and neck cancer, skin cancer, and diseases of central nervous, gastrointestinal, and genitourinary systems. The use of HSI/MSI suggest an improvement in the detection of diseases and clinical practice compared with traditional RGB analysis, and brings new opportunities in histological analysis of samples, such as digital staining or alleviating the inter-laboratory variability of digitized samples. Nevertheless, the number of studies in this field is currently limited, and more research is needed to confirm the advantages of this technology compared to conventional imagery.

Accurate ray tracing model of an imaging system based on image mapper

The image mapper plays a key role in the imaging process of the image mapping spectrometer (IMS), which is a snapshot imaging spectrometer with superiority in light throughput, temporal resolution, and compactness. In this paper, an accurate ray tracing model of the imaging units of the IMS, especially the image mapper, is presented in the form of vector operation. Based on the proposed model, the behavior of light reflection on the image mapper is analyzed thoroughly, including the precise position of the reflection point and interaction between adjacent facets. Rigorous spatial correspondence between object points and pixels on the detector is determined by tracing the chief ray of an arbitrary point in the field. The shadowing effect, which is shadowing between adjacent facets and blocking caused by the facets' side walls, is analyzed based on our model. The experimental results verify the fidelity of the model and the existence of the shadowing effect. The research is meaningful for comprehending the imaging mechanism of the IMS and facilitates the design and analysis process in the future.

Compressive hyperspectral imaging with non-zero mean noise

Compressive hyperspectral imaging (CHI) has attracted widespread attention due to its advantage of snapshot by encoding the 3D spectral image into a 2D measurement. The bottleneck of CHI for the real application now lies in the limited reconstruction quality, for which one of the fundamental reason is the inaccurate modeling of the measurement noise. Our key observation is that in practical scenarios, the measurement is inevitably contaminated with a positive offset (i.e., noise mean) due to the unideal imaging conditions (e.g. stray light and dark current), resulting in serious degradations of the reconstruction quality. In this paper, we propose to model the real noise with non-zero mean that generalizes the traditional zero mean noise to faithfully characterizing the optical imaging principle, and then introduce a novel reconstruction method with a goal to boost the reconstruction quality. During the reconstruction, the noise mean is estimated gradually to its convergence by measuring the deviation of the intermediate reconstruction result from the system measurement. We demonstrate the superior performance of our method by experiments on synthetic data and real capture data with a hardware prototype.

Structured illumination imaging without grating rotation based on mirror operation on 1D Fourier spectrum

Structured illumination microscopy (SIM) is a rapidly developing a super-resolution optical microscopy technique. With SIM, the grating is needed in order to rotate several angles for illuminating the sample in different directions. Multiple rotations reduce the imaging speed and grating rotation angle errors damage the image recovery quality. We introduce mirror transformation on one-dimension (1D) Fourier spectrum to SIM for resolving the problems of low imaging speed and severe impact on image reconstruction quality by grating rotation angle errors. When mirror operation and SIM are combined, the grating is placed at an orientation for obtaining three shadow images. The three shadow images are acquired by CCD at three different phase shift for a direction of grating. Thus, the SIM imaging speed is faster and the effect on image reconstruction quality by grating rotation angle errors is greatly reduced.

Complete plenoptic imaging using a single detector

Multi-dimensional imaging is a powerful technique for many applications, such as biological analysis, remote sensing, and object recognition. Most existing multi-dimensional imaging systems rely on scanning or camera array, which make the system bulky and unstable. To some extent, these problems can be mitigated by employing compressed sensing algorithms. However, they are computationally expensive and highly rely on the ill-posed assumption that the information is sparse in a given domain. Here, we propose a snapshot spectral-volumetric imaging (SSVI) system by introducing the paradigm of light-field imaging into Fourier transform imaging spectroscopy. We demonstrate that SSVI can reconstruct a complete plenoptic function, P(x,y,z,θ,φ,λ,t), of the incoming light rays using a single detector. Compared with other multidimensional imagers, SSVI features prominent advantages in compactness, robustness, and low cost.

Optimized Multi-Spectral Filter Array Based Imaging of Natural Scenes

Multi-spectral imaging using a camera with more than three channels is an efficient method to acquire and reconstruct spectral data and is used extensively in tasks like object recognition, relighted rendering, and color constancy. Recently developed methods are used to only guide content-dependent filter selection where the set of spectral reflectances to be recovered are known a priori. We present the first content-independent spectral imaging pipeline that allows optimal selection of multiple channels. We also present algorithms for optimal placement of the channels in the color filter array yielding an efficient demosaicing order resulting in accurate spectral recovery of natural reflectance functions. These reflectance functions have the property that their power spectrum statistically exhibits a power-law behavior. Using this property, we propose power-law based error descriptors that are minimized to optimize the imaging pipeline. We extensively verify our models and optimizations using large sets of commercially available wide-band filters to demonstrate the greater accuracy and efficiency of our multi-spectral imaging pipeline over existing methods.

Snapshot Hyperspectral Light-Sheet Imaging of Signal Transduction in Live Pancreatic Islets

The observation of ionic signaling dynamics in intact pancreatic islets has contributed greatly to our understanding of both α- and β-cell function. Insulin secretion from β-cells depends on the firing of action potentials and consequent rises of intracellular calcium activity ([Ca(2+)]i). Zinc (Zn(2+)) is cosecreted with insulin, and has been postulated to play a role in cell-to-cell cross talk within an islet, in particular inhibiting glucagon secretion from α-cells. Thus, measuring [Ca(2+)]i and Zn(2+) dynamics from both α- and β-cells will elucidate mechanisms underlying islet hormone secretion. [Ca(2+)]i and intracellular Zn(2+) can be measured using fluorescent biosensors, but the most efficient sensors have overlapping spectra that complicate their discrimination. Hyperspectral imaging can be used to distinguish signals from multiple fluorophores, but available hyperspectral implementations are either too slow to measure the dynamics of ionic signals or not suitable for thick samples. We have developed a five-dimensional (x,y,z,t,λ) imaging system that leverages a snapshot hyperspectral imaging method, image mapping spectrometry, and light-sheet microscopy. This system provides subsecond temporal resolution from deep within multicellular structures. Using a single excitation wavelength (488 nm) we acquired images from triply labeled samples with two biosensors and a genetically expressing fluorescent protein (spectrally overlapping with one of the biosensors) with high temporal resolution. Measurements of [Ca(2+)]i and Zn(2+) within both α- and β-cells as a function of glucose concentration show heterogeneous uptake of Zn(2+) into α-cells that correlates to the known heterogeneities in [Ca(2+)]i. These differences in intracellular Zn(2+) among α-cells may contribute to the inhibition in glucagon secretion observed at elevated glucose levels.

A review of snapshot multidimensional optical imaging: measuring photon tags in parallel

Multidimensional optical imaging has seen remarkable growth in the past decade. Rather than measuring only the two-dimensional spatial distribution of light, as in conventional photography, multidimensional optical imaging captures light in up to nine dimensions, providing unprecedented information about incident photons' spatial coordinates, emittance angles, wavelength, time, and polarization. Multidimensional optical imaging can be accomplished either by scanning or parallel acquisition. Compared with scanning-based imagers, parallel acquisition-also dubbed snapshot imaging-has a prominent advantage in maximizing optical throughput, particularly when measuring a datacube of high dimensions. Here, we first categorize snapshot multidimensional imagers based on their acquisition and image reconstruction strategies, then highlight the snapshot advantage in the context of optical throughput, and finally we discuss their state-of-the-art implementations and applications.

Optical hyperspectral imaging in microscopy and spectroscopy - a review of data acquisition

Rather than simply acting as a photographic camera capturing two-dimensional (x, y) intensity images or a spectrometer acquiring spectra (λ), a hyperspectral imager measures entire three-dimensional (x, y, λ) datacubes for multivariate analysis, providing structural, molecular, and functional information about biological cells or tissue with unprecedented detail. Such data also gives clinical insights for disease diagnosis and treatment. We summarize the principles underpinning this technology, highlight its practical implementation, and discuss its recent applications at microscopic to macroscopic scales. Datacube acquisition strategies in hyperspectral imaging x, y, spatial coordinates; λ, wavelength.

Optical hyperspectral imaging in microscopy and spectroscopy - a review of data acquisition

Rather than simply acting as a photographic camera capturing two-dimensional (x, y) intensity images or a spectrometer acquiring spectra (λ), a hyperspectral imager measures entire three-dimensional (x, y, λ) datacubes for multivariate analysis, providing structural, molecular, and functional information about biological cells or tissue with unprecedented detail. Such data also gives clinical insights for disease diagnosis and treatment. We summarize the principles underpinning this technology, highlight its practical implementation, and discuss its recent applications at microscopic to macroscopic scales. Datacube acquisition strategies in hyperspectral imaging x, y, spatial coordinates; λ, wavelength.

Depth resolved hyperspectral imaging spectrometer based on structured light illumination and Fourier transform interferometry

A depth resolved hyperspectral imaging spectrometer can provide depth resolved imaging both in the spatial and the spectral domain. Images acquired through a standard imaging Fourier transform spectrometer do not have the depth-resolution. By post processing the spectral cubes (x, y, λ) obtained through a Sagnac interferometer under uniform illumination and structured illumination, spectrally resolved images with depth resolution can be recovered using structured light illumination algorithms such as the HiLo method. The proposed scheme is validated with in vitro specimens including fluorescent solution and fluorescent beads with known spectra. The system is further demonstrated in quantifying spectra from 3D resolved features in biological specimens. The system has demonstrated depth resolution of 1.8 μm and spectral resolution of 7 nm respectively.

Optical sectioning by wide-field photobleaching imprinting microscopy

We present a generic wide-field optical sectioning scheme, photobleaching imprinting microscopy (PIM), for depth-resolved cross-sectional fluorescence imaging. Wide-field PIM works by extracting a nonlinear component that depends on the excitation fluence as a result of photobleaching-induced fluorescence decay. Since no specific fluorescent dyes or illumination modules are required, wide-field PIM is easy to implement on a standard microscope. Moreover, wide-field PIM is superior to deconvolution microscopy in removing background fluorescence, yielding a six-fold improvement in image contrast.

Snapshot 3D optical coherence tomography system using image mapping spectrometry

A snapshot 3-Dimensional Optical Coherence Tomography system was developed using Image Mapping Spectrometry. This system can give depth information (Z) at different spatial positions (XY) within one camera integration time to potentially reduce motion artifact and enhance throughput. The current (x,y,λ) datacube of (85×356×117) provides a 3D visualization of sample with 400 μm depth and 13.4 μm in transverse resolution. Axial resolution of 16.0 μm can also be achieved in this proof-of-concept system. We present an analysis of the theoretical constraints which will guide development of future systems with increased imaging depth and improved axial and lateral resolutions.

Rapid fabrication of miniature lens arrays by four-axis single point diamond machining

A novel method for fabricating lens arrays and other non-rotationally symmetric free-form optics is presented. This is a diamond machining technique using 4 controlled axes of motion - X, Y, Z, and C. As in 3-axis diamond micro-milling, a diamond ball endmill is mounted to the work spindle of a 4-axis ultra-precision computer numerical control (CNC) machine. Unlike 3-axis micro-milling, the C-axis is used to hold the cutting edge of the tool in contact with the lens surface for the entire cut. This allows the feed rates to be doubled compared to the current state of the art of micro-milling while producing an optically smooth surface with very low surface form error and exceptionally low radius error.

Three dimensional HiLo-based structured illumination for a digital scanned laser sheet microscopy (DSLM) in thick tissue imaging

Laser sheet based microscopy has become widely accepted as an effective active illumination method for real time three-dimensional (3D) imaging of biological tissue samples. The light sheet geometry, where the camera is oriented perpendicular to the sheet itself, provides an effective method of eliminating some of the scattered light and minimizing the sample exposure to radiation. However, residual background noise still remains, limiting the contrast and visibility of potentially interesting features in the samples. In this article, we investigate additional structuring of the illumination for improved background rejection, and propose a new technique, "3D HiLo" where we combine two HiLo images processed from orthogonal directions to improve the condition of the 3D reconstruction. We present a comparative study of conventional structured illumination based demodulation methods, namely 3Phase and HiLo with a newly implemented 3D HiLo approach and demonstrate that the latter yields superior signal-to-background ratio in both lateral and axial dimensions, while simultaneously suppressing image processing artifacts.

Image mapping spectrometry: calibration and characterization

Image mapping spectrometry (IMS) is a hyperspectral imaging technique that simultaneously captures spatial and spectral information about an object in real-time. We present a new calibration procedure for the IMS as well as the first detailed evaluation of system performance. We correlate optical components and device calibration to performance metrics such as light throughput, scattered light, distortion, spectral image coregistration, and spatial/spectral resolution. Spectral sensitivity and motion artifacts are also evaluated with a dynamic biological experiment. The presented methodology of evaluation is useful in assessment of a variety of hyperspectral and multi-spectral modalities. Results are important to any potential users/developers of an IMS instrument and to anyone who may wish to compare the IMS to other imaging spectrometers.

Snapshot spectrally encoded fluorescence imaging through a fiber bundle

Fiber optic endomicroscopy is a valuable tool for clinical diagnostics and animal studies because it can capture images of tissue in vivo with subcellular resolution. Current configurations for endomicroscopes have either limited spatial resolution or require a scanning mechanism at the distal end of the fiber, which can slow imaging speed and increase the probe size. We present a novel configuration that provides high contrast 350 × 350 pixel images at 7.2 frames per second, without the need for mechanical scanning at the proximal or distal end of the fiber. The proof-of-concept benchtop system is tested in fluorescence mode and can resolve 1.5 µm features of a high resolution 1951 USAF target.

Snapshot advantage: a review of the light collection improvement for parallel high-dimensional measurement systems

The snapshot advantage is a large increase in light collection efficiency available to high-dimensional measurement systems that avoid filtering and scanning. After discussing this advantage in the context of imaging spectrometry, where the greatest effort towards developing snapshot systems has been made, we describe the types of measurements where it is applicable. We then generalize it to the larger context of high-dimensional measurements, where the advantage increases geometrically with measurement dimensionality.

Quantitative sectioning and noise analysis for structured illumination microscopy

Structured illumination (SI) has long been regarded as a nonquantitative technique for obtaining sectioned microscopic images. Its lack of quantitative results has restricted the use of SI sectioning to qualitative imaging experiments, and has also limited researchers' ability to compare SI against competing sectioning methods such as confocal microscopy. We show how to modify the standard SI sectioning algorithm to make the technique quantitative, and provide formulas for calculating the noise in the sectioned images. The results indicate that, for an illumination source providing the same spatially-integrated photon flux at the object plane, and for the same effective slice thicknesses, SI sectioning can provide higher SNR images than confocal microscopy for an equivalent setup when the modulation contrast exceeds about 0.09.