Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy

Label-Free Optical Technologies to Enhance Noninvasive Endoscopic Imaging of Early-Stage Cancers

White light endoscopic imaging allows for the examination of internal human organs and is essential in the detection and treatment of early-stage cancers. To facilitate diagnosis of precancerous changes and early-stage cancers, label-free optical technologies that provide enhanced malignancy-specific contrast and depth information have been extensively researched. The rapid development of technology in the past two decades has enabled integration of these optical technologies into clinical endoscopy. In recent years, the significant advantages of using these adjunct optical devices have been shown, suggesting readiness for clinical translation. In this review, we provide an overview of the working principles and miniaturization considerations and summarize the clinical and preclinical demonstrations of several such techniques for early-stage cancer detection. We also offer an outlook for the integration of multiple technologies and the use of computer-aided diagnosis in clinical endoscopy.

Optical Design of a Hyperspectral Remote-Sensing System Based on an Image-Slicer Integral Field Unit in the Short-Wave Infrared Band

Grating-type spectral imaging systems are frequently employed in scenes for high-resolution remote-sensing observations of the Earth. However, the entrance of the grating-type spectral imaging system is a slit or a pinhole. This structure relies on the push broom method, which presents a challenge in capturing spectral information of transiently changing targets. To address this issue, the IFU is used to slice the focal plane of the telescope system, thereby expanding the instantaneous field of view (IFOV) of the grating-type spectral imaging system. The aberrations introduced by the expansion of the single-slice field of view (FOV) of the IFU are corrected, and the conversion of the IFU's FOV from arcseconds to degrees is achieved. The design of a spectral imaging system based on an image-slicer IFU for remote sensing is finally completed. The system has a wavelength range of 1400 nm to 2000 nm, and a spectral resolution of better than 3 nm. Compared with the traditional grating-type spectral imaging system, its IFOV is expanded by a factor of four. And it allows for the capture of complete spectral information of transiently changing targets through a single exposure. The simulation results demonstrate that the system has good performance at each sub-slit, thereby validating the effectiveness and advantages of the proposed system for dynamic target capture in remote sensing.

Hyperspectral imaging for non-invasive blood oxygen saturation assessment

Hyperspectral Imaging (HSI) seamlessly integrates imaging and spectroscopy, capturing both spatial and spectral data concurrently. With widespread applications in medical diagnostics, HSI serves as a noninvasive tool for gaining insights into tissue characteristics. The distinctive spectral profiles of biological tissues set HSI apart from traditional microscopy in enabling in vivo tissue analysis. Despite its potential, existing HSI techniques face challenges such as alignment issues, low light throughput, and tissue heating due to intense illumination. This study introduces an innovative HSI system featuring active sequential bandpass illumination seamlessly integrated into conventional optical instruments. The primary focus is on analyzing oxyhemoglobin and deoxyhemoglobin saturation in animal tissue samples using multivariate linear regression. This approach holds promise for enhancing noninvasive medical diagnostics. A key feature of the system, active bandpass illumination, effectively prevents tissue overheating, thereby bolstering its suitability for medical applications.

High-spatial density snapshot imaging spectrometer enabled by 2-photon fabricated custom fiber bundles

We report on a proof-of-concept snapshot imaging spectrometer developed using an array of optical fibers fabricated with 2-photon polymerization (2PP). The dense input array maps to an output array with engineered void spaces for spectral information. Previously, the development and fabrication of custom fiber arrays for imaging spectrometers have been a complex, time-consuming, and costly process, requiring a semi-manual assembly of commercial components. This work applies an automatic development process based on 2PP additive manufacturing with the Nanoscribe GmbH Quantum X system. The technique allows printing of arbitrary optical quality structures with submicron resolution with less than 5 nm roughness, enabling small core fibers/integrated arrays. Specifically, we developed an array prototype of 40 × 80 with 6-micron pitch at the input and 80-micron pitch at the output. The air-clad fibers had a core diameter of 5 µm. Fabricated optical fiber arrays were incorporated into a prism-based imaging spectrometer system with 48 spectral channels to demonstrate multi-spectral imaging. Imaging of a USAF target and color printed letter C as well as spectral comparisons to a commercial spectrometer were used to validate the performance of the system. These results clearly demonstrate the functionality and potential applications of the 3D-printed fiber-based snapshot imaging spectrometer.

Coded aperture snapshot hyperspectral light field tomography

Multidimensional imaging has emerged as a powerful technology capable of simultaneously acquiring spatial, spectral, and depth information about a scene. However, existing approaches often rely on mechanical scanning or multi-modal sensing configurations, leading to prolonged acquisition times and increased system complexity. Coded aperture snapshot spectral imaging (CASSI) has introduced compressed sensing to recover three-dimensional (3D) spatial-spectral datacubes from single snapshot two-dimensional (2D) measurements. Despite its advantages, the reconstruction problem remains severely underdetermined due to the high compression ratio, resulting in limited spatial and spectral reconstruction quality. To overcome this challenge, we developed a novel two-stage cascaded compressed sensing scheme called coded aperture snapshot hyperspectral light field tomography (CASH-LIFT). By appropriately distributing the computation load to each stage, this method utilizes the compressibility of natural scenes in multiple domains, reducing the ill-posed nature of datacube recovery and achieving enhanced spatial resolution, suppressed aliasing artifacts, and improved spectral fidelity. Additionally, leveraging the snapshot 3D imaging capability of LIFT, our approach efficiently records a five-dimensional (5D) plenoptic function in a single snapshot.

Fabrication of a multifaceted mapping mirror using two-photon polymerization for a snapshot image mapping spectrometer

A design and fabrication technique for making high-precision and large-format multifaceted mapping mirrors is presented. The method is based on two-photon polymerization, which allows more flexibility in the mapping mirror design. The mirror fabricated in this paper consists of 36 2D tilted square pixels, instead of the continuous facet design used in diamond cutting. The paper presents a detailed discussion of the fabrication parameters and optimization process, with particular emphasis on the optimization of stitching defects by compensating for the overall tilt angle and reducing the printing field of view. The fabricated mirrors were coated with a thin layer of aluminum (93 nm) using sputter coating to enhance the reflection rate over the target wave range. The mapping mirror was characterized using a white light interferometer and a scanning electron microscope, which demonstrates its optical quality surface (with a surface roughness of 12 nm) and high-precision tilt angles (with an average of 2.03% deviation). Finally, the incorporation of one of the 3D printed mapping mirrors into an image mapping spectrometer prototype allowed for the acquisition of high-quality images of the USAF resolution target and bovine pulmonary artery endothelial cells stained with three fluorescent dyes, demonstrating the potential of this technology for practical applications.

Spatial-spectral resolution tunable snapshot imaging spectrometer: analytical design and implementation

A snapshot imaging spectrometer is a powerful tool for dynamic target tracking and real-time recognition compared with a scanning imaging spectrometer. However, all the current snapshot spectral imaging techniques suffer from a major trade-off between the spatial and spectral resolutions. In this paper, an integral field snapshot imaging spectrometer (TIF-SIS) with a continuously tunable spatial-spectral resolution and light throughput is proposed and demonstrated. The proposed TIF-SIS is formed by a fore optics, a lenslet array, and a collimated dispersive subsystem. Theoretical analyses indicate that the spatial-spectral resolution and light throughput of the system can be continuously tuned through adjusting the F number of the fore optics, the rotation angle of the lenslet array, or the focal length of the collimating lens. Analytical relationships between the spatial and spectral resolutions and the first-order parameters of the system with different geometric arrangements of the lenslet unit are obtained. An experimental TIF-SIS consisting of a self-fabricated lenslet array with a pixelated scale of 100×100 and a fill factor of 0.716 is built. The experimental results show that the spectral resolution of the system can be steadily improved from 4.17 to 0.82 nm with a data cube (N x×N y×N λ) continuously tuned from 35×35×36 to 40×40×183 in the visible wavelength range from 500 to 650 nm, which is consistent with the theoretical prediction. The proposed method for real-time tuning of the spatial-spectral resolution and light throughput opens new possibilities for broader applications, especially for recognition of things with weak spectral signature and biomedical investigations where a high light throughput and tunable resolution are needed.

Bio-inspired Compact, High-resolution Snapshot Hyperspectral Imaging System with 3D Printed Glass Lightguide Array

To address the major challenges to obtain high spatial resolution in snapshot hyperspectral imaging, 3D printed glass lightguide array has been developed to sample the intermediate image in high spatial resolution and redistribute the pixels in the output end to achieve high spectral resolution. Curved 3D printed lightguide array can significantly simplify the snapshot hyperspectral imaging system, achieve better imaging performance, and reduce the system complexity and cost. We have developed two-photon polymerization process to print glass lightguide array, and demonstrated the system performance with biological samples. This new snapshot technology will catalyze new hyperspectral imaging system development and open doors for new applications from UV to IR.

Design and Validation of a Custom-Made Hyperspectral Microscope Imaging System for Biomedical Applications

Hyperspectral microscope imaging (HMI) is an emerging modality that integrates spatial information collected by standard laboratory microscopy and the spectral-based contrast obtained by hyperspectral imaging and may be instrumental in establishing novel quantitative diagnostic methodologies, particularly in histopathology. Further expansion of HMI capabilities hinges upon the modularity and versatility of systems and their proper standardization. In this report, we describe the design, calibration, characterization, and validation of the custom-made laboratory HMI system based on a Zeiss Axiotron fully motorized microscope and a custom-developed Czerny-Turner-type monochromator. For these important steps, we rely on a previously designed calibration protocol. Validation of the system demonstrates a performance comparable to classic spectrometry laboratory systems. We further demonstrate validation against a laboratory hyperspectral imaging system for macroscopic samples, enabling future comparison of spectral imaging results across length scales. An example of the utility of our custom-made HMI system on a standard hematoxylin and eosin-stained histology slide is also shown.

Deep Hybrid 2-D-3-D CNN Based on Dual Second-Order Attention With Camera Spectral Sensitivity Prior for Spectral Super-Resolution

A largely ignored fact in spectral super-resolution (SSR) is that the subsistent mapping methods neglect the auxiliary prior of camera spectral sensitivity (CSS) and only pay attention to wider or deeper network framework design while ignoring to excavate the spatial and spectral dependencies among intermediate layers, hence constraining representational capability of convolutional neural networks (CNNs). To conquer these drawbacks, we propose a novel deep hybrid 2-D-3-D CNN based on dual second-order attention with CSS prior (HSACS), which can excavate sufficient spatial-spectral context information. Specifically, dual second-order attention embedded in the residual block for more powerful spatial-spectral feature representation and relation learning is composed of a brand new trainable 2-D second-order channel attention (SCA) or 3-D second-order band attention (SBA) and a structure tensor attention (STA). Concretely, the band and channel attention modules are developed to adaptively recalibrate the band-wise and interchannel features via employing second-order band or channel feature statistics for more discriminative representations. Besides, the STA is promoted to rebuild the significant high-frequency spatial details for enough spatial feature extraction. Moreover, the CSS is first employed as a superior prior to avoid its effect of SSR quality, on the strength of which the resolved RGB can be calculated naturally through the super-reconstructed hyperspectral image (HSI); then, the final loss consists of the discrepancies of RGB and the HSI as a finer constraint. Experimental results demonstrate the superiority and progressiveness of the presented approach in terms of quantitative metrics and visual effect over SOTA SSR methods.

Tunable image projection spectrometry

We present tunable image projection spectrometry (TIPS), a Fourier-domain line-scan spectral imager with a tunable compression ratio. Compared to state-of-the-art spatial-domain pushbroom hyperspectral cameras, TIPS requires much fewer measurements and provides a higher light throughput. Using a rotating Dove prism and a cylindrical field lens, TIPS scans an input scene in the Fourier domain and captures a subset of multi-angled one-dimensional (1D) en face projections of the input scene, allowing a tailored data compression ratio for a given scene. We demonstrate the spectral imaging capability of TIPS with a hematoxylin and eosin (H&E) stained pathology slide. Moreover, we showed the spectral information obtained can be further converted to depths when combining TIPS with a low-coherence full-field spectral-domain interferometer.

A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging

Hyperspectral imaging enables many versatile applications for its competence in capturing abundant spatial and spectral information, which is crucial for identifying substances. However, the devices for acquiring hyperspectral images are typically expensive and very complicated, hindering the promotion of their application in consumer electronics, such as daily food inspection and point-of-care medical screening, etc. Recently, many computational spectral imaging methods have been proposed by directly reconstructing the hyperspectral information from widely available RGB images. These reconstruction methods can exclude the usage of burdensome spectral camera hardware while keeping a high spectral resolution and imaging performance. We present a thorough investigation of more than 25 state-of-the-art spectral reconstruction methods which are categorized as prior-based and data-driven methods. Simulations on open-source datasets show that prior-based methods are more suitable for rare data situations, while data-driven methods can unleash the full potential of deep learning in big data cases. We have identified current challenges faced by those methods (e.g., loss function, spectral accuracy, data generalization) and summarized a few trends for future work. With the rapid expansion in datasets and the advent of more advanced neural networks, learnable methods with fine feature representation abilities are very promising. This comprehensive review can serve as a fruitful reference source for peer researchers, thus paving the way for the development of computational hyperspectral imaging.

Coded Hyperspectral Image Reconstruction Using Deep External and Internal Learning

To solve the low spatial and/or temporal resolution problem which the conventional hyperspectral cameras often suffer from, coded hyperspectral imaging systems have attracted more attention recently. Recovering a hyperspectral image (HSI) from its corresponding coded image is an ill-posed inverse problem, and learning accurate prior of HSI is essential to solve this inverse problem. In this paper, we present an effective convolutional neural network (CNN) based method for coded HSI reconstruction, which learns the deep prior from the external dataset as well as the internal information of input coded image with spatial-spectral constraint. Specifically, we first develop a CNN-based channel attention reconstruction network to effectively exploit the spatial-spectral correlation of the HSI. Then, the reconstruction network is learned by leveraging an arbitrary external hyperspectral dataset to exploit the general spatial-spectral correlation under adversarial loss. Finally, we customize the network by internal learning with spatial-spectral constraint and total variation regularization for each coded image, which can make use of the internal imaging model to learn specific prior for current desirable image and effectively avoids overfitting. Experimental results using both synthetic data and real images show that our method outperforms the state-of-the-art methods on several popular coded hyperspectral imaging systems under both comprehensive quantitative metrics and perceptive quality.

Hybrid non-sequential modeling of an image mapping spectrometer

An image mapping spectrometer (IMS) is a kind of snapshot imaging spectrometer characterized by containing several array components including the image mapper, prism array, and reimaging lens array. We propose a hybrid non-sequential modeling method of IMS and present the complete optical model of the system built in Zemax. This method utilizes the spatial periodicity of the array components and requires only a small number of input parameters. Moreover, we design a collimating lens of a large relative aperture, sufficient working distance, and low aberration to meet the requirements of an IMS with good optical performance and compact volume. The designed lens is quantitatively evaluated in the entire IMS model, and the results demonstrate that the lens has excellent optical performance. The evaluation on the collimating lens also demonstrates the capability of the proposed modeling method in the design and optimization of systems such as the IMS that contain multiple array components. The designed collimating lens is manufactured and assembled in the experimental setup of the IMS. The proposed modeling method is verified by experimental results.

A Snapshot Imaging Spectrometer Based on Uniformly Distributed-Slit Array (UDA)

Herein, we propose a system for a snapshot video hyperspectral imaging method based on a uniformly distributed-slit array (UDA) coding plate that not only effectively improves the scanning speed of spectrometers but also achieves a high spectral fidelity of snapshot videos. A mathematical model and optical link simulation of the new system are established. The analysis results show that the proposed method can more efficiently collect information and restore the spectral data cube, and the spectral smile of the system is less than 4.86 μm. The results of the spectral performance and external imaging tests of the system show that the system has the ability to collect spatial spectrum video information with a frame rate of 10 Hz and identify dynamic targets, laying a foundation for the design of a system with a higher frame rate and resolution.

Joint Camera Spectral Response Selection and Hyperspectral Image Recovery

Hyperspectral image (HSI) recovery from a single RGB image has attracted much attention, whose performance has recently been shown to be sensitive to the camera spectral response (CSR). In this paper, we present an efficient convolutional neural network (CNN) based method, which can jointly select the optimal CSR from a candidate dataset and learn a mapping to recover HSI from a single RGB image captured with this algorithmically selected camera under multi-chip or single-chip setups. Given a specific CSR, we first present a HSI recovery network, which accounts for the underlying characteristics of the HSI, including spectral nonlinear mapping and spatial similarity. Later, we append a CSR selection layer onto the recovery network, and the optimal CSR under both multi-chip and single-chip setups can thus be automatically determined from the network weights under the nonnegative sparse constraint. Experimental results on three hyperspectral datasets and two camera spectral response datasets demonstrate that our HSI recovery network outperforms state-of-the-art methods in terms of both quantitative metrics and perceptive quality, and the selection layer always returns a CSR consistent to the best one determined by exhaustive search. Finally, we show that our method can also perform well in the real capture system, and collect a hyperspectral flower dataset to evaluate the effect from HSI recovery on classification problem.

Snapshot hyperspectral light field tomography

We present snapshot hyperspectral light field tomography (Hyper-LIFT), a highly efficient method in recording a 5D (x, y, spatial coordinates; θ, φ, angular coordinates; λ, wavelength) plenoptic function. Using a Dove prism array and a cylindrical lens array, we simultaneously acquire multi-angled 1D en face projections of the object like those in standard sparse-view computed tomography. We further disperse those projections and measure the spectra in parallel using a 2D image sensor. Within a single snapshot, the resultant system can capture a 5D data cube with 270 × 270 × 4 × 4 × 360 voxels. We demonstrated the performance of Hyper-LIFT in imaging spectral volumetric scenes.

Multimodal hyperspectral fluorescence and spatial frequency domain imaging for tissue health diagnostics of the oral cavity

A multimodal, hyperspectral imaging system was built for diagnostics of oral tissues. The system, termed Hyperspectral-Fluorescence-Spatial Frequency Domain Imaging (Hy-F-SFDI), combines the principles of spatial frequency domain imaging, quantitative light fluorescence, and CIELAB color measurement. Hy-F-SFDI employs a compact LED projector, excitation LED, and a 16 channel hyperspectral camera mounted on a custom platform for tissue imaging. A two layer Monte Carlo approach was used to generate a reference table for quick tissue analysis. To demonstrate the clinical capabilities of Hy-F-SFDI, we used the system to quantify gingival tissue hemoglobin volume fraction, detect caries, bacterial activity, and measure tooth color of a volunteer at different time points. Hy-F-SFDI was able to measure quantitative changes in tissue parameters.

Radiometric and design model for the tunable light-guide image processing snapshot spectrometer (TuLIPSS)

The tunable light-guide image processing snapshot spectrometer (TuLIPSS) is a novel remote sensing instrument that can capture a spectral image cube in a single snapshot. The optical modelling application for the absolute signal intensity on a single pixel of the sensor in TuLIPSS has been developed through a numerical simulation of the integral performance of each optical element in the TuLIPSS system. The absolute spectral intensity of TuLIPSS can be determined either from the absolute irradiance of the observed surface or from the tabulated spectral reflectance of various land covers and by the application of a global irradiance approach. The model is validated through direct comparison of the simulated results with observations. Based on tabulated spectral reflectance, the deviation between the simulated results and the measured observations is less than 5% of the spectral light flux across most of the detection bandwidth for a Lambertian-like surface such as concrete. Additionally, the deviation between the simulated results and the measured observations using global irradiance information is less than 10% of the spectral light flux across most of the detection bandwidth for all surfaces tested. This optical modelling application of TuLIPSS can be used to assist the optimal design of the instrument and explore potential applications. The influence of the optical components on the light throughput is discussed with the optimal design being a compromise among the light throughput, spectral resolution, and cube size required by the specific application under consideration. The TuLIPSS modelling predicts that, for the current optimal low-cost configuration, the signal to noise ratio can exceed 10 at 10 ms exposure time, even for land covers with weak reflectance such as asphalt and water. Overall, this paper describes the process by which the optimal design is achieved for particular applications and directly connects the parameters of the optical components to the TuLIPSS performance.

Unsupervised learning for hyperspectral recovery based on a single RGB image

Hyperspectral imagery often suffers from the degradation of spatial, spectral, or temporal resolution due to the limitations of hyperspectral imaging devices. To address this problem, hyperspectral recovery from a single red-green-blue (RGB) image has recently achieved significant progress via deep learning. However, current deep learning-based methods are all learned in a supervised way under the availability of RGB and correspondingly hyperspectral images, which is unrealistic for practical applications. Hence, we propose to recover hyperspectral images from a single RGB image in an unsupervised way. Moreover, based on the statistical property of hyperspectral images, a customized loss function is proposed to boost the performance. Extensive experiments on the BGU iCVL Hyperspectral Image Dataset demonstrate the effectiveness of the proposed method.

Combination of Structured Illumination Microscopy with Hyperspectral Imaging for Cell Analysis

Existing structured illumination microscopy (SIM) allows super-resolution live-cell imaging in few color channels that provide merely morphological information but cannot acquire the sample spectrum that is strongly relevant to the underlying physicochemical property. We develop hyperspectral SIM which enables high-speed spectral super-resolution imaging in SIM for the first time. Through optically mapping the three-dimensional (x, y, and λ) datacube of the sample to the detector plane, hyperspectral SIM allows snapshot spectral imaging of the SIM raw image, detecting the sample spectrum while retaining the high-speed and super-resolution characteristics of SIM. We demonstrate hyperspectral SIM imaging and reconstruct a datacube containing 31 super-resolution images of different wavelengths from only 9 exposures, achieving a 15 nm spectral resolution. We show time-lapse hyperspectral SIM imaging that achieves an imaging speed of 2.7 s per datacube-31-fold faster than the existing wavelength scanning strategy. To demonstrate the great prospects for further combining hyperspectral SIM with various spectral analysis methods, we also perform spectral unmixing of the hyperspectral SIM result while imaging the spectrally overlapped sample.

Snapshot Imaging Spectrometer Based on Pixel-Level Filter Array (PFA)

Snapshot spectral imaging technology plays an important role in many fields. However, most existing snapshot imaging spectrometers have the shortcomings of a large volume or heavy computational burden. In this paper, we present a novel snapshot imaging spectrometer based on the pixel-level filter array (PFA), which can simultaneously obtain both spectral and spatial information. The system is composed of a fore-optics, a PFA, a relay lens, and a monochromatic sensor. The incoming light first forms an intermediate image on the PFA through the fore-optics. Then, the relay lens reimages the spectral images on the PFA onto the monochromatic sensor. Through the use of the PFA, we can capture a three-dimensional (spatial coordinates and wavelength) datacube in a single exposure. Compared with existing technologies, our system possesses the advantages of a simple implementation, low cost, compact structure, and high energy efficiency by removing stacked dispersive or interferometric elements. Moreover, the characteristic of the direct imaging mode ensures the low computational burden of the system, thus shortening the imaging time. The principle and design of the system are described in detail. An experimental prototype is built and field experiments are carried out to verify the feasibility of the proposed scheme.

Microlens array snapshot hyperspectral microscopy system for the biomedical domain

We propose a microlens array-type snapshot hyperspectral microscope system that can provide spatial spectrum sampling according to detector frame rates for the biomedical domain. The system uses a shared optical path design. One path is used to perform direct microscopic imaging with high spatial resolution, while the other is used to collect microscopic images through a microlens array; the images are then spatially cut and reimaged such that they are spaced simultaneously by the prism-grating type hyperspectral imager's dispersion. Rapid acquisition of a three-dimensional data cube measuring 28×14×180 (x×y×λ) can be performed at the detector's frame rate. The system has a spatial resolution of 2.5 µm and can achieve 180-channel sampling of a 100 nm spectrum in the 400-800 nm spectral range with spectral resolution of approximately 0.56 nm. Spectral imaging results from biological samples show that the microlens array-type snapshot hyperspectral microscope system may potentially be applied in real-time biological spectral imaging.

Joint artifact correction and super-resolution of image slicing and mapping system via a convolutional neural network

As the key component of the image mapping spectrometer, the image mapper introduces complex image degradation in the reconstructed images, including low spatial resolution and intensity artifacts. In this paper, we propose a novel image processing method based on the convolutional neural network to perform artifact correction and super-resolution (SR) simultaneously. The proposed joint network contains two branches to handle the artifact correction task and SR task in parallel. The artifact correction module is designed to remove the artifacts in the image and the SR module is used to improve the spatial resolution. An attention fusion module is constructed to combine the features extracted by the artifact correction and SR modules. The fused features are used to reconstruct an artifact-free high-resolution image. We present extensive simulation results to demonstrate that the proposed joint method outperforms state-of-the-art methods and can be generalized to other image mapper designs. We also provide experimental results to prove the efficiency of the joint network.

Snapshot spectroscopic microscopy with double spherical slicer mirrors

Snapshot hyperspectral microscopic imaging can obtain the morphological characteristics and chemical specificity of samples simultaneously and instantaneously. We demonstrate a double-slicer spectroscopic microscopy (DSSM) that uses two spherical slicer mirrors to magnify the target image and slice it. These slits are lined up and dispersed, then mapped onto an area-array detector. An anamorphosis unit optimizes the capacity of the limited pixels. With a single shot and image recombination, a data cube can be constructed for sample analysis, and a model of DSSM is simulated. The system covers the spectral range from 500 nm to 642.5 nm with 20 spectral channels. The spatial resolution is 417 nm, and the spectral resolution is 7.5 nm.

Snapshot multidimensional photography through active optical mapping

Multidimensional photography can capture optical fields beyond the capability of conventional image sensors that measure only two-dimensional (2D) spatial distribution of light. By mapping a high-dimensional datacube of incident light onto a 2D image sensor, multidimensional photography resolves the scene along with other information dimensions, such as wavelength and time. However, the application of current multidimensional imagers is fundamentally restricted by their static optical architectures and measurement schemes—the mapping relation between the light datacube voxels and image sensor pixels is fixed. To overcome this limitation, we propose tunable multidimensional photography through active optical mapping. A high-resolution spatial light modulator, referred to as an active optical mapper, permutes and maps the light datacube voxels onto sensor pixels in an arbitrary and programmed manner. The resultant system can readily adapt the acquisition scheme to the scene, thereby maximising the measurement flexibility. Through active optical mapping, we demonstrate our approach in two niche implementations: hyperspectral imaging and ultrafast imaging.

Multidimensional photography has traditionally been restricted by their static optical architectures and measurement schemes. Here, the authors present a tunable multidimensional photography approach employing active optical mapping, which allows them to adapt the acquisition schemes to the scene.

Full-field spectral-domain optical interferometry for snapshot three-dimensional microscopy

Prevalent techniques in label-free linear optical microscopy are either confined to imaging in two dimensions or rely on scanning, both of which restrict their applications in imaging subtle biological dynamics. In this paper, we present the theoretical basis along with demonstrations supporting that full-field spectral-domain interferometry can be used for imaging samples in 3D with no moving parts in a single shot. Consequently, we propose a novel optical imaging modality that combines low-coherence interferometry with hyperspectral imaging using a light-emitting diode and an image mapping spectrometer, called Snapshot optical coherence microscopy (OCM). Having first proved the feasibility of Snapshot OCM through theoretical modeling and a comprehensive simulation, we demonstrate an implementation of the technique using off-the-shelf components capable of capturing an entire volume in 5 ms. The performance of Snapshot OCM, when imaging optical targets, shows its capability to axially localize and section images over an axial range of ±10 µm, while maintaining a transverse resolution of 0.8 µm, an axial resolution of 1.4 µm, and a sensitivity of up to 80 dB. Additionally, its performance in imaging weakly scattering live cells shows its capability to not only localize the cells in a densely populated culture but also to generate detailed phase profiles of the structures at each depth for long durations. Consolidating the advantages of several widespread optical microscopy modalities, Snapshot OCM has the potential to be a versatile imaging technique for a broad range of applications.

Development of a fast calibration method for image mapping spectrometry

An image mapping spectrometer (IMS) is a snapshot hyperspectral imager that simultaneously captures both the spatial (x, y) and spectral (λ) information of incoming light. The IMS maps a three-dimensional (3D) datacube (x, y, λ) to a two-dimensional (2D) detector array (x, y) for parallel measurement. To reconstruct the original 3D datacube, one must construct a lookup table that connects voxels in the datacube and pixels in the raw image. Previous calibration methods suffer from either low speed or poor image quality. We herein present a slit-scan calibration method that can significantly reduce the calibration time while maintaining high accuracy. Moreover, we quantitatively analyzed the major artifact in the IMS, the striped image, and developed three numerical methods to correct for it.

Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review [Invited]

Hyperspectral imaging (HSI) and multispectral imaging (MSI) technologies have the potential to transform the fields of digital and computational pathology. Traditional digitized histopathological slides are imaged with RGB imaging. Utilizing HSI/MSI, spectral information across wavelengths within and beyond the visual range can complement spatial information for the creation of computer-aided diagnostic tools for both stained and unstained histological specimens. In this systematic review, we summarize the methods and uses of HSI/MSI for staining and color correction, immunohistochemistry, autofluorescence, and histopathological diagnostic research. Studies include hematology, breast cancer, head and neck cancer, skin cancer, and diseases of central nervous, gastrointestinal, and genitourinary systems. The use of HSI/MSI suggest an improvement in the detection of diseases and clinical practice compared with traditional RGB analysis, and brings new opportunities in histological analysis of samples, such as digital staining or alleviating the inter-laboratory variability of digitized samples. Nevertheless, the number of studies in this field is currently limited, and more research is needed to confirm the advantages of this technology compared to conventional imagery.

Programmable hyperspectral microscopy for high-contrast biomedical imaging in a snapshot

SIGNIFICANCE

Hyperspectral microscopy has been intensively explored in biomedical applications. However, due to its huge three-dimensional hyperspectral data cube, it typically suffers from slow data acquisition, mass data transmission and storage, and computationally expensive postprocessing.

AIM

To overcome the above limitations, a programmable hyperspectral microscopy technique was developed, which can perform hardware-based hyperspectral data postprocessing by the physical process of optical imaging in a snapshot.

APPROACH

A programmable hyperspectral microscopy system was developed to collect coded microscopic images from samples under multiplexed illumination. Principal component analysis followed by linear discriminant analysis scheme was coded into multiplexed illumination and realized by the physical process of optical imaging. The contrast enhancement was evaluated on two representative types of microscopic samples, i.e., tissue section and cell samples.

RESULTS

Compared to the microscopic images collected under white light illumination, the contrasts of coded microscopic images were significantly improved by 41% and 59% for tissue section and cell samples, respectively.

CONCLUSIONS

The proposed method can perform hyperspectral data acquisition and postprocessing simultaneously by its physical process, while preserving the most important spectral information to maximize the difference between the target and background, thus opening a new avenue for high-contrast microscopic imaging in a snapshot.

Adversarial Networks for Scale Feature-Attention Spectral Image Reconstruction from a Single RGB

Hyperspectral images reconstruction focuses on recovering the spectral information from a single RGBimage. In this paper, we propose two advanced Generative Adversarial Networks (GAN) for the heavily underconstrained inverse problem. We first propose scale attention pyramid UNet (SAPUNet), which uses U-Net with dilated convolution to extract features. We establish the feature pyramid inside the network and use the attention mechanism for feature selection. The superior performance of this model is due to the modern architecture and capturing of spatial semantics. To provide a more accurate solution, we propose another distinct architecture, named W-Net, that builds one more branch compared to U-Net to conduct boundary supervision. SAPUNet and scale attention pyramid WNet (SAPWNet) provide improvements on the Interdisciplinary Computational Vision Lab at Ben Gurion University (ICVL) datasetby 42% and 46.6%, and 45% and 50% in terms of root mean square error (RMSE) and relative RMSE, respectively. The experimental results demonstrate that our proposed models are more accurate than the state-of-the-art hyperspectral recovery methods.

A plug-and-play Hyperspectral Imaging Sensor using low-cost equipment

Hyperspectral Imaging Sensors (HSI) obtain spectral information from an object, and they are used to solve problems in Remote Sensing, Food Analysis, Precision Agriculture, and others. This paper took advantage of modern high-resolution cameras, electronics, and optics to develop a robust, low-cost, and easy to assemble HSI device. This device could be used to evaluate new algorithms for hyperspectral image analysis and explore its feasibility to develop new applications on a low-budget. It weighs up to 300 g, detects wavelengths from 400 nm-1052 nm, and generates up to 315 different wavebands with a spectral resolution up to 2.0698 nm. Its spatial resolution of 116 × 110 pixels works for many applications. Furthermore, with only 2% of the cost of commercial HSI devices with similar characteristics, it has shown high spectral accuracy in controlled light conditions as well as ambient light conditions. Unlike related works, the proposed HSI system includes a framework to build the proposed HSI from scratch. This framework decreases the complexity of building an HSI device as well as the processing time. It contains every needed 3D model, a calibration method, the image acquisition software, and the methodology to build and calibrate the proposed HSI device. Therefore, the proposed HSI system is portable, reusable, and lightweight.

A background correction method to compensate illumination variation in hyperspectral imaging

Hyperspectral imaging (HSI) can measure both spatial (morphological) and spectral (biochemical) information from biological tissues. While HSI appears promising for biomedical applications, interpretation of hyperspectral images can be challenging when data is acquired in complex biological environments. Variations in surface topology or optical power distribution at the sample, encountered for example during endoscopy, can lead to errors in post-processing of the HSI data, compromising disease diagnostic capabilities. Here, we propose a background correction method to compensate for such variations, which estimates the optical properties of illumination at the target based on the normalised spectral profile of the light source and the measured HSI intensity values at a fixed wavelength where the absorption characteristics of the sample are relatively low (in this case, 800 nm). We demonstrate the feasibility of the proposed method by imaging blood samples, tissue-mimicking phantoms, and ex vivo chicken tissue. Moreover, using synthetic HSI data composed from experimentally measured spectra, we show the proposed method would improve statistical analysis of HSI data. The proposed method could help the implementation of HSI techniques in practical clinical applications, where controlling the illumination pattern and power is difficult.

Snapshot hyperspectral light field imaging using image mapping spectrometry

In this Letter, we present a snapshot hyperspectral light field imaging system using a single camera. By integrating an unfocused light field camera with a snapshot hyperspectral imager, the image mapping spectrometer, we captured a five-dimensional (5D) ($x,y,u,v,\lambda $x,y,u,v,λ) ($x,y,$x,y, spatial coordinates; $u,v,$u,v, emittance angles; $\lambda ,$λ, wavelength) datacube in a single camera exposure. The corresponding volumetric image ($x,y,z$x,y,z) at each wavelength is then computed through a scale-depth space transform. We demonstrated the snapshot advantage of our system by imaging the spectral-volumetric scenes in real time.

Accurate ray tracing model of an imaging system based on image mapper

The image mapper plays a key role in the imaging process of the image mapping spectrometer (IMS), which is a snapshot imaging spectrometer with superiority in light throughput, temporal resolution, and compactness. In this paper, an accurate ray tracing model of the imaging units of the IMS, especially the image mapper, is presented in the form of vector operation. Based on the proposed model, the behavior of light reflection on the image mapper is analyzed thoroughly, including the precise position of the reflection point and interaction between adjacent facets. Rigorous spatial correspondence between object points and pixels on the detector is determined by tracing the chief ray of an arbitrary point in the field. The shadowing effect, which is shadowing between adjacent facets and blocking caused by the facets' side walls, is analyzed based on our model. The experimental results verify the fidelity of the model and the existence of the shadowing effect. The research is meaningful for comprehending the imaging mechanism of the IMS and facilitates the design and analysis process in the future.

Ultrahigh resolution and color gamut with scattering-reducing transmissive pixels

While plasmonic designs have dominated recent trends in structural color, schemes using localized surface plasmon resonances and surface plasmon polaritons that simultaneously achieve high color vibrancy at ultrahigh resolution have been elusive because of tradeoffs between size and performance. Herein we demonstrate vibrant and size-invariant transmissive type multicolor pixels composed of hybrid TiO_x-Ag core-shell nanowires based on reduced scattering at their electric dipolar Mie resonances. This principle permits the hybrid nanoresonator to achieve the widest color gamut (~74% sRGB area coverage), linear color mixing, and the highest reported single color dots-per-inch (58,000~141,000) in transmission mode. Exploiting such features, we further show that an assembly of distinct nanoresonators can constitute a multicolor pixel for use in multispectral imaging, with a size that is ~10-folds below the Nyquist limit using a typical high NA objective lens.

Tradeoffs between size and performance have limited plasmonic structural color vibrancy at high resolution. Here the authors present a nanophotonic resonant metal-coated nanowire capable of being used as a size invariant, vibrant multicolor pixel.

Spectral reflectance recovery using optimal illuminations

The spectral reflectance of objects provides intrinsic information on material properties that have been proven beneficial in a diverse range of applications, e.g., remote sensing, agriculture and diagnostic medicine, to name a few. Existing methods for the spectral reflectance recovery from RGB or monochromatic images either ignore the effect from the illumination or implement/optimize the illumination under the linear representation assumption of the spectral reflectance. In this paper, we present a simple and efficient convolutional neural network (CNN)-based spectral reflectance recovery method with optimal illuminations. Specifically, we design illumination optimization layer to optimally multiplex illumination spectra in a given dataset or to design the optimal one under physical restrictions. Meanwhile, we develop the nonlinear representation for spectral reflectance in a data-driven way and jointly optimize illuminations under this representation in a CNN-based end-to-end architecture. Experimental results on both synthetic and real data show that our method outperforms the state-of-the-arts and verifies the advantages of deeply optimal illumination and nonlinear representation of the spectral reflectance.

A Dye-Free Analog to Retinal Angiography Using Hyperspectral Unmixing to Retrieve Oxyhemoglobin Abundance

Purpose

Retinal angiography evaluates retinal and choroidal perfusion and vascular integrity and is used to manage many ophthalmic diseases, such as age-related macular degeneration. The most common method, fluorescein angiography (FA), is invasive and can lead to untoward effects. As an emerging replacement, noninvasive OCT angiography (OCTA) is used regularly as a dye-free substitute with superior resolution and additional depth-sectioning abilities; however, general trends in FA as signified by varying intensity in images are not always reproducible in the fine structural detail in an OCTA image stack because of the source of their respective signals, OCT speckle decorrelation versus fluorescein emission.

Methods

We present a noninvasive/dye-free analog to angiography imaging using retinal hyperspectral imaging with a nonscanning spectral imager, the image mapping spectrometer (IMS), to reproduce perfusion-related data based on the abundance of oxyhemoglobin (HbO₂) in the retina. With a new unmixing procedure of the IMS-acquired spectral data cubes (350 × 350 × 43), we produced noninvasive HbO₂ maps unmixed from reflectance spectra.

Results

Here, we present 15 HbO₂ maps from seven healthy and eight diseased retinas and compare these maps with corresponding FA and OCTA results with a discussion of each technique.

Conclusions

Our maps showed visual agreement with hypo- and hyperfluorescence trends in venous phase FA images, suggesting that our method provides a new use for hyperspectral imaging as a noninvasive angiography-analog technique and as a complementary technique to OCTA.

Translational Relevance

The application of hyperspectral imaging and spectral analysis can potentially improve/broaden retinal disease screening and enable a noninvasive technique, which complements OCTA.

Light-guide snapshot imaging spectrometer for remote sensing applications

A fiber-based snapshot imaging spectrometer was developed with a maximum of 31853 (~188 x 170) spatial sampling and 61 spectral channels in the 450nm-750nm range. A compact, custom-fabricated fiber bundle was used to sample the object image at the input and create void spaces between rows at the output for dispersion. The bundle was built using multicore 6x6 fiber block ribbons. To avoid overlap between the cores in the direction of dispersion, we selected a subset of cores using two alternative approaches; a lenslet array and a photomask. To calibrate the >30000 spatial samples of the system, a rapid spatial calibration method was developed based on phase-shifting interferometry (PSI). System crosstalk and spectral resolution were also characterized. Preliminary hyperspectral imaging results of the Rice University campus landscape, obtained with the spectrometer, are presented to demonstrate the system's spectral imaging capability for distant scenes. The spectrum of different plant species with different health conditions, obtained with the spectrometer, was in accordance with reference instrument measurements. We also imaged Houston traffic to demonstrate the system's snapshot hyperspectral imaging capability. Potential applications of the system include terrestrial monitoring, land use, air pollution, water resources, and lightning spectroscopy. The fiber-based system design potentially allows tuning between spatial and spectral sampling to meet specific imaging requirements.

Snapshot channeled imaging spectrometer using geometric phase holograms

In this paper, we present the design and experimental demonstration of a snapshot imaging spectrometer based on channeled imaging spectrometry (CIS) and channeled imaging polarimetry (CIP). Using a geometric phase microlens array (GPMLA) with multiple focal lengths, the proposed spectrometer selects wavelength components within its designed operating waveband of 450-700 nm. Compared to other snapshot spectral imagers, its key components are especially suitable for roll-to-roll (R2R) rapid fabrication, which gives the spectrometer potential for low-cost mass production. The principles and proof-of-concept experimental system of the sensor are described in detail, followed by lab validation and outdoor measurement results which demonstrate the sensor's ability to resolve spectral and spatial contents under both experimental and natural illumination conditions.

Stokes shift microscopy by excitation and emission imaging

In this contribution, we present a new method, based on a tunable excitation laser source and a robust common path interferometer in the detection channel. Its purpose is to image spectral excitation and emission information on a monochrome complementary metal oxide semiconductor (CMOS) camera. This allows us to spatially obtain both excitation and emission spectra of the whole imaged area and create derived images such as red-green-blue (RGB), excitation and emission maxima, and Stokes shift images. Our presented method is a further development of hyperspectral imaging that usually is limited to recording spatially resolved emission spectra. Taking advantage of the full camera chip should speed up the acquisition versus line scan or pointwise hyperspectral imaging.

Compact snapshot optically replicating and remapping imaging spectrometer (ORRIS) using a focal plane continuous variable filter

In this Letter, a novel snapshot spectral imaging technique, optically replicating and remapping imaging spectrometer, is presented. It is based on the combination of shifting subimages by a specially designed lenslet array (LA) and filtering subimages by a focal plane continuous variable filter (CVF). The 3D datacube is recovered by just using a simple image remapping process. The use of the LA and the focal plane CVF makes the system compact and low in cost. A handheld proof-of-principle prototype has been built and demonstrated; it covers a wavelength range of 380-860 nm with 80 spectral channels with a spatial resolution of 400×400  pixels.

High performance image mapping spectrometer (IMS) for snapshot hyperspectral imaging applications

A high performance, snapshot Image Mapping Spectrometer was developed that provides fast image acquisition (100 Hz) of 16 bit hyperspectral data cubes (210x210x46) over a spectral range of 515-842 nm. Essential details of the opto-mechanical design are presented. Spectral accuracy, precision, and image reconstruction metrics such as resolution are discussed. Fluorescently stained cell samples were used to directly compare the data obtained using newly developed and the reference image mapping spectrometer. Additional experimental results are provided to demonstrate the abilities of the new spectrometer to acquire highly-resolved, motion-artifact-free hyperspectral images at high temporal sampling rates.

HyperReconNet: Joint Coded Aperture Optimization and Image Reconstruction for Compressive Hyperspectral Imaging

Coded aperture snapshot spectral imaging (CASSI) system encodes the 3D hyperspectral image (HSI) within a single 2D compressive image and then reconstructs the underlying HSI by employing an inverse optimization algorithm, which equips with the distinct advantage of snapshot but usually results in low reconstruction accuracy. To improve the accuracy, existing methods attempt to design either alternative coded apertures or advanced reconstruction methods, but cannot connect these two aspects via a unified framework, which limits the accuracy improvement. In this paper, we propose a convolution neural network (CNN) based endto- end method to boost the accuracy by jointly optimizing the coded aperture and the reconstruction method. On the one hand, based on the nature of CASSI forward model, we design a repeated pattern for the coded aperture, whose entities are learned by acting as the network weights. On the other hand, we conduct the reconstruction through simultaneously exploiting intrinsic properties within HSI - the extensive correlations across the spatial and the spectral dimensions. By leveraging the power of deep learning, the coded aperture design and the image reconstruction are connected and optimized via a unified framework. Experimental results show that our method outperforms the state-of-the-art methods under both comprehensive quantitative metrics and perceptive quality.

Super-multiplexed fluorescence microscopy via photostability contrast

Fluorescence microscopy is widely used to observe and quantify the inner workings of the cell. Traditionally, multiple types of cellular structures or biomolecules are visualized simultaneously in a sample by using spectrally distinct fluorescent labels. The wide emission spectra of most fluorophores limits spectral multiplexing to four or five labels in a standard fluorescence microscope. Further multiplexing requires another dimension of contrast. Here, we show that photostability differences can be used to distinguish between fluorescent labels. By combining photobleaching characteristics with a novel unmixing algorithm, we resolve up to three fluorescent labels in a single spectral channel and unmix fluorescent labels with nearly identical emission spectra. We apply our technique to organic dyes, autofluorescent biomolecules and fluorescent proteins. Our approach has the potential to triple the multiplexing capabilities of any digital widefield or confocal fluorescence microscope with no additional hardware, making it readily accessible to a wide range of researchers.

Hyperspectral imaging for simultaneous measurements of two FRET biosensors in pancreatic β-cells

Fluorescent protein (FP) biosensors based on Förster resonance energy transfer (FRET) are commonly used to study molecular processes in living cells. There are FP-FRET biosensors for many cellular molecules, but it remains difficult to perform simultaneous measurements of multiple biosensors. The overlapping emission spectra of the commonly used FPs, including CFP/YFP and GFP/RFP make dual FRET measurements challenging. In addition, a snapshot imaging modality is required for simultaneous imaging. The Image Mapping Spectrometer (IMS) is a snapshot hyperspectral imaging system that collects high resolution spectral data and can be used to overcome these challenges. We have previously demonstrated the IMS’s capabilities for simultaneously imaging GFP and CFP/YFP-based biosensors in pancreatic β-cells. Here, we demonstrate a further capability of the IMS to image simultaneously two FRET biosensors with a single excitation band, one for cAMP and the other for Caspase-3. We use these measurements to measure simultaneously cAMP signaling and Caspase-3 activation in pancreatic β-cells during oxidative stress and hyperglycemia, which are essential components in the pathology of diabetes.

Computational integral field spectroscopy with diverse imaging

Integral field spectroscopy (IFS) is a well-established method for measuring spectral intensity data of the form s(x,y,λ), where x, y are spatial coordinates and λ is the wavelength. In most flavors of IFS, there is a trade-off between sampling (x,y) and the measured wavelength band Δλ. Here we present the first, to our knowledge, attempt to overcome this trade-off by use of computational imaging and measurement diversity. We implement diversity by including a grating in our design, which allows rotation of the dispersed spectra between measurements. The raw intensity data captured from the rotated grating positions are then processed by an inverse algorithm that utilizes sparsity in the data. We present simulated results from spatial-spectral data in the experimental dataset. We used non-overlapping portions of the dataset to train our sparsity priors in the form of the dictionary, and to test the reconstruction quality. We found that, depending on the level of noise in the measurement, diversity up to a maximum number of measurements is beneficial in terms of reducing error, and yields diminishing returns if even more measurements are taken.

Multiplexed Optical Imaging of Tumor-Directed Nanoparticles: A Review of Imaging Systems and Approaches

In recent decades, various classes of nanoparticles have been developed for optical imaging of cancers. Many of these nanoparticles are designed to specifically target tumor sites, and specific cancer biomarkers, to facilitate the visualization of tumors. However, one challenge for accurate detection of tumors is that the molecular profiles of most cancers vary greatly between patients as well as spatially and temporally within a single tumor mass. To overcome this challenge, certain nanoparticles and imaging systems have been developed to enable multiplexed imaging of large panels of cancer biomarkers. Multiplexed molecular imaging can potentially enable sensitive tumor detection, precise delineation of tumors during interventional procedures, and the prediction/monitoring of therapy response. In this review, we summarize recent advances in systems that have been developed for the imaging of optical nanoparticles that can be heavily multiplexed, which include surface-enhanced Raman-scattering nanoparticles (SERS NPs) and quantum dots (QDs). In addition to surveying the optical properties of these various types of nanoparticles, and the most-popular multiplexed imaging approaches that have been employed, representative preclinical and clinical imaging studies are also highlighted.