Snapshot Hyperspectral Volumetric Microscopy

Development of a large volume line scanning, high spectral range and resolution 3D hyperspectral photoluminescence imaging microscope for diamond and other high refractive index materials

Hyperspectral photoluminescence (PL) imaging is a powerful technique that can be used to understand the spatial distribution of emitting species in many materials. Volumetric hyperspectral imaging of weakly emitting color centers often necessitates considerable data collection times when using commercial systems. We report the development of a line-scanning hyperspectral imaging microscope capable of measuring the luminescence emission spectra for diamond volumes up to 2.20 × 30.00 × 6.30 mm with a high lateral spatial resolution of 1-3 µm. In an single X-λ measurement, spectra covering a 711 nm range, in a band from 400-1100 nm, with a spectral resolution up to 0.25 nm can be acquired. Data sets can be acquired with 723 (X) × 643 (Y) × 1172 (λ) pixels at a rate of 6 minutes/planar image slice, allowing for volumetric hyperspectral imaging with high sampling. This instrument demonstrates the ability to detect emission from several different color centers in diamond both at the surface and internally, providing a non-destructive method to probe their 3D spatial distribution, and is currently not achievable with any other commonly used system or technique.

3D Amplified Single-Cell RNA and Protein Imaging Identifies Oncogenic Transcript Subtypes in B-Cell Acute Lymphoblastic Leukemia

Simultaneous in situ detection of transcript and protein markers at the single-cell level is essential for gaining a better understanding of tumor heterogeneity and for predicting and monitoring treatment responses. However, the limited accessibility to advanced 3D imaging techniques has hindered their rapid implementation. Here, we present a 3D single-cell imaging technique, termed 3D digital rolling circle amplification (4DRCA), capable of the multiplexed and amplified simultaneous digital quantification of single-cell RNAs and proteins using standard fluorescence microscopy and off-the-shelf reagents. We generated spectrally distinguishable DNA amplicons from molecular markers through an integrative protocol combining single-cell RNA and protein assays and directly enumerated the amplicons by leveraging an open-source algorithm for 3D deconvolution with a custom-built automatic gating algorithm. With 4DRCA, we were able to simultaneously quantify surface protein markers and cytokine transcripts in T-lymphocytes. We also show that 4DRCA can distinguish BCR-ABL1 fusion transcript positive B-cell acute lymphoblastic leukemia cells with or without CD19 protein expression. The accessibility and extensibility of 4DRCA render it broadly applicable to other cell-based diagnostic workflows, enabling sensitive and accurate single-cell RNA and protein profiling.

Visible and NIR microscopic hyperspectrum reconstruction from RGB images with deep convolutional neural networks

We investigate the microscopic hyperspectral reconstruction from RGB images with a deep convolutional neural network (DCNN) in this paper. Based on the microscopic hyperspectral imaging system, a homemade dataset consisted of microscopic hyperspectral and RGB image pairs is constructed. For considering the importance of spectral correlation between neighbor spectral bands in microscopic hyperspectrum reconstruction, the 2D convolution is replaced by 3D convolution in the DCNN framework, and a metric (weight factor) used to evaluate the performance reconstructed hyperspectrum is also introduced into the loss function used in training. The effects of the dimension of convolution kernel and the weight factor in the loss function on the performance of the reconstruction model are studied. The overall results indicate that our model can show better performance than the traditional models applied to reconstruct the hyperspectral images based on DCNN for the public and the homemade microscopic datasets. In addition, we furthermore explore the microscopic hyperspectrum reconstruction from RGB images in infrared region, and the results show that the model proposed in this paper has great potential to expand the reconstructed hyperspectrum wavelength range from the visible to near infrared bands.

Microsphere-assisted super-resolved hyperspectral microscopy

Hyperspectral microscopy (HSM) combines conventional microscopy with basic hyperspectral imaging and results in 3D microscopic spatio-spectral information. The combination comes along with a variety of applications, such as detection and classification of different material properties through spectral fingerprints, which otherwise cannot be detected with a color camera alone. However, similar to other microscopies, the resolution of HSM is limited by diffraction. In recent years microsphere (MS)-assisted microscopy has attracted intensive attention for resolution enhancement. Here, we extend the MS idea into HSM. We show that MS-HSM reveals extra hyperspectral information about the specimen which is lost without it. Moreover, the insertion of MS results in super-resolved images. The resolution enhancement and obtaining the extra hyperspectral information can be tuned by the MS size.

Spatial-spectral resolution tunable snapshot imaging spectrometer: analytical design and implementation

A snapshot imaging spectrometer is a powerful tool for dynamic target tracking and real-time recognition compared with a scanning imaging spectrometer. However, all the current snapshot spectral imaging techniques suffer from a major trade-off between the spatial and spectral resolutions. In this paper, an integral field snapshot imaging spectrometer (TIF-SIS) with a continuously tunable spatial-spectral resolution and light throughput is proposed and demonstrated. The proposed TIF-SIS is formed by a fore optics, a lenslet array, and a collimated dispersive subsystem. Theoretical analyses indicate that the spatial-spectral resolution and light throughput of the system can be continuously tuned through adjusting the F number of the fore optics, the rotation angle of the lenslet array, or the focal length of the collimating lens. Analytical relationships between the spatial and spectral resolutions and the first-order parameters of the system with different geometric arrangements of the lenslet unit are obtained. An experimental TIF-SIS consisting of a self-fabricated lenslet array with a pixelated scale of 100×100 and a fill factor of 0.716 is built. The experimental results show that the spectral resolution of the system can be steadily improved from 4.17 to 0.82 nm with a data cube (N x×N y×N λ) continuously tuned from 35×35×36 to 40×40×183 in the visible wavelength range from 500 to 650 nm, which is consistent with the theoretical prediction. The proposed method for real-time tuning of the spatial-spectral resolution and light throughput opens new possibilities for broader applications, especially for recognition of things with weak spectral signature and biomedical investigations where a high light throughput and tunable resolution are needed.

An integrated imaging sensor for aberration-corrected 3D photography

Planar digital image sensors facilitate broad applications in a wide range of areas^1-5, and the number of pixels has scaled up rapidly in recent years^2,6. However, the practical performance of imaging systems is fundamentally limited by spatially nonuniform optical aberrations originating from imperfect lenses or environmental disturbances^7,8. Here we propose an integrated scanning light-field imaging sensor, termed a meta-imaging sensor, to achieve high-speed aberration-corrected three-dimensional photography for universal applications without additional hardware modifications. Instead of directly detecting a two-dimensional intensity projection, the meta-imaging sensor captures extra-fine four-dimensional light-field distributions through a vibrating coded microlens array, enabling flexible and precise synthesis of complex-field-modulated images in post-processing. Using the sensor, we achieve high-performance photography up to a gigapixel with a single spherical lens without a data prior, leading to orders-of-magnitude reductions in system capacity and costs for optical imaging. Even in the presence of dynamic atmosphere turbulence, the meta-imaging sensor enables multisite aberration correction across 1,000 arcseconds on an 80-centimetre ground-based telescope without reducing the acquisition speed, paving the way for high-resolution synoptic sky surveys. Moreover, high-density accurate depth maps can be retrieved simultaneously, facilitating diverse applications from autonomous driving to industrial inspections.

Batch fabrication and compact integration of customized multispectral filter arrays towards snapshot imaging

Snapshot multispectral imaging (MSI) has been widely employed in the rapid visual inspection by virtues of the non-invasive detection mode and short integration time. As the critical functional elements of snapshot MSI, narrowband, customizable, and pixel-level multispectral filter arrays (MSFAs) that are compatible with imaging sensors are difficult to be efficiently manufactured. Meanwhile, monolithically integrating MSFAs into snapshot multispectral imagers still remains challenging considering the strict alignment precision. Here, we propose a cost-efficient, wafer-level, and customized approach for fabricating transmissive MSFAs based on Fabry-Perot structures, both in the pixel-level and window-tiled configuration, by utilizing the conventional lithography combined with the deposition method. The MSFA chips own a total dimension covering the area of 4.8 mm × 3.6 mm with 4 × 4 bands, possessing the capability to maintain narrow line widths (∼25 nm) across the whole visible frequencies. After the compact integration with the imaging sensor, the MSFAs are validated to be effective in filtering and target identification. Our proposed fabrication method and imaging mode show great potentials to be an alternative to MSFAs production and MSI, by reducing both complexity and cost of manufacturing, while increasing flexibility and customization of imaging system.

Hyperspectral Three-Dimensional Fluorescence Imaging Using Snapshot Optical Tomography

Hyperspectral three-dimensional (3D) imaging can provide both 3D structural and functional information of a specimen. The imaging throughput is typically very low due to the requirement of scanning mechanisms for different depths and wavelengths. Here we demonstrate hyperspectral 3D imaging using Snapshot projection optical tomography (SPOT) and Fourier-transform spectroscopy (FTS). SPOT allows us to instantaneously acquire the projection images corresponding to different viewing angles, while FTS allows us to perform hyperspectral imaging at high spectral resolution. Using fluorescent beads and sunflower pollens, we demonstrate the imaging performance of the developed system.

Compressive spectral image fusion via a single aperture high throughput imaging system

Spectral image fusion techniques combine the detailed spatial information of a multispectral (MS) image and the rich spectral information of a hyperspectral (HS) image into a high-spatial and high-spectral resolution image. Due to the data deluge entailed by such images, new imaging modalities have exploited their intrinsic correlations in such a way that, a computational algorithm can fuse them from few multiplexed linear projections. The latter has been coined compressive spectral image fusion. State-of-the-art research work have focused mainly on the algorithmic part, simulating instrumentation characteristics and assuming independently registered sensors to conduct compressed MS and HS imaging. In this manuscript, we report on the construction of a unified computational imaging framework that includes a proof-of-concept optical testbed to simultaneously acquire MS and HS compressed projections, and an alternating direction method of multipliers algorithm to reconstruct high-spatial and high-spectral resolution images from the fused compressed measurements. The testbed employs a digital micro-mirror device (DMD) to encode and split the input light towards two compressive imaging arms, which collect MS and HS measurements, respectively. This strategy entails full light throughput sensing since no light is thrown away by the coding process. Further, different resolutions can be dynamically tested by binning the DMD and sensors pixels. Real spectral responses and optical characteristics of the employed equipment are obtained through a per-pixel point spread function calibration approach to enable accurate compressed image fusion performance. The proposed framework is demonstrated through real experiments within the visible spectral range using as few as 5% of the data.

DMD-based hyperspectral microscopy with flexible multiline parallel scanning

As one of the most common hyperspectral microscopy (HSM) techniques, line-scanning HSM is currently utilized in many fields. However, its scanning efficiency is still considered to be inadequate since many biological and chemical processes occur too rapidly to be captured. Accordingly, in this work, a digital micromirror device (DMD) based on microelectromechanical systems (MEMS) is utilized to demonstrate a flexible multiline scanning HSM system. To the best of our knowledge, this is the first line-scanning HSM system in which the number of scanning lines N can be tuned by simply changing the DMD's parallel scanning units according to diverse applications. This brilliant strategy of effortless adjustability relies only on on-chip scanning methods and totally exploits the benefits of parallelization, aiming to achieve nearly an N-time improvement in the detection efficiency and an N-time decrease in the scanning time and data volume compared with the single-line method under the same operating conditions. To validate this, we selected a few samples of different spectral wavebands to perform reflection imaging, transmission imaging, and fluorescence imaging with varying numbers of scanning lines. The results show the great potential of our DMD-based HSM system for the rapid development of cellular biology, material analysis, and so on. In addition, its on-chip scanning process eliminates the inherent microscopic architecture, making the whole system compact, lightweight, portable, and not subject to site constraints.

Snapshot spectral polarimetric light field imaging using a single detector

In this Letter, we investigate a snapshot spectral-polarimetric-volumetric imaging (SSPVI) system using a single detector. Through compressed acquisition and reconstruction, SSPVI can achieve spectral imaging (x,y,λ), polarization imaging (x,y,ψ,χ), and light field imaging (x,y,θ,φ) simultaneously. The newly discovered performance is showcased by attaining the spectral-polarimetric-volumetric video and different laboratory accuracy experiments. These never-seen-before capacities of the camera open new prospects for many applications, such as biological analysis, object recognition, and remote sensing.

Quasi-spectral characterization of intracellular regions in bright-field light microscopy images

Investigation of cell structure is hardly imaginable without bright-field microscopy. Numerous modifications such as depth-wise scanning or videoenhancement make this method being state-of-the-art. This raises a question what maximal information can be extracted from ordinary (but well acquired) bright-field images in a model-free way. Here we introduce a method of a physically correct extraction of features for each pixel when these features resemble a transparency spectrum. The method is compatible with existent ordinary bright-field microscopes and requires mathematically sophisticated data processing. Unsupervised clustering of the spectra yields reasonable semantic segmentation of unstained living cells without any a priori information about their structures. Despite the lack of reference data (to prove strictly that the proposed feature vectors coincide with transparency), we believe that this method is the right approach to an intracellular (semi)quantitative and qualitative chemical analysis.

A hyperspectral projector for simultaneous 3D spatial and hyperspectral imaging via structured illumination

Both 3D imaging and hyperspectral imaging provide important information of the scene and combining them is beneficial in helping us perceive and understand real-world structures. Previous hyperspectral 3D imaging systems typically require a hyperspectral imaging system as the detector suffers from complicated hardware design, high cost, and high acquisition and reconstruction time. Here, we report a low-cost, high-frame rate, simple-design, and compact hyperspectral stripe projector (HSP) system based on a single digital micro-mirror device, capable of producing hyperspectral patterns where each row of pixels has an independently programmable spectrum. We demonstrate two example applications using the HSP via hyperspectral structured illumination: hyperspectral 3D surface imaging and spectrum-dependent hyperspectral compressive imaging of volume density of participating medium. The hyperspectral patterns simultaneously encode the 3D spatial and spectral information of the target, requiring only a grayscale sensor as the detector. The reported HSP and its applications provide a solution for combining structured illumination techniques with hyperspectral imaging in a simple, efficient, and low-cost manner. The work presented here represents a novel structured illumination technique that provides the basis and inspiration of future variations of hardware systems and software encoding schemes.

Snapshot hyperspectral light field imaging using image mapping spectrometry

In this Letter, we present a snapshot hyperspectral light field imaging system using a single camera. By integrating an unfocused light field camera with a snapshot hyperspectral imager, the image mapping spectrometer, we captured a five-dimensional (5D) ($x,y,u,v,\lambda $x,y,u,v,λ) ($x,y,$x,y, spatial coordinates; $u,v,$u,v, emittance angles; $\lambda ,$λ, wavelength) datacube in a single camera exposure. The corresponding volumetric image ($x,y,z$x,y,z) at each wavelength is then computed through a scale-depth space transform. We demonstrated the snapshot advantage of our system by imaging the spectral-volumetric scenes in real time.

Applications of hyperspectral imaging in the detection and diagnosis of solid tumors

Hyperspectral imaging (HSI) is an emerging new technology in solid tumor diagnosis and detection. It incorporates traditional imaging and spectroscopy together to obtain both spatial and spectral information from tissues simultaneously in a non-invasive manner. This imaging modality is based on the principle that different tissues inherit different spectral reflectance responses that present as unique spectral fingerprints. HSI captures those composition-specific fingerprints to identify cancerous and normal tissues. It becomes a promising tool for performing tumor diagnosis and detection from the label-free histopathological examination to real-time intraoperative assistance. This review introduces the basic principles of HSI and summarizes its methodology and recent advances in solid tumor detection. In particular, the advantages of HSI applied to solid tumors are highlighted to show its potential for clinical use.

Phase-space deconvolution for light field microscopy

Light field microscopy, featuring with snapshot large-scale three-dimensional (3D) fluorescence imaging, has aroused great interests in various biological applications, especially for high-speed 3D calcium imaging. Traditional 3D deconvolution algorithms based on the beam propagation model facilitate high-resolution 3D reconstructions. However, such a high-precision model is not robust enough for the experimental data with different system errors such as optical aberrations and background fluorescence, which bring great periodic artifacts and reduce the image contrast. In order to solve this problem, here we propose a phase-space deconvolution method for light field microscopy, which fully exploits the smoothness prior in the phase-space domain. By modeling the imaging process in the phase-space domain, we convert the spatially-nonuniform point spread function (PSF) into a spatially-uniform one with a much smaller size. Experiments on various biological samples and resolution charts are demonstrated to verify the contrast enhancement with much fewer artifacts and 10-times less computational cost by our method without any hardware modifications required.

A clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract

Hyperspectral imaging (HSI) enables visualisation of morphological and biochemical information, which could improve disease diagnostic accuracy. Unfortunately, the wide range of image distortions that arise during flexible endoscopy in the clinic have made integration of HSI challenging. To address this challenge, we demonstrate a hyperspectral endoscope (HySE) that simultaneously records intrinsically co-registered hyperspectral and standard-of-care white light images, which allows image distortions to be compensated computationally and an accurate hyperspectral data cube to be reconstructed as the endoscope moves in the lumen. Evaluation of HySE performance shows excellent spatial, spectral and temporal resolution and high colour fidelity. Application of HySE enables: quantification of blood oxygenation levels in tissue mimicking phantoms; differentiation of spectral profiles from normal and pathological ex vivo human tissues; and recording of hyperspectral data under freehand motion within an intact ex vivo pig oesophagus model. HySE therefore shows potential for enabling HSI in clinical endoscopy.

Multicolor fluorescent imaging by space-constrained computational spectral imaging

Spectral imaging is a powerful technique used to simultaneously study multiple fluorophore labels with overlapping emissions. Here, we present a computational spectral imaging method, which uses sample spatial fluorescence information as a reconstruction constraint. Our method addresses both the under-sampling issue of compressive spectral imaging and the low throughput issue of scanning spectral imaging. With simulated and experimental data, we have demonstrated the reconstruction precision of our method in two and three-color imaging. We have experimentally validated this method for differentiating cellular structures labeled with two red-colored fluorescent proteins, tdTomato and mCherry, which have highly overlapping emission spectra. Our method has the advantage of totally free wavelength choice and can also be combined with conventional filter-based sequential multi-color imaging to further improve multiplexing capability.

Complete plenoptic imaging using a single detector

Multi-dimensional imaging is a powerful technique for many applications, such as biological analysis, remote sensing, and object recognition. Most existing multi-dimensional imaging systems rely on scanning or camera array, which make the system bulky and unstable. To some extent, these problems can be mitigated by employing compressed sensing algorithms. However, they are computationally expensive and highly rely on the ill-posed assumption that the information is sparse in a given domain. Here, we propose a snapshot spectral-volumetric imaging (SSVI) system by introducing the paradigm of light-field imaging into Fourier transform imaging spectroscopy. We demonstrate that SSVI can reconstruct a complete plenoptic function, P(x,y,z,θ,φ,λ,t), of the incoming light rays using a single detector. Compared with other multidimensional imagers, SSVI features prominent advantages in compactness, robustness, and low cost.

5D hyperspectral imaging: fast and accurate measurement of surface shape and spectral characteristics using structured light

Measuring the shape (coordinates x, y, z ) and spectral characteristics (wavelength-dependent reflectance R (λi)) of macroscopic objects as a function of time (t) is of great interest in areas such as medical imaging, precision agriculture, or optical sorting. Here, we present an approach that allows to determine all these quantities with high resolution and accuracy, enabling measurement in five dimensions. We call this approach 5D hyperspectral imaging. We describe the design and implementation of a 5D sensor operating in the visible to near-infrared spectral range, which provides excellent spatial and spectral resolution, great depth accuracy, and high frame rates. The results of various experiments strongly indicate the great benefit of the new technology.

Terapixel hyperspectral whole-slide imaging via slit-array detection and projection

Digital pathology via whole-slide imaging (WSI) systems has recently been approved for the primary diagnostic use in the US. Acquiring whole-slide images with spectral information at each pixel permits the use of multiplexed antibody labeling and allow for the measurement of cellularly resolved chemical information. Here, we report the development of a high-throughput terapixel hyperspectral WSI system using prism-based slit-array dispersion. We demonstrate a slit-array detection scheme for absorption-based measurements and a slit-array projection scheme for fluorescence-based measurements. The spectral resolution and spectral range in the reported schemes can be adjusted by changing the orientation of the slit-array mask. We use our system to acquire 74 5-megapixel brightfield images at different wavelengths in ∼1  s, corresponding to a throughput of 0.375  gigapixels  /  s. A terapixel whole-slide spatial-spectral data cube can be obtained in ∼45  min. The reported system is compatible with existing WSI systems and can be developed as an add-on module for whole-slide spectral imaging. It may find broad applications in high-throughput chemical imaging with multiple antibody labeling. The use of slit array for structured illumination may also provide insights for developing high-throughput hyperspectral confocal imaging systems.

Dual light-emitting diode-based multichannel microscopy for whole-slide multiplane, multispectral and phase imaging

We report the development of a multichannel microscopy for whole-slide multiplane, multispectral and phase imaging. We use trinocular heads to split the beam path into 6 independent channels and employ a camera array for parallel data acquisition, achieving a maximum data throughput of approximately 1 gigapixel per second. To perform single-frame rapid autofocusing, we place 2 near-infrared light-emitting diodes (LEDs) at the back focal plane of the condenser lens to illuminate the sample from 2 different incident angles. A hot mirror is used to direct the near-infrared light to an autofocusing camera. For multiplane whole-slide imaging (WSI), we acquire 6 different focal planes of a thick specimen simultaneously. For multispectral WSI, we relay the 6 independent image planes to the same focal position and simultaneously acquire information at 6 spectral bands. For whole-slide phase imaging, we acquire images at 3 focal positions simultaneously and use the transport-of-intensity equation to recover the phase information. We also provide an open-source design to further increase the number of channels from 6 to 15. The reported platform provides a simple solution for multiplexed fluorescence imaging and multimodal WSI. Acquiring an instant focal stack without z-scanning may also enable fast 3-dimensional dynamic tracking of various biological samples.

Spectral unmixing method for multi-pixel energy dispersive x-ray diffraction systems

An algorithm of spectral unmixing (SU) is presented, allowing the improvement of material classification accuracy based on the low spatial resolution images obtained by multi-pixel energy dispersive x-ray diffraction (EDXRD) systems. The method, which consists of signal subspace identification and endmember extraction, performs well even when the pixel count is rather small. Combined with SU, EDXRD has been utilized for liquid security screening for the first time. The spectra and abundance distributions of endmembers are extracted from the measured data sets corresponding to objects of different material composition, which demonstrates the validity of the proposed method.