Pre-processing visualization of hyperspectral fluorescent data with Spectrally Encoded Enhanced Representations

Deep learning-enhanced snapshot hyperspectral confocal microscopy imaging system

Laser-scanning confocal hyperspectral microscopy is a powerful technique to identify the different sample constituents and their spatial distribution in three-dimensional (3D). However, it suffers from low imaging speed because of the mechanical scanning methods. To overcome this challenge, we propose a snapshot hyperspectral confocal microscopy imaging system (SHCMS). It combined coded illumination microscopy based on a digital micromirror device (DMD) with a snapshot hyperspectral confocal neural network (SHCNet) to realize single-shot confocal hyperspectral imaging. With SHCMS, high-contrast 160-bands confocal hyperspectral images of potato tuber autofluorescence can be collected by only single-shot, which is almost 5 times improvement in the number of spectral channels than previously reported methods. Moreover, our approach can efficiently record hyperspectral volumetric imaging due to the optical sectioning capability. This fast high-resolution hyperspectral imaging method may pave the way for real-time highly multiplexed biological imaging.

Information-Rich Multi-Functional OCT for Adult Zebrafish Intra- and Extracranial Imaging

The zebrafish serves as a valuable animal model for both intra- and extracranial research, particularly in relation to the brain and skull. To effectively investigate the development and regeneration of adult zebrafish, a versatile in vivo imaging technique capable of showing both intra- and extracranial conditions is essential. In this paper, we utilized a high-resolution multi-functional optical coherence tomography (OCT) to obtain rich intra- and extracranial imaging outcomes of adult zebrafish, encompassing pigmentation distribution, tissue-specific information, cranial vascular imaging, and the monitoring of traumatic brain injury (TBI). Notably, it is the first that the channels through the zebrafish cranial suture, which may have a crucial function in maintaining the patency of the cranial sutures, have been observed. Rich imaging results demonstrated that a high-resolution multi-functional OCT system can provide a wealth of novel and interpretable biological information for intra- and extracranial studies of adult zebrafish.

A single-shot hyperspectral phasor camera for fast, multi-color fluorescence microscopy

Hyperspectral fluorescence imaging improves multiplexed observations of biological samples by utilizing multiple color channels across the spectral range to compensate for spectral overlap between labels. Typically, spectral resolution comes at a cost of decreased detection efficiency, which both hampers imaging speed and increases photo-toxicity to the samples. Here, we present a high-speed, high-efficiency snapshot spectral acquisition method, based on optical compression of the fluorescence spectra via Fourier transform, that overcomes the challenges of discrete spectral sampling: single-shot hyperspectral phasor camera (SHy-Cam). SHy-Cam captures fluorescence spatial and spectral information in a single exposure with a standard scientific CMOS camera, with photon efficiency of over 80%, easily and with acquisition rates exceeding 30 datasets per second, making it a powerful tool for multi-color in vivo imaging. Its simple design, using readily available optical components, and its easy integration provide a low-cost solution for multi-color fluorescence imaging with increased efficiency and speed.

Neural Networks for Hyperspectral Imaging of Historical Paintings: A Practical Review

Hyperspectral imaging (HSI) has become widely used in cultural heritage (CH). This very efficient method for artwork analysis is connected with the generation of large amounts of spectral data. The effective processing of such heavy spectral datasets remains an active research area. Along with the firmly established statistical and multivariate analysis methods, neural networks (NNs) represent a promising alternative in the field of CH. Over the last five years, the application of NNs for pigment identification and classification based on HSI datasets has drastically expanded due to the flexibility of the types of data they can process, and their superior ability to extract structures contained in the raw spectral data. This review provides an exhaustive analysis of the literature related to NNs applied for HSI data in the CH field. We outline the existing data processing workflows and propose a comprehensive comparison of the applications and limitations of the various input dataset preparation methods and NN architectures. By leveraging NN strategies in CH, the paper contributes to a wider and more systematic application of this novel data analysis method.

SPLIT-PIN software enabling confocal and super-resolution imaging with a virtually closed pinhole

In point-scanning microscopy, optical sectioning is achieved using a small aperture placed in front of the detector, i.e. the detection pinhole, which rejects the out-of-focus background. The maximum level of optical sectioning is theoretically obtained for the minimum size of the pinhole aperture, but this is normally prevented by the dramatic reduction of the detected signal when the pinhole is closed, leading to a compromise between axial resolution and signal-to-noise ratio. We have recently demonstrated that, instead of closing the pinhole, one can reach a similar level of optical sectioning by tuning the pinhole size in a confocal microscope and by analyzing the resulting image series. The method, consisting in the application of the separation of photons by lifetime tuning (SPLIT) algorithm to series of images acquired with tunable pinhole size, is called SPLIT-pinhole (SPLIT-PIN). Here, we share and describe a SPLIT-PIN software for the processing of series of images acquired at tunable pinhole size, which generates images with reduced out-of-focus background. The software can be used on series of at least two images acquired on available commercial microscopes equipped with a tunable pinhole, including confocal and stimulated emission depletion (STED) microscopes. We demonstrate applicability on different types of imaging modalities: (1) confocal imaging of DNA in a non-adherent cell line; (2) removal of out-of-focus background in super-resolved STED microscopy; (3) imaging of live intestinal organoids stained with a membrane dye.

A phasor-based approach to improve optical sectioning in any confocal microscope with a tunable pinhole

Confocal fluorescence microscopy is a well‐established imaging technique capable of generating thin optical sections of biological specimens. Optical sectioning in confocal microscopy is mainly determined by the size of the pinhole, a small aperture placed in front of a point detector. In principle, imaging with a closed pinhole provides the highest degree of optical sectioning. In practice, the dramatic reduction of signal‐to‐noise ratio (SNR) at smaller pinhole sizes makes challenging the use of pinhole sizes significantly smaller than 1 Airy Unit (AU). Here, we introduce a simple method to “virtually” perform confocal imaging at smaller pinhole sizes without the dramatic reduction of SNR. The method is based on the sequential acquisition of multiple confocal images acquired at different pinhole aperture sizes and image processing based on a phasor analysis. The implementation is conceptually similar to separation of photons by lifetime tuning (SPLIT), a technique that exploits the phasor analysis to achieve super‐resolution, and for this reason we call this method SPLIT‐pinhole (SPLIT‐PIN). We show with simulated data that the SPLIT‐PIN image can provide improved optical sectioning (i.e., virtually smaller pinhole size) but better SNR with respect to an image obtained with closed pinhole. For instance, two images acquired at 2 and 1 AU can be combined to obtain a SPLIT‐PIN image with a virtual pinhole size of 0.2 AU but with better SNR. As an example of application to biological imaging, we show that SPLIT‐PIN improves confocal imaging of the apical membrane in an in vitro model of the intestinal epithelium.

Live-cell fluorescence spectral imaging as a data science challenge

Live-cell fluorescence spectral imaging is an evolving modality of microscopy that uses specific properties of fluorophores, such as excitation or emission spectra, to detect multiple molecules and structures in intact cells. The main challenge of analyzing live-cell fluorescence spectral imaging data is the precise quantification of fluorescent molecules despite the weak signals and high noise found when imaging living cells under non-phototoxic conditions. Beyond the optimization of fluorophores and microscopy setups, quantifying multiple fluorophores requires algorithms that separate or unmix the contributions of the numerous fluorescent signals recorded at the single pixel level. This review aims to provide both the experimental scientist and the data analyst with a straightforward description of the evolution of spectral unmixing algorithms for fluorescence live-cell imaging. We show how the initial systems of linear equations used to determine the concentration of fluorophores in a pixel progressively evolved into matrix factorization, clustering, and deep learning approaches. We outline potential future trends on combining fluorescence spectral imaging with label-free detection methods, fluorescence lifetime imaging, and deep learning image analysis.

Spatial transcriptomics using combinatorial fluorescence spectral and lifetime encoding, imaging and analysis

Multiplexed mRNA profiling in the spatial context provides new information enabling basic research and clinical applications. Unfortunately, existing spatial transcriptomics methods are limited due to either low multiplexing or complexity. Here, we introduce a spatialomics technology, termed Multi Omic Single-scan Assay with Integrated Combinatorial Analysis (MOSAICA), that integrates in situ labeling of mRNA and protein markers in cells or tissues with combinatorial fluorescence spectral and lifetime encoded probes, spectral and time-resolved fluorescence imaging, and machine learning-based decoding. We demonstrate MOSAICA's multiplexing scalability in detecting 10-plex targets in fixed colorectal cancer cells using combinatorial labeling of five fluorophores with facile error-detection and removal of autofluorescence. MOSAICA's analysis is strongly correlated with sequencing data (Pearson's r = 0.96) and was further benchmarked using RNAscopeTM and LGC StellarisTM. We further apply MOSAICA for multiplexed analysis of clinical melanoma Formalin-Fixed Paraffin-Embedded (FFPE) tissues. We finally demonstrate simultaneous co-detection of protein and mRNA in cancer cells.

Hyperspectral Mapping for the Detection of SARS-CoV-2 Using Nanomolecular Probes with Yoctomole Sensitivity

Efficient monitoring of SARS-CoV-2 outbreak requires the use of a sensitive and rapid diagnostic test. Although SARS-CoV-2 RNA can be detected by RT-qPCR, the molecular-level quantification of the viral load is still challenging, time-consuming, and labor-intensive. Here, we report an ultrasensitive hyperspectral sensor (HyperSENSE) based on hafnium nanoparticles (HfNPs) for specific detection of COVID-19 causative virus, SARS-CoV-2. Density functional theoretical calculations reveal that HfNPs exhibit higher changes in their absorption wavelength and light scattering when bound to their target SARS-CoV-2 RNA sequence relative to the gold nanoparticles. The assay has a turnaround time of a few seconds and has a limit of detection in the yoctomolar range, which is 1 000 000-fold times higher than the currently available COVID-19 tests. We demonstrated in ∼100 COVID-19 clinical samples that the assay is highly sensitive and has a specificity of 100%. We also show that HyperSENSE can rapidly detect other viruses such as influenza A H1N1. The outstanding sensitivity indicates the potential of the current biosensor in detecting the prevailing presymptomatic and asymptomatic COVID-19 cases. Thus, integrating hyperspectral imaging with nanomaterials establishes a diagnostic platform for ultrasensitive detection of COVID-19 that can potentially be applied to any emerging infectious pathogen.

Rapid statistical discrimination of fluorescence images of T cell receptors on immobilizing surfaces with different coating conditions

The spatial organization of T cell receptors (TCRs) correlates with membrane-associated signal amplification, dispersion, and regulation during T cell activation. Despite its potential clinical importance, quantitative analysis of the spatial arrangement of TCRs from standard fluorescence images remains difficult. Here, we report Statistical Classification Analyses of Membrane Protein Images or SCAMPI as a technique capable of analyzing the spatial arrangement of TCRs on the plasma membrane of T cells. We leveraged medical image analysis techniques that utilize pixel-based values. We transformed grayscale pixel values from fluorescence images of TCRs into estimated model parameters of partial differential equations. The estimated model parameters enabled an accurate classification using linear discrimination techniques, including Fisher Linear Discriminant (FLD) and Logistic Regression (LR). In a proof-of-principle study, we modeled and discriminated images of fluorescently tagged TCRs from Jurkat T cells on uncoated cover glass surfaces (Null) or coated cover glass surfaces with either positively charged poly-L-lysine (PLL) or TCR cross-linking anti-CD3 antibodies (OKT3). Using 80 training images and 20 test images per class, our statistical technique achieved 85% discrimination accuracy for both OKT3 versus PLL and OKT3 versus Null conditions. The run time of image data download, model construction, and image discrimination was 21.89 s on a laptop computer, comprised of 20.43 s for image data download, 1.30 s on the FLD-SCAMPI analysis, and 0.16 s on the LR-SCAMPI analysis. SCAMPI represents an alternative approach to morphology-based qualifications for discriminating complex patterns of membrane proteins conditioned on a small sample size and fast runtime. The technique paves pathways to characterize various physiological and pathological conditions using the spatial organization of TCRs from patient T cells.

Phasor-based hyperspectral snapshot microscopy allows fast imaging of live, three-dimensional tissues for biomedical applications

Hyperspectral imaging is highly sought after in many fields including mineralogy and geology, environment and agriculture, astronomy and, importantly, biomedical imaging and biological fluorescence. We developed ultrafast phasor-based hyperspectral snapshot microscopy based on sine/cosine interference filters for biomedical imaging not feasible with conventional hyperspectral detection methods. Current approaches rely on slow spatial or spectral scanning limiting their application in living biological tissues, while faster snapshot methods such as image mapping spectrometry and multispectral interferometry are limited in spatial and/or spectral resolution, are computationally demanding, and imaging devices are very expensive to manufacture. Leveraging light sheet microscopy, phasor-based hyperspectral snapshot microscopy improved imaging speed 10-100 fold which, combined with minimal light exposure and high detection efficiency, enabled hyperspectral metabolic imaging of live, three-dimensional mouse tissues not feasible with other methods. As a fit-free method that does not require any a priori information often unavailable in complex and evolving biological systems, the rule of linear combinations of the phasor could spectrally resolve subtle differences between cell types in the developing zebrafish retina and spectrally separate and track multiple organelles in 3D cultured cells over time. The sine/cosine snapshot method is adaptable to any microscope or imaging device thus making hyperspectral imaging and fit-free analysis based on linear combinations broadly available to researchers and the public.

Phasor-based image segmentation: machine learning clustering techniques

The phasor approach is a well-established method for data visualization and image analysis in spectral and lifetime fluorescence microscopy. Nevertheless, it is typically applied in a user-dependent manner by manually selecting regions of interest on the phasor space to find distinct regions in the fluorescence images. In this paper we present our work on using machine learning clustering techniques to establish an unsupervised and automatic method that can be used for identifying populations of fluorescent species in spectral and lifetime imaging. We demonstrate our method using both synthetic data, created by sampling photon arrival times and plotting the distributions on the phasor plot, and real live cells samples, by staining cellular organelles with a selection of commercial probes.