Pre-processing visualization of hyperspectral fluorescent data with Spectrally Encoded Enhanced Representations

Noninvasive Oral Hyperspectral Imaging-Driven Digital Diagnosis of Heart Failure With Preserved Ejection Fraction: Model Development and Validation Study

BACKGROUND

Oral microenvironmental disorders are associated with an increased risk of heart failure with preserved ejection fraction (HFpEF). Hyperspectral imaging (HSI) technology enables the detection of substances that are visually indistinguishable to the human eye, providing a noninvasive approach with extensive applications in medical diagnostics.

OBJECTIVE

The objective of this study is to develop and validate a digital, noninvasive oral diagnostic model for patients with HFpEF using HSI combined with various machine learning algorithms.

METHODS

Between April 2023 and August 2023, a total of 140 patients were recruited from Renmin Hospital of Wuhan University to serve as the training and internal testing groups for this study. Subsequently, from August 2024 to September 2024, an additional 35 patients were enrolled from Three Gorges University and Yichang Central People's Hospital to constitute the external testing group. After preprocessing to ensure image quality, spectral and textural features were extracted from the images. We extracted 25 spectral bands from each patient image and obtained 8 corresponding texture features to evaluate the performance of 28 machine learning algorithms for their ability to distinguish control participants from participants with HFpEF. The model demonstrating the optimal performance in both internal and external testing groups was selected to construct the HFpEF diagnostic model. Hyperspectral bands significant for identifying participants with HFpEF were identified for further interpretative analysis. The Shapley Additive Explanations (SHAP) model was used to provide analytical insights into feature importance.

RESULTS

Participants were divided into a training group (n=105), internal testing group (n=35), and external testing group (n=35), with consistent baseline characteristics across groups. Among the 28 algorithms tested, the random forest algorithm demonstrated superior performance with an area under the receiver operating characteristic curve (AUC) of 0.884 and an accuracy of 82.9% in the internal testing group, as well as an AUC of 0.812 and an accuracy of 85.7% in the external testing group. For model interpretation, we used the top 25 features identified by the random forest algorithm. The SHAP analysis revealed discernible distinctions between control participants and participants with HFpEF, thereby validating the diagnostic model's capacity to accurately identify participants with HFpEF.

CONCLUSIONS

This noninvasive and efficient model facilitates the identification of individuals with HFpEF, thereby promoting early detection, diagnosis, and treatment. Our research presents a clinically advanced diagnostic framework for HFpEF, validated using independent data sets and demonstrating significant potential to enhance patient care.

TRIAL REGISTRATION

China Clinical Trial Registry ChiCTR2300078855; https://www.chictr.org.cn/showproj.html?proj=207133.

Advancements in phasor-based FLIM: multi-component analysis and lifetime probes in biological imaging

Fluorescence lifetime imaging microscopy (FLIM) is a reliable method that achieves imaging by detecting fluorescence lifetimes within samples. Owing to its unique temporal characteristic, it can complement fluorescence intensity measurement. Technological and methodological advancements in FLIM have broadened its applications across various domains. The processing of fluorescence lifetime data is crucial for enhancing the speed and accuracy of imaging. Thus, various lifetime fitting algorithms have been developed to improve the imaging speed. The phasor analysis (PA) method is an approach for processing fluorescence lifetime data, capable of directly converting lifetime signals into visual graphics without fitting, which outperforms traditional approaches in speed. Furthermore, lifetime probes with distinct lifetimes are readily implemented for visualization and cluster analysis combined with PA, facilitating the prediction of specific biological states or functions. This review examines various lifetime probes employed in phasor-based FLIM and discusses their roles in the PA method. The methods for multi-component PA within complex biological environments were also described. Additionally, we focused on the advantages of the phasor vector rule and the unmixing of multi-component analysis based on PA. The integration of lifetime probes with phasor-based FLIM facilitates rapid and intuitive detection methods for analyzing complex biological environments.

A broadband hyperspectral image sensor with high spatio-temporal resolution

Hyperspectral imaging provides high-dimensional spatial-temporal-spectral information showing intrinsic matter characteristics1-5. Here we report an on-chip computational hyperspectral imaging framework with high spatial and temporal resolution. By integrating different broadband modulation materials on the image sensor chip, the target spectral information is non-uniformly and intrinsically coupled to each pixel with high light throughput. Using intelligent reconstruction algorithms, multi-channel images can be recovered from each frame, realizing real-time hyperspectral imaging. Following this framework, we fabricated a broadband visible-near-infrared (400-1,700 nm) hyperspectral image sensor using photolithography, with an average light throughput of 74.8% and 96 wavelength channels. The demonstrated resolution is 1,024 × 1,024 pixels at 124 fps. We demonstrated its wide applications, including chlorophyll and sugar quantification for intelligent agriculture, blood oxygen and water quality monitoring for human health, textile classification and apple bruise detection for industrial automation, and remote lunar detection for astronomy. The integrated hyperspectral image sensor weighs only tens of grams and can be assembled on various resource-limited platforms or equipped with off-the-shelf optical systems. The technique transforms the challenge of high-dimensional imaging from a high-cost manufacturing and cumbersome system to one that is solvable through on-chip compression and agile computation.

Deep learning-enhanced snapshot hyperspectral confocal microscopy imaging system

Laser-scanning confocal hyperspectral microscopy is a powerful technique to identify the different sample constituents and their spatial distribution in three-dimensional (3D). However, it suffers from low imaging speed because of the mechanical scanning methods. To overcome this challenge, we propose a snapshot hyperspectral confocal microscopy imaging system (SHCMS). It combined coded illumination microscopy based on a digital micromirror device (DMD) with a snapshot hyperspectral confocal neural network (SHCNet) to realize single-shot confocal hyperspectral imaging. With SHCMS, high-contrast 160-bands confocal hyperspectral images of potato tuber autofluorescence can be collected by only single-shot, which is almost 5 times improvement in the number of spectral channels than previously reported methods. Moreover, our approach can efficiently record hyperspectral volumetric imaging due to the optical sectioning capability. This fast high-resolution hyperspectral imaging method may pave the way for real-time highly multiplexed biological imaging.

Multidimensional quantitative characterization of basal cell carcinoma by spectral- and time-resolved two-photon microscopy

Basal cell carcinoma (BCC) is a common type of skin cancer. Conventional approaches to BCC diagnosis often involve invasive histological examinations that can distort or even destroy information derived from the biomolecules in the sample. Therefore, a non-invasive, label-free examination method for the clinical diagnosis of BCC represents a critical advance. This study combined spectral- and time-resolved two-photon microscopy with a spectral phasor to extract rich biochemical information describing macroscopic tumor morphology and microscopic tumor metabolism. The proposed optical imaging technique achieved the rapid and efficient separation of tumor structures in systematic research conducted on normal and BCC human skin tissues. The results demonstrate that a combination of multidimensional data (e.g., fluorescence intensity, spectrum, and lifetime) with a spectral phasor can accurately identify tumor boundaries and achieve rapid separation. This label-free, real-time, multidimensional imaging technique serves as a complement to the conventional tumor diagnostic toolbox and demonstrates significant potential for the early diagnosis of BCC and wider applications in intraoperative auxiliary imaging.

Information-Rich Multi-Functional OCT for Adult Zebrafish Intra- and Extracranial Imaging

The zebrafish serves as a valuable animal model for both intra- and extracranial research, particularly in relation to the brain and skull. To effectively investigate the development and regeneration of adult zebrafish, a versatile in vivo imaging technique capable of showing both intra- and extracranial conditions is essential. In this paper, we utilized a high-resolution multi-functional optical coherence tomography (OCT) to obtain rich intra- and extracranial imaging outcomes of adult zebrafish, encompassing pigmentation distribution, tissue-specific information, cranial vascular imaging, and the monitoring of traumatic brain injury (TBI). Notably, it is the first that the channels through the zebrafish cranial suture, which may have a crucial function in maintaining the patency of the cranial sutures, have been observed. Rich imaging results demonstrated that a high-resolution multi-functional OCT system can provide a wealth of novel and interpretable biological information for intra- and extracranial studies of adult zebrafish.

A single-shot hyperspectral phasor camera for fast, multi-color fluorescence microscopy

Hyperspectral fluorescence imaging improves multiplexed observations of biological samples by utilizing multiple color channels across the spectral range to compensate for spectral overlap between labels. Typically, spectral resolution comes at a cost of decreased detection efficiency, which both hampers imaging speed and increases photo-toxicity to the samples. Here, we present a high-speed, high-efficiency snapshot spectral acquisition method, based on optical compression of the fluorescence spectra via Fourier transform, that overcomes the challenges of discrete spectral sampling: single-shot hyperspectral phasor camera (SHy-Cam). SHy-Cam captures fluorescence spatial and spectral information in a single exposure with a standard scientific CMOS camera, with photon efficiency of over 80%, easily and with acquisition rates exceeding 30 datasets per second, making it a powerful tool for multi-color in vivo imaging. Its simple design, using readily available optical components, and its easy integration provide a low-cost solution for multi-color fluorescence imaging with increased efficiency and speed.

Neural Networks for Hyperspectral Imaging of Historical Paintings: A Practical Review

Hyperspectral imaging (HSI) has become widely used in cultural heritage (CH). This very efficient method for artwork analysis is connected with the generation of large amounts of spectral data. The effective processing of such heavy spectral datasets remains an active research area. Along with the firmly established statistical and multivariate analysis methods, neural networks (NNs) represent a promising alternative in the field of CH. Over the last five years, the application of NNs for pigment identification and classification based on HSI datasets has drastically expanded due to the flexibility of the types of data they can process, and their superior ability to extract structures contained in the raw spectral data. This review provides an exhaustive analysis of the literature related to NNs applied for HSI data in the CH field. We outline the existing data processing workflows and propose a comprehensive comparison of the applications and limitations of the various input dataset preparation methods and NN architectures. By leveraging NN strategies in CH, the paper contributes to a wider and more systematic application of this novel data analysis method.

SPLIT-PIN software enabling confocal and super-resolution imaging with a virtually closed pinhole

In point-scanning microscopy, optical sectioning is achieved using a small aperture placed in front of the detector, i.e. the detection pinhole, which rejects the out-of-focus background. The maximum level of optical sectioning is theoretically obtained for the minimum size of the pinhole aperture, but this is normally prevented by the dramatic reduction of the detected signal when the pinhole is closed, leading to a compromise between axial resolution and signal-to-noise ratio. We have recently demonstrated that, instead of closing the pinhole, one can reach a similar level of optical sectioning by tuning the pinhole size in a confocal microscope and by analyzing the resulting image series. The method, consisting in the application of the separation of photons by lifetime tuning (SPLIT) algorithm to series of images acquired with tunable pinhole size, is called SPLIT-pinhole (SPLIT-PIN). Here, we share and describe a SPLIT-PIN software for the processing of series of images acquired at tunable pinhole size, which generates images with reduced out-of-focus background. The software can be used on series of at least two images acquired on available commercial microscopes equipped with a tunable pinhole, including confocal and stimulated emission depletion (STED) microscopes. We demonstrate applicability on different types of imaging modalities: (1) confocal imaging of DNA in a non-adherent cell line; (2) removal of out-of-focus background in super-resolved STED microscopy; (3) imaging of live intestinal organoids stained with a membrane dye.

A phasor-based approach to improve optical sectioning in any confocal microscope with a tunable pinhole

Confocal fluorescence microscopy is a well‐established imaging technique capable of generating thin optical sections of biological specimens. Optical sectioning in confocal microscopy is mainly determined by the size of the pinhole, a small aperture placed in front of a point detector. In principle, imaging with a closed pinhole provides the highest degree of optical sectioning. In practice, the dramatic reduction of signal‐to‐noise ratio (SNR) at smaller pinhole sizes makes challenging the use of pinhole sizes significantly smaller than 1 Airy Unit (AU). Here, we introduce a simple method to “virtually” perform confocal imaging at smaller pinhole sizes without the dramatic reduction of SNR. The method is based on the sequential acquisition of multiple confocal images acquired at different pinhole aperture sizes and image processing based on a phasor analysis. The implementation is conceptually similar to separation of photons by lifetime tuning (SPLIT), a technique that exploits the phasor analysis to achieve super‐resolution, and for this reason we call this method SPLIT‐pinhole (SPLIT‐PIN). We show with simulated data that the SPLIT‐PIN image can provide improved optical sectioning (i.e., virtually smaller pinhole size) but better SNR with respect to an image obtained with closed pinhole. For instance, two images acquired at 2 and 1 AU can be combined to obtain a SPLIT‐PIN image with a virtual pinhole size of 0.2 AU but with better SNR. As an example of application to biological imaging, we show that SPLIT‐PIN improves confocal imaging of the apical membrane in an in vitro model of the intestinal epithelium.

Live-cell fluorescence spectral imaging as a data science challenge

Live-cell fluorescence spectral imaging is an evolving modality of microscopy that uses specific properties of fluorophores, such as excitation or emission spectra, to detect multiple molecules and structures in intact cells. The main challenge of analyzing live-cell fluorescence spectral imaging data is the precise quantification of fluorescent molecules despite the weak signals and high noise found when imaging living cells under non-phototoxic conditions. Beyond the optimization of fluorophores and microscopy setups, quantifying multiple fluorophores requires algorithms that separate or unmix the contributions of the numerous fluorescent signals recorded at the single pixel level. This review aims to provide both the experimental scientist and the data analyst with a straightforward description of the evolution of spectral unmixing algorithms for fluorescence live-cell imaging. We show how the initial systems of linear equations used to determine the concentration of fluorophores in a pixel progressively evolved into matrix factorization, clustering, and deep learning approaches. We outline potential future trends on combining fluorescence spectral imaging with label-free detection methods, fluorescence lifetime imaging, and deep learning image analysis.

Spatial transcriptomics using combinatorial fluorescence spectral and lifetime encoding, imaging and analysis

Multiplexed mRNA profiling in the spatial context provides new information enabling basic research and clinical applications. Unfortunately, existing spatial transcriptomics methods are limited due to either low multiplexing or complexity. Here, we introduce a spatialomics technology, termed Multi Omic Single-scan Assay with Integrated Combinatorial Analysis (MOSAICA), that integrates in situ labeling of mRNA and protein markers in cells or tissues with combinatorial fluorescence spectral and lifetime encoded probes, spectral and time-resolved fluorescence imaging, and machine learning-based decoding. We demonstrate MOSAICA's multiplexing scalability in detecting 10-plex targets in fixed colorectal cancer cells using combinatorial labeling of five fluorophores with facile error-detection and removal of autofluorescence. MOSAICA's analysis is strongly correlated with sequencing data (Pearson's r = 0.96) and was further benchmarked using RNAscopeTM and LGC StellarisTM. We further apply MOSAICA for multiplexed analysis of clinical melanoma Formalin-Fixed Paraffin-Embedded (FFPE) tissues. We finally demonstrate simultaneous co-detection of protein and mRNA in cancer cells.

Hyperspectral Mapping for the Detection of SARS-CoV-2 Using Nanomolecular Probes with Yoctomole Sensitivity

Efficient monitoring of SARS-CoV-2 outbreak requires the use of a sensitive and rapid diagnostic test. Although SARS-CoV-2 RNA can be detected by RT-qPCR, the molecular-level quantification of the viral load is still challenging, time-consuming, and labor-intensive. Here, we report an ultrasensitive hyperspectral sensor (HyperSENSE) based on hafnium nanoparticles (HfNPs) for specific detection of COVID-19 causative virus, SARS-CoV-2. Density functional theoretical calculations reveal that HfNPs exhibit higher changes in their absorption wavelength and light scattering when bound to their target SARS-CoV-2 RNA sequence relative to the gold nanoparticles. The assay has a turnaround time of a few seconds and has a limit of detection in the yoctomolar range, which is 1 000 000-fold times higher than the currently available COVID-19 tests. We demonstrated in ∼100 COVID-19 clinical samples that the assay is highly sensitive and has a specificity of 100%. We also show that HyperSENSE can rapidly detect other viruses such as influenza A H1N1. The outstanding sensitivity indicates the potential of the current biosensor in detecting the prevailing presymptomatic and asymptomatic COVID-19 cases. Thus, integrating hyperspectral imaging with nanomaterials establishes a diagnostic platform for ultrasensitive detection of COVID-19 that can potentially be applied to any emerging infectious pathogen.

Rapid statistical discrimination of fluorescence images of T cell receptors on immobilizing surfaces with different coating conditions

The spatial organization of T cell receptors (TCRs) correlates with membrane-associated signal amplification, dispersion, and regulation during T cell activation. Despite its potential clinical importance, quantitative analysis of the spatial arrangement of TCRs from standard fluorescence images remains difficult. Here, we report Statistical Classification Analyses of Membrane Protein Images or SCAMPI as a technique capable of analyzing the spatial arrangement of TCRs on the plasma membrane of T cells. We leveraged medical image analysis techniques that utilize pixel-based values. We transformed grayscale pixel values from fluorescence images of TCRs into estimated model parameters of partial differential equations. The estimated model parameters enabled an accurate classification using linear discrimination techniques, including Fisher Linear Discriminant (FLD) and Logistic Regression (LR). In a proof-of-principle study, we modeled and discriminated images of fluorescently tagged TCRs from Jurkat T cells on uncoated cover glass surfaces (Null) or coated cover glass surfaces with either positively charged poly-L-lysine (PLL) or TCR cross-linking anti-CD3 antibodies (OKT3). Using 80 training images and 20 test images per class, our statistical technique achieved 85% discrimination accuracy for both OKT3 versus PLL and OKT3 versus Null conditions. The run time of image data download, model construction, and image discrimination was 21.89 s on a laptop computer, comprised of 20.43 s for image data download, 1.30 s on the FLD-SCAMPI analysis, and 0.16 s on the LR-SCAMPI analysis. SCAMPI represents an alternative approach to morphology-based qualifications for discriminating complex patterns of membrane proteins conditioned on a small sample size and fast runtime. The technique paves pathways to characterize various physiological and pathological conditions using the spatial organization of TCRs from patient T cells.

Phasor-based hyperspectral snapshot microscopy allows fast imaging of live, three-dimensional tissues for biomedical applications

Hyperspectral imaging is highly sought after in many fields including mineralogy and geology, environment and agriculture, astronomy and, importantly, biomedical imaging and biological fluorescence. We developed ultrafast phasor-based hyperspectral snapshot microscopy based on sine/cosine interference filters for biomedical imaging not feasible with conventional hyperspectral detection methods. Current approaches rely on slow spatial or spectral scanning limiting their application in living biological tissues, while faster snapshot methods such as image mapping spectrometry and multispectral interferometry are limited in spatial and/or spectral resolution, are computationally demanding, and imaging devices are very expensive to manufacture. Leveraging light sheet microscopy, phasor-based hyperspectral snapshot microscopy improved imaging speed 10-100 fold which, combined with minimal light exposure and high detection efficiency, enabled hyperspectral metabolic imaging of live, three-dimensional mouse tissues not feasible with other methods. As a fit-free method that does not require any a priori information often unavailable in complex and evolving biological systems, the rule of linear combinations of the phasor could spectrally resolve subtle differences between cell types in the developing zebrafish retina and spectrally separate and track multiple organelles in 3D cultured cells over time. The sine/cosine snapshot method is adaptable to any microscope or imaging device thus making hyperspectral imaging and fit-free analysis based on linear combinations broadly available to researchers and the public.

Phasor-based image segmentation: machine learning clustering techniques

The phasor approach is a well-established method for data visualization and image analysis in spectral and lifetime fluorescence microscopy. Nevertheless, it is typically applied in a user-dependent manner by manually selecting regions of interest on the phasor space to find distinct regions in the fluorescence images. In this paper we present our work on using machine learning clustering techniques to establish an unsupervised and automatic method that can be used for identifying populations of fluorescent species in spectral and lifetime imaging. We demonstrate our method using both synthetic data, created by sampling photon arrival times and plotting the distributions on the phasor plot, and real live cells samples, by staining cellular organelles with a selection of commercial probes.