Compressive hyperspectral time-resolved wide-field fluorescence lifetime imaging

New approaches for low phototoxicity imaging of living cells and tissues

Fluorescence microscopy is a powerful tool used in scientific and medical research, but it is inextricably linked to phototoxicity. Neglecting phototoxicity can lead to erroneous or inconclusive results. Recently, several reports have addressed this issue, but it is still underestimated by many researchers, even though it can lead to cell death. Phototoxicity can be reduced by appropriate microscopic techniques and carefully designed experiments. This review focuses on recent strategies to reduce phototoxicity in microscopic imaging of living cells and tissues. We describe digital image processing and new hardware solutions. We point out new modifications of microscopy methods and hope that this review will interest microscopy hardware engineers. Our aim is to underscore the challenges and potential solutions integral to the design of microscopy systems. Simultaneously, we intend to engage biologists, offering insight into the latest technological advancements in imaging that can enhance their understanding and practice.

Time-gated single-pixel imaging of Cherenkov emission from a medical linear accelerator

Cherenkov imaging is an ideal tool for real-time in vivo verification of a radiation therapy dose. Given that radiation is pulsed from a medical linear accelerator (LINAC) together with weak Cherenkov emissions, time-gated high-sensitivity imaging is required for robust measurements. Instead of using an expensive camera system with limited efficiency of detection in each pixel, a single-pixel imaging (SPI) approach that maintains promising sensitivity over the entire spectral band could be used to provide a low-cost and viable alternative. A prototype SPI system was developed and demonstrated here in Cherenkov imaging of LINAC dose delivery to a water tank. Validation experiments were performed using four regular fields and an intensity-modulated radiotherapy (IMRT) delivery plan. The Cherenkov image-based projection percent depth dose curves (pPDDs) were compared to pPDDs simulated by the treatment planning system (TPS), with an overall average error of 0.48, 0.42, 0.65, and 1.08% for the 3, 5, 7, and 9 cm square beams, respectively. The composite image of the IMRT plan achieved a 85.9% pass rate using 3%/3 mm gamma index criteria, in comparing Cherenkov intensity and TPS dose. This study validates the feasibility of applying SPI to the Cherenkov imaging of radiotherapy dose for the first time to our knowledge.

Wide-field illumination diffuse optical tomography within a framework of single-pixel time-domain spatial frequency domain imaging

We present a wide-field illumination time-domain (TD) diffusion optical tomography (DOT) for three-dimensional (3-D) reconstruction within a shallow region under the illuminated surface of the turbid medium. The methodological foundation is laid on the single-pixel spatial frequency domain (SFD) imaging that facilitates the adoption of the well-established time-correlated single-photon counting (TCSPC)-based TD detection and generalized pulse spectrum techniques (GPST)-based reconstruction. To ameliorate the defects of the conventional diffusion equation (DE) in the forward modeling of TD-SFD-DOT, mainly the low accuracy in the near-field region and in profiling early-photon migration, we propose a modified model employing the time-dependent δ-P1 approximation and verify its improved accuracy in comparison with both the Monte Carlo and DE-based ones. For a simplified inversion process, a modified GPST approach is extended to TD-SFD-DOT that enables the effective separation of the absorption and scattering coefficients using a steady-state equivalent strategy. Furthermore, we set up a single-pixel TD-SFD-DOT system that employs the TCSPC-based TD detection in the SFD imaging framework. For assessments of the reconstruction approach and the system performance, phantom experiments are performed for a series of scenarios. The results show the effectiveness of the proposed methodology for rapid 3-D reconstruction of the absorption and scattering coefficients within a depth range of about 5 mean free pathlengths.

Applications of machine learning in time-domain fluorescence lifetime imaging: a review

Many medical imaging modalities have benefited from recent advances in Machine Learning (ML), specifically in deep learning, such as neural networks. Computers can be trained to investigate and enhance medical imaging methods without using valuable human resources. In recent years, Fluorescence Lifetime Imaging (FLIm) has received increasing attention from the ML community. FLIm goes beyond conventional spectral imaging, providing additional lifetime information, and could lead to optical histopathology supporting real-time diagnostics. However, most current studies do not use the full potential of machine/deep learning models. As a developing image modality, FLIm data are not easily obtainable, which, coupled with an absence of standardisation, is pushing back the research to develop models which could advance automated diagnosis and help promote FLIm. In this paper, we describe recent developments that improve FLIm image quality, specifically time-domain systems, and we summarise sensing, signal-to-noise analysis and the advances in registration and low-level tracking. We review the two main applications of ML for FLIm: lifetime estimation and image analysis through classification and segmentation. We suggest a course of action to improve the quality of ML studies applied to FLIm. Our final goal is to promote FLIm and attract more ML practitioners to explore the potential of lifetime imaging.

Multiplexed near infrared fluorescence lifetime imaging in turbid media

Significance

Fluorescence lifetime imaging (FLI) plays a pivotal role in enhancing our understanding of biological systems, providing a valuable tool for non-invasive exploration of biomolecular and cellular dynamics, both in vitro and in vivo. Its ability to selectively target and multiplex various entities, alongside heightened sensitivity and specificity, offers rapid and cost-effective insights.

Aim

Our aim is to investigate the multiplexing capabilities of near-infrared (NIR) FLI within a scattering medium that mimics biological tissues. We strive to develop a comprehensive understanding of FLI's potential for multiplexing diverse targets within a complex, tissue-like environment.

Approach

We introduce an innovative Monte Carlo (MC) simulation approach that accurately describes the scattering behavior of fluorescent photons within turbid media. Applying phasor analyses, we enable the multiplexing of distinct targets within a single FLI image. Leveraging the state-of-the-art single-photon avalanche diode (SPAD) time-gated camera, SPAD512S, we conduct experimental wide-field FLI in the NIR regime.

Results

Our study demonstrates the successful multiplexing of dual targets within a single FLI image, reaching a depth of 1 cm within tissue-like phantoms. Through our novel MC simulation approach and phasor analyses, we showcase the effectiveness of our methodology in overcoming the challenges posed by scattering media.

Conclusions

This research underscores the potential of NIR FLI for multiplexing applications in complex biological environments. By combining advanced simulation techniques with cutting-edge experimental tools, we introduce significant results in the non-invasive exploration of biomolecular dynamics, to advance the field of FLI research.

Nanocrystals for Deep-Tissue In Vivo Luminescence Imaging in the Near-Infrared Region

In vivo imaging technologies have emerged as a powerful tool for both fundamental research and clinical practice. In particular, luminescence imaging in the tissue-transparent near-infrared (NIR, 700-1700 nm) region offers tremendous potential for visualizing biological architectures and pathophysiological events in living subjects with deep tissue penetration and high imaging contrast owing to the reduced light-tissue interactions of absorption, scattering, and autofluorescence. The distinctive quantum effects of nanocrystals have been harnessed to achieve exceptional photophysical properties, establishing them as a promising category of luminescent probes. In this comprehensive review, the interactions between light and biological tissues, as well as the advantages of NIR light for in vivo luminescence imaging, are initially elaborated. Subsequently, we focus on achieving deep tissue penetration and improved imaging contrast by optimizing the performance of nanocrystal fluorophores. The ingenious design strategies of NIR nanocrystal probes are discussed, along with their respective biomedical applications in versatile in vivo luminescence imaging modalities. Finally, thought-provoking reflections on the challenges and prospects for future clinical translation of nanocrystal-based in vivo luminescence imaging in the NIR region are wisely provided.

Fast data fitting scheme for compressive multispectral fluorescence lifetime imaging

A single-pixel camera combined with compressive sensing techniques is a promising fluorescence microscope scheme for acquiring a multidimensional dataset (space, spectrum, and lifetime) and for reducing the measurement time with respect to conventional microscope schemes. However, upon completing the acquisition, a computational step is necessary for image reconstruction and data analysis, which can be time-consuming, potentially canceling out the beneficial effect of compressive sensing. In this work, we propose and experimentally validate a fast-fit workflow based on global analysis and multiple linear fits, which significantly reduces the computation time from tens of minutes to less than 1 s. Moreover, as the method is interlaced with the measurement flow, it can be applied in parallel with the acquisitions.

Macroscopic temporally and spectrally resolved fluorescence imaging enhanced by laser-wavelength multiplexing

We present a laser scanning system for macroscopic samples that records fully resolved decay curves in individual pixels, resolves the images in 16 wavelength channels, and records simultaneously at several laser wavelengths. By using confocal detection, the system delivers images that are virtually free of lateral scattering and out-of-focus haze. Image formats can be up to 256 × 256 pixels and up to 1024 time channels. We demonstrate the performance of the system both on model experiments with fluorescent micro-beads and on the tumor model in the living mice.

Optical single-pixel volumetric imaging by three-dimensional light-field illumination

Three-dimensional single-pixel imaging (3D SPI) has become an attractive imaging modality for both biomedical research and optical sensing. 3D-SPI techniques generally depend on time-of-flight or stereovision principle to extract depth information from backscattered light. However, existing implementations for these two optical schemes are limited to surface mapping of 3D objects at depth resolutions, at best, at the millimeter level. Here, we report 3D light-field illumination single-pixel microscopy (3D-LFI-SPM) that enables volumetric imaging of microscopic objects with a near-diffraction-limit 3D optical resolution. Aimed at 3D space reconstruction, 3D-LFI-SPM optically samples the 3D Fourier spectrum by combining 3D structured light-field illumination with single-element intensity detection. We build a 3D-LFI-SPM prototype that provides an imaging volume of ∼390 × 390 × 3,800 μm3 and achieves 2.7-μm lateral resolution and better than 37-μm axial resolution. Its capability of 3D visualization of label-free optical absorption contrast is demonstrated by imaging single algal cells in vivo. Our approach opens broad perspectives for 3D SPI with potential applications in various fields, such as biomedical functional imaging.

Single photon single pixel imaging into thick scattering medium

Imaging into thick scattering medium is a long-standing challenge. Beyond the quasi-ballistic regime, multiple scattering scrambles the spatiotemporal information of incident/emitted light, making canonical imaging based on light focusing nearly impossible. Diffusion optical tomography (DOT) is one of the most popular approach to look inside scattering medium, but quantitatively inverting the diffusion equation is ill-posed, and prior information of the medium is typically necessary, which is nontrivial to obtain. Here, we show theoretically and experimentally that, by synergizing the one-way light scattering characteristic of single pixel imaging with ultrasensitive single photon detection and a metric-guided image reconstruction, single photon single pixel imaging can serve as a simple and powerful alternative to DOT for imaging into thick scattering medium without prior knowledge or inverting the diffusion equation. We demonstrated an image resolution of 12 mm inside a 60 mm thick (∼ 78 mean free paths) scattering medium.

Deep learning-based fusion of widefield diffuse optical tomography and micro-CT structural priors for accurate 3D reconstructions

Widefield illumination and detection strategies leveraging structured light have enabled fast and robust probing of tissue properties over large surface areas and volumes. However, when applied to diffuse optical tomography (DOT) applications, they still require a time-consuming and expert-centric solving of an ill-posed inverse problem. Deep learning (DL) models have been recently proposed to facilitate this challenging step. Herein, we expand on a previously reported deep neural network (DNN) -based architecture (modified AUTOMAP - ModAM) for accurate and fast reconstructions of the absorption coefficient in 3D DOT based on a structured light illumination and detection scheme. Furthermore, we evaluate the improved performances when incorporating a micro-CT structural prior in the DNN-based workflow, named Z-AUTOMAP. This Z-AUTOMAP significantly improves the widefield imaging process's spatial resolution, especially in the transverse direction. The reported DL-based strategies are validated both in silico and in experimental phantom studies using spectral micro-CT priors. Overall, this is the first successful demonstration of micro-CT and DOT fusion using deep learning, greatly enhancing the prospect of rapid data-integration strategies, often demanded in challenging pre-clinical scenarios.

Nanosecond compressive fluorescence lifetime microscopy imaging via the RATS method with a direct reconstruction of lifetime maps

The RAndom Temporal Signals (RATS) method has proven to be a useful and versatile method for measuring photoluminescence (PL) dynamics and fluorescence lifetime imaging (FLIM). Here, we present two fundamental development steps in the method. First, we demonstrate that by using random digital laser modulation in RATS, it is possible to implement the measurement of PL dynamics with temporal resolution in units of nanoseconds. Secondly, we propose an alternative approach to evaluating FLIM measurements based on a single-pixel camera experiment. In contrast to the standard evaluation, which requires a lengthy iterative reconstruction of PL maps for each timepoint, here we use a limited set of predetermined PL lifetimes and calculate the amplitude maps corresponding to each lifetime. The alternative approach significantly saves post-processing time and, in addition, in a system with noise present, it shows better stability in terms of the accuracy of the FLIM spectrogram. Besides simulations that confirmed the functionality of the extension, we implemented the new advancements into a microscope optical setup for mapping PL dynamics on the micrometer scale. The presented principles were also verified experimentally by mapping a LuAG:Ce crystal surface.

Programmed Control of Fluorescence Blinking Patterns based on Electron Transfer in DNA

Fluorescence imaging uses changes in the fluorescence intensity and emission wavelength to analyze multiple targets simultaneously. To increase the number of targets that can be identified simultaneously, fluorescence blinking can be used as an additional parameter. To understand and eventually control blinking, we used DNA as a platform to elucidate the processes of electron transfer (ET) leading to blinking, down to the rate constants. With a fixed ET distance, various blinking patterns were observed depending on the DNA sequence between the donor and acceptor units of the DNA platform. The blinking pattern was successfully described with a combination of ET rate constants. Therefore, molecules with various blinking patterns can be developed by tuning ET. It is expected that the number of targets that can be analyzed simultaneously will increase by the power of the number of blinking patterns.

Tunable image projection spectrometry

We present tunable image projection spectrometry (TIPS), a Fourier-domain line-scan spectral imager with a tunable compression ratio. Compared to state-of-the-art spatial-domain pushbroom hyperspectral cameras, TIPS requires much fewer measurements and provides a higher light throughput. Using a rotating Dove prism and a cylindrical field lens, TIPS scans an input scene in the Fourier domain and captures a subset of multi-angled one-dimensional (1D) en face projections of the input scene, allowing a tailored data compression ratio for a given scene. We demonstrate the spectral imaging capability of TIPS with a hematoxylin and eosin (H&E) stained pathology slide. Moreover, we showed the spectral information obtained can be further converted to depths when combining TIPS with a low-coherence full-field spectral-domain interferometer.

Computational macroscopic lifetime imaging and concentration unmixing of autofluorescence

Single-pixel computational imaging can leverage highly sensitive detectors that concurrently acquire data across spectral and temporal domains. For molecular imaging, such methodology enables to collect rich intensity and lifetime multiplexed fluorescence datasets. Herein we report on the application of a single-pixel structured light-based platform for macroscopic imaging of tissue autofluorescence. The super-continuum visible excitation and hyperspectral single-pixel detection allow for parallel characterization of autofluorescence intensity and lifetime. Furthermore, we exploit a deep learning based data processing pipeline, to perform autofluorescence unmixing while yielding the autofluorophores' concentrations. The full scheme (setup and processing) is validated in silico and in vitro with clinically relevant autofluorophores flavin adenine dinucleotide, riboflavin, and protoporphyrin. The presented results demonstrate the potential of the methodology for macroscopically quantifying the intensity and lifetime of autofluorophores, with higher specificity for cases of mixed emissions, which are ubiquitous in autofluorescence and multiplexed in vivo imaging.

Polarity-Sensitive and Membrane-Specific Probe Quantitatively Monitoring Ferroptosis through Fluorescence Lifetime Imaging

As a new form of regulated cell death, ferroptosis is closely related to various diseases. To interpret this biological behavior and monitor related pathological processes, it is necessary to develop appropriate detection strategies and tools. Considering that ferroptosis is featured with remarkable lipid peroxidation of various cell membranes, it is logical to detect membranes' structural and environmental changes for the direct assessment of ferroptosis. For this sake, we designed novel polarity-sensitive fluorescent probes Mem-C₁C₁₈ and Mem-C₁₈C₁₈, which have superior plasma membrane anchorage, high brightness, and sensitive responses to environmental polarity by changing their fluorescence lifetimes. Mem-C₁C₁₈ with much less tendency to aggregate than Mem-C₁₈C₁₈ outperformed the latter in high resolution fluorescence labeling of artificial vesicle membranes and plasma membranes of live cells. Thus, Mem-C₁C₁₈ was selected to monitor plasma membranes damaged along ferroptosis process for the first time, in combination with the technique of fluorescence lifetime imaging (FLIM). After treating HeLa cells with Erastin, a typical ferroptosis inducer, the mean fluorescence lifetime of Mem-C₁C₁₈ displayed a considerable increase from 3.00 to 4.93 ns, with a 64% increase (corresponding to the polarity parameter Δf increased from 0.213 to 0.232). Therefore, our idea to utilize a probe to quantitate the changes in polarity of plasma membranes proves to be an effective method in the evaluation of the ferroptosis process.

Secondary Complementary Balancing Compressive Imaging with a Free-Space Balanced Amplified Photodetector

Single-pixel imaging (SPI) has attracted widespread attention because it generally uses a non-pixelated photodetector and a digital micromirror device (DMD) to acquire the object image. Since the modulated patterns seen from two reflection directions of the DMD are naturally complementary, one can apply complementary balanced measurements to greatly improve the measurement signal-to-noise ratio and reconstruction quality. However, the balance between two reflection arms significantly determines the quality of differential measurements. In this work, we propose and demonstrate a simple secondary complementary balancing mechanism to minimize the impact of the imbalance on the imaging system. In our SPI setup, we used a silicon free-space balanced amplified photodetector with 5 mm active diameter which could directly output the difference between two optical input signals in two reflection arms. Both simulation and experimental results have demonstrated that the use of secondary complementary balancing can result in a better cancellation of direct current components of measurements, and can acquire an image quality slightly better than that of single-arm single-pixel complementary measurement scheme (which is free from the trouble of optical imbalance) and over 20 times better than that of double-arm dual-pixel complementary measurement scheme under optical imbalance conditions.

An autofocus algorithm considering wavelength changes for large scale microscopic hyperspectral pathological imaging system

Microscopic hyperspectral imaging technology has been widely used to acquire pathological information of tissue sections. Autofocus is one of the most important steps in microscopic hyperspectral imaging systems to capture large scale or even whole slide images of pathological slides with high quality and high speed. However, there are quite few autofocus algorithm put forward for the microscopic hyperspectral imaging system. Therefore, this article proposes a Laplace operator based autofocus algorithm for microscopic hyperspectral imaging system which takes the influence of wavelength changes into consideration. Through the proposed algorithm, the focal length for each wavelength can be adjusted automatically to ensure that each single band image can be autofocused precisely with adaptive image sharpness evaluation method. In addition, to increase the capture speed, the relationship of wavelength and focal length is derived and the focal offsets among different single band images are calculated for pre-focusing. We have employed the proposed method on our own datasets and the experimental results show that it can capture large-scale microscopic hyperspectral pathology images with high precise.

Noise effect on 2D photoluminescence decay analysis using the RATS method in a single-pixel camera configuration

Using a random temporal signal for sample excitation (RATS method) is a new, capable approach to measuring photoluminescence (PL) dynamics. The method can be used in single-point measurement (0D), but also it can be converted to PL decay imaging (2D) using a single-pixel camera configuration. In both cases, the reconstruction of the PL decay and PL snapshot is affected by ubiquitous noise. This article provides a detailed analysis of the noise effect on the RATS method and possible strategies for its suppression. We carried out an extensive set of simulations focusing on the effect of noise introduced through the random excitation signal and the corresponding PL waveform. We show that the PL signal noise level is critical for the method. Furthermore, we analyze the role of acquisition time, where we demonstrate the need for a non-periodic excitation signal. We show that it is beneficial to increase the acquisition time and that increasing the number of measurements in the single-pixel camera configuration has a minimal effect above a certain threshold. Finally, we study the effect of a regularization parameter used in the deconvolution step, and we observe that there is an optimum value set by the noise present in the PL dataset. Our results provide a guideline for optimization of the RATS measurement, but we also study effects generally occurring in PL decay measurements methods relying on the deconvolution step.

Deep learning in macroscopic diffuse optical imaging

SIGNIFICANCE

Biomedical optics system design, image formation, and image analysis have primarily been guided by classical physical modeling and signal processing methodologies. Recently, however, deep learning (DL) has become a major paradigm in computational modeling and has demonstrated utility in numerous scientific domains and various forms of data analysis.

AIM

We aim to comprehensively review the use of DL applied to macroscopic diffuse optical imaging (DOI).

APPROACH

First, we provide a layman introduction to DL. Then, the review summarizes current DL work in some of the most active areas of this field, including optical properties retrieval, fluorescence lifetime imaging, and diffuse optical tomography.

RESULTS

The advantages of using DL for DOI versus conventional inverse solvers cited in the literature reviewed herein are numerous. These include, among others, a decrease in analysis time (often by many orders of magnitude), increased quantitative reconstruction quality, robustness to noise, and the unique capability to learn complex end-to-end relationships.

CONCLUSIONS

The heavily validated capability of DL's use across a wide range of complex inverse solving methodologies has enormous potential to bring novel DOI modalities, otherwise deemed impractical for clinical translation, to the patient's bedside.

Compressed sensing in fluorescence microscopy

Compressed sensing (CS) is a signal processing approach that solves ill-posed inverse problems, from under-sampled data with respect to the Nyquist criterium. CS exploits sparsity constraints based on the knowledge of prior information, relative to the structure of the object in the spatial or other domains. It is commonly used in image and video compression as well as in scientific and medical applications, including computed tomography and magnetic resonance imaging. In the field of fluorescence microscopy, it has been demonstrated to be valuable for fast and high-resolution imaging, from single-molecule localization, super-resolution to light-sheet microscopy. Furthermore, CS has found remarkable applications in the field of mesoscopic imaging, facilitating the study of small animals' organs and entire organisms. This review article illustrates the working principles of CS, its implementations in optical imaging and discusses several relevant uses of CS in the field of fluorescence imaging from super-resolution microscopy to mesoscopy.

Macroscopic Fluorescence Lifetime Imaging for Monitoring of Drug-Target Engagement

Precision medicine promises to improve therapeutic efficacy while reducing adverse effects, especially in oncology. However, despite great progresses in recent years, precision medicine for cancer treatment is not always part of routine care. Indeed, the ability to specifically tailor therapies to distinct patient profiles requires still significant improvements in targeted therapy development as well as decreases in drug treatment failures. In this regard, preclinical animal research is fundamental to advance our understanding of tumor biology, and diagnostic and therapeutic response. Most importantly, the ability to measure drug-target engagement accurately in live and intact animals is critical in guiding the development and optimization of targeted therapy. However, a major limitation of preclinical molecular imaging modalities is their lack of capability to directly and quantitatively discriminate between drug accumulation and drug-target engagement at the pathological site. Recently, we have developed Macroscopic Fluorescence Lifetime Imaging (MFLI) as a unique feature of optical imaging to quantitate in vivo drug-target engagement. MFLI quantitatively reports on nanoscale interactions via lifetime-sensing of Förster Resonance Energy Transfer (FRET) in live, intact animals. Hence, MFLI FRET acts as a direct reporter of receptor dimerization and target engagement via the measurement of the fraction of labeled-donor entity undergoing binding to its respective receptor. MFLI is expected to greatly impact preclinical imaging and also adjacent fields such as image-guided surgery and drug development.

Single-Detector Spectrometer Using a Superconducting Nanowire

Designing a spectrometer without the need for wavelength multiplexing optics can effectively reduce the complexity and physical footprint. On the basis of the computational spectroscopic strategy and combining a broadband-responsive dynamic detector, we successfully demonstrate an optics-free single-detector spectrometer that maps the tunable quantum efficiency of a superconducting nanowire into a matrix to build a solvable mathematical equation. Such a spectrometer can realize a broadband spectral responsivity ranging from 660 to 1900 nm. The spectral resolution at the telecom is sub-10 nm, exceeding the energy resolving capacity of existing infrared single-photon detectors. Meanwhile, benefiting from the optics-free setup, precise time-of-flight measurements can be simultaneously achieved. We have demonstrated a spectral LiDAR with eight spectral channels. This spectrometer scheme paves the way for applying superconducting nanowire detectors in multifunctional spectroscopy and represents a conceptual advancement for on-chip spectroscopy and spectral imaging.

3D denoised completion network for deep single-pixel reconstruction of hyperspectral images

Single-pixel imaging acquires an image by measuring its coefficients in a transform domain, thanks to a spatial light modulator. However, as measurements are sequential, only a few coefficients can be measured in the real-time applications. Therefore, single-pixel reconstruction is usually an underdetermined inverse problem that requires regularization to obtain an appropriate solution. Combined with a spectral detector, the concept of single-pixel imaging allows for hyperspectral imaging. While each channel can be reconstructed independently, we propose to exploit the spectral redundancy between channels to regularize the reconstruction problem. In particular, we introduce a denoised completion network that includes 3D convolution filters. Contrary to black-box approaches, our network combines the classical Tikhonov theory with the deep learning methodology, leading to an explainable network. Considering both simulated and experimental data, we demonstrate that the proposed approach yields hyperspectral images with higher quantitative metrics than the approaches developed for grayscale images.

Full spectrum fluorescence lifetime imaging with 0.5 nm spectral and 50 ps temporal resolution

The use of optical techniques to interrogate wide ranging samples from semiconductors to biological tissue for rapid analysis and diagnostics has gained wide adoption over the past decades. The desire to collect ever more spatially, spectrally and temporally detailed optical signatures for sample characterization has specifically driven a sharp rise in new optical microscopy technologies. Here we present a high-speed optical scanning microscope capable of capturing time resolved images across 512 spectral and 32 time channels in a single acquisition with the potential for ~0.2 frames per second (256 × 256 image pixels). Each pixel in the resulting images contains a detailed data cube for the study of diverse time resolved light driven phenomena. This is enabled by integration of system control electronics and on-chip processing which overcomes the challenges presented by high data volume and low imaging speed, often bottlenecks in previous systems.

High data volumes from multidimensional imaging techniques can lead to slow collection and processing times. Here, the authors implement multispectral fluorescence lifetime imaging microscopy (FLIM) that uses time-correlated photon counting technology to reach simultaneously high imaging rates combined with high spectral and temporal resolution.

Compressed sensing time-resolved spectrometer for quantification of light absorbers in turbid media

Time-resolved (TR) spectroscopy is well-suited to address the challenges of quantifying light absorbers in highly scattering media such as living tissue; however, current TR spectrometers are either based on expensive array detectors or rely on wavelength scanning. Here, we introduce a TR spectrometer architecture based on compressed sensing (CS) and time-correlated single-photon counting. Using both CS and basis scanning, we demonstrate that-in homogeneous and two-layer tissue-mimicking phantoms made of Intralipid and Indocyanine Green-the CS method agrees with or outperforms uncompressed approaches. Further, we illustrate the superior depth sensitivity of TR spectroscopy and highlight the potential of the device to quantify absorption changes in deeper (>1 cm) tissue layers.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

A Polarity-Sensitive Ratiometric Fluorescence Probe for Monitoring Changes in Lipid Droplets and Nucleus during Ferroptosis

Ferroptosis regulates cell death through reactive oxygen species (ROS)-associated lipid peroxide accumulation, which is expected to affect the structure and polarity of lipid droplets (LDs), but with no clear evidence. Herein, we report the first example of an LD/nucleus dual-targeted ratiometric fluorescent probe, CQPP, for monitoring polarity changes in the cellular microenvironment. Due to the donor-acceptor structure of CQPP, it offers ratiometric fluorescence emission and fluorescence lifetime signals that reflect polarity variations. Using nucleus imaging as a reference, CQPP was applied to report the increase in LD polarity and the homogenization of polarity between LDs and cytoplasm in the ferroptosis model. This LD/nucleus dual-targeted fluorescent probe shows the great potential of using fluorescence imaging to study ferroptosis and ferroptosis-related diseases.

Single-pixel image reconstruction from experimental data using neural networks

Single-pixel cameras that measure image coefficients have various promising applications, in particular for hyper-spectral imaging. Here, we investigate deep neural networks that when fed with experimental data can output high-quality images in real time. Assuming that the measurements are corrupted by mixed Poisson-Gaussian noise, we propose to map the raw data from the measurement domain to the image domain based on a Tikhonov regularization. This step can be implemented as the first layer of a deep neural network, followed by any architecture of layers that acts in the image domain. We also describe a framework for training the network in the presence of noise. In particular, our approach includes an estimation of the image intensity and experimental parameters, together with a normalization scheme that allows varying noise levels to be handled during training and testing. Finally, we present results from simulations and experimental acquisitions with varying noise levels. Our approach yields images with improved peak signal-to-noise ratios, even for noise levels that were foreseen during the training of the networks, which makes the approach particularly suitable to deal with experimental data. Furthermore, while this approach focuses on single-pixel imaging, it can be adapted for other computational optics problems.

Luminescence lifetime imaging of three-dimensional biological objects

A major focus of current biological studies is to fill the knowledge gaps between cell, tissue and organism scales. To this end, a wide array of contemporary optical analytical tools enable multiparameter quantitative imaging of live and fixed cells, three-dimensional (3D) systems, tissues, organs and organisms in the context of their complex spatiotemporal biological and molecular features. In particular, the modalities of luminescence lifetime imaging, comprising fluorescence lifetime imaging (FLI) and phosphorescence lifetime imaging microscopy (PLIM), in synergy with Förster resonance energy transfer (FRET) assays, provide a wealth of information. On the application side, the luminescence lifetime of endogenous molecules inside cells and tissues, overexpressed fluorescent protein fusion biosensor constructs or probes delivered externally provide molecular insights at multiple scales into protein-protein interaction networks, cellular metabolism, dynamics of molecular oxygen and hypoxia, physiologically important ions, and other physical and physiological parameters. Luminescence lifetime imaging offers a unique window into the physiological and structural environment of cells and tissues, enabling a new level of functional and molecular analysis in addition to providing 3D spatially resolved and longitudinal measurements that can range from microscopic to macroscopic scale. We provide an overview of luminescence lifetime imaging and summarize key biological applications from cells and tissues to organisms.

Multispectral compressive fluorescence lifetime imaging microscopy with a SPAD array detector

Multispectral/hyperspectral fluorescence lifetime imaging microscopy (λFLIM) is a promising tool for studying functional and structural biological processes. The rich information content provided by a multidimensional dataset is often in contrast with the acquisition speed. In this work, we develop and experimentally demonstrate a wide-field λFLIM setup, based on a novel time-resolved 18×1 single-photon avalanche diode array detector working in a single-pixel camera scheme, which parallelizes the spectral detection, reducing measurement time. The proposed system, which implements a single-pixel camera with a compressive sensing scheme, represents an optimal microscopy framework towards the design of λFLIM setups.

Fluorescence lifetime imaging via spatio-temporal speckle patterns in a single-pixel camera configuration

Photoluminescence (PL) spectroscopy offers excellent methods for mapping the PL decay on the nanosecond time scale. However, capturing maps of emission dynamics on the microsecond timescale can be highly time-consuming. We present a new approach to fluorescence lifetime imaging (FLIM), which combines the concept of random temporal speckles excitation (RATS) with the concept of a single-pixel camera based on spatial speckles. The spatio-temporal speckle pattern makes it possible to map PL dynamics with unmatched simplicity. Moreover, the method can acquire all the data necessary to map PL decay on the microsecond timescale within minutes. We present proof-of-principle measurements for two samples and compare the reconstructed decays to the non-imaging measurements. Finally, we discuss the effect of the preprocessing routine and other factors on the reconstruction noise level. The presented method is suitable for lifetime imaging processes in several samples, including monitoring charge carrier dynamics in perovskites or monitoring solid-state luminophores with a long lifetime of PL.

Applications of compressive sensing in spatial frequency domain imaging

SIGNIFICANCE

Spatial frequency domain imaging (SFDI) is an imaging modality that projects spatially modulated light patterns to determine optical property maps for absorption and reduced scattering of biological tissue via a pixel-by-pixel data acquisition and analysis procedure. Compressive sensing (CS) is a signal processing methodology which aims to reproduce the original signal with a reduced number of measurements, addressing the pixel-wise nature of SFDI. These methodologies have been combined for complex heterogenous data in both the image detection and data analysis stage in a compressive sensing SFDI (cs-SFDI) approach, showing reduction in both the data acquisition and overall computational time.

AIM

Application of CS in SFDI data acquisition and image reconstruction significantly improves data collection and image recovery time without loss of quantitative accuracy.

APPROACH

cs-SFDI has been applied to an increased heterogenic sample from the AppSFDI data set (back of the hand), highlighting the increased number of CS measurements required as compared to simple phantoms to accurately obtain optical property maps. A novel application of CS to the parameter recovery stage of image analysis has also been developed and validated.

RESULTS

Dimensionality reduction has been demonstrated using the increased heterogenic sample at both the acquisition and analysis stages. A data reduction of 30% for the cs-SFDI and up to 80% for the parameter recover was achieved as compared to traditional SFDI, while maintaining an error of <10  %   for the recovered optical property maps.

CONCLUSION

The application of data reduction through CS demonstrates additional capabilities for multi- and hyperspectral SFDI, providing advanced optical and physiological property maps.

High compression deep learning based single-pixel hyperspectral macroscopic fluorescence lifetime imaging in vivo

Single pixel imaging frameworks facilitate the acquisition of high-dimensional optical data in biological applications with photon starved conditions. However, they are still limited to slow acquisition times and low pixel resolution. Herein, we propose a convolutional neural network for fluorescence lifetime imaging with compressed sensing at high compression (NetFLICS-CR), which enables in vivo applications at enhanced resolution, acquisition and processing speeds, without the need for experimental training datasets. NetFLICS-CR produces intensity and lifetime reconstructions at 128 × 128 pixel resolution over 16 spectral channels while using only up to 1% of the required measurements, therefore reducing acquisition times from ∼2.5 hours at 50% compression to ∼3 minutes at 99% compression. Its potential is demonstrated in silico, in vitro and for mice in vivo through the monitoring of receptor-ligand interactions in liver and bladder and further imaging of intracellular delivery of the clinical drug Trastuzumab to HER2-positive breast tumor xenografts. The data acquisition time and resolution improvement through NetFLICS-CR, facilitate the translation of single pixel macroscopic flurorescence lifetime imaging (SP-MFLI) for in vivo monitoring of lifetime properties and drug uptake.

Multiplexed non-invasive tumor imaging of glucose metabolism and receptor-ligand engagement using dark quencher FRET acceptor

Rationale: Following an ever-increased focus on personalized medicine, there is a continuing need to develop preclinical molecular imaging modalities to guide the development and optimization of targeted therapies. Near-Infrared (NIR) Macroscopic Fluorescence Lifetime Förster Resonance Energy Transfer (MFLI-FRET) imaging offers a unique method to robustly quantify receptor-ligand engagement in live intact animals, which is critical to assess the delivery efficacy of therapeutics. However, to date, non-invasive imaging approaches that can simultaneously measure cellular drug delivery efficacy and metabolic response are lacking. A major challenge for the implementation of concurrent optical and MFLI-FRET in vivo whole-body preclinical imaging is the spectral crowding and cross-contamination between fluorescent probes. Methods: We report on a strategy that relies on a dark quencher enabling simultaneous assessment of receptor-ligand engagement and tumor metabolism in intact live mice. Several optical imaging approaches, such as in vitro NIR FLI microscopy (FLIM) and in vivo wide-field MFLI, were used to validate a novel donor-dark quencher FRET pair. IRDye 800CW 2-deoxyglucose (2-DG) imaging was multiplexed with MFLI-FRET of NIR-labeled transferrin FRET pair (Tf-AF700/Tf-QC-1) to monitor tumor metabolism and probe uptake in breast tumor xenografts in intact live nude mice. Immunohistochemistry was used to validate in vivo imaging results. Results: First, we establish that IRDye QC-1 (QC-1) is an effective NIR dark acceptor for the FRET-induced quenching of donor Alexa Fluor 700 (AF700). Second, we report on simultaneous in vivo imaging of the metabolic probe 2-DG and MFLI-FRET imaging of Tf-AF700/Tf-QC-1 uptake in tumors. Such multiplexed imaging revealed an inverse relationship between 2-DG uptake and Tf intracellular delivery, suggesting that 2-DG signal may predict the efficacy of intracellular targeted delivery. Conclusions: Overall, our methodology enables for the first time simultaneous non-invasive monitoring of intracellular drug delivery and metabolic response in preclinical studies.

UNMIX-ME: spectral and lifetime fluorescence unmixing via deep learning

Hyperspectral fluorescence lifetime imaging allows for the simultaneous acquisition of spectrally resolved temporal fluorescence emission decays. In turn, the acquired rich multidimensional data set enables simultaneous imaging of multiple fluorescent species for a comprehensive molecular assessment of biotissues. However, to enable quantitative imaging, inherent spectral overlap between the considered fluorescent probes and potential bleed-through must be considered. Such a task is performed via either spectral or lifetime unmixing, typically independently. Herein, we present "UNMIX-ME" (unmix multiple emissions), a deep learning-based fluorescence unmixing routine, capable of quantitative fluorophore unmixing by simultaneously using both spectral and temporal signatures. UNMIX-ME was trained and validated using an in silico framework replicating the data acquisition process of a compressive hyperspectral fluorescent lifetime imaging platform (HMFLI). It was benchmarked against a conventional LSQ method for tri and quadri-exponential simulated samples. Last, UNMIX-ME's potential was assessed for NIR FRET in vitro and in vivo preclinical applications.

Super Sub-Nyquist Single-Pixel Imaging by Total Variation Ascending Ordering of the Hadamard Basis

Single pixel imaging (SPI) captures images without array detectors or raster scanning. When combined with compressive sensing techniques it enables novel solutions for high-speed optical imaging and spectroscopy. However, when it comes to the real-time capture and analysis of a fast event, the challenge is the inherent trade-off between frame rate and image resolution. Due to the lack of sufficient sparsity and the intrinsic iterative process, conventional compressed sensing techniques have limited improvement in capturing natural scenes and displaying the images in real time. In this work, we demonstrate a novel alternative compressive imaging approach employing an efficient and easy-implementation sampling scheme based on reordering the deterministic Hadamard basis through their total variation. By this means, the number of measurements and acquisition are reduced significantly without needing complex minimization algorithms. We can recover a 128 × 128 image with a sampling ratio of 5% at the signal peak signal-to-noise ratio (PSNR) of 23.8 dB, achieving super sub-Nyquist sampling SPI. Compared to other widely used sampling e.g. standard Hadamard protocols and Gaussian matrix methods, this approach results in a significant improvement both in the compression ratio and image reconstruction quality, enabling SPI for high frame rate imaging or video applications.

Single-shot ultrafast imaging attaining 70 trillion frames per second

Real-time imaging of countless femtosecond dynamics requires extreme speeds orders of magnitude beyond the limits of electronic sensors. Existing femtosecond imaging modalities either require event repetition or provide single-shot acquisition with no more than 10¹³ frames per second (fps) and 3 × 10² frames. Here, we report compressed ultrafast spectral photography (CUSP), which attains several new records in single-shot multi-dimensional imaging speeds. In active mode, CUSP achieves both 7 × 10¹³ fps and 10³ frames simultaneously by synergizing spectral encoding, pulse splitting, temporal shearing, and compressed sensing-enabling unprecedented quantitative imaging of rapid nonlinear light-matter interaction. In passive mode, CUSP provides four-dimensional (4D) spectral imaging at 0.5 × 10¹² fps, allowing the first single-shot spectrally resolved fluorescence lifetime imaging microscopy (SR-FLIM). As a real-time multi-dimensional imaging technology with the highest speeds and most frames, CUSP is envisioned to play instrumental roles in numerous pivotal scientific studies without the need for event repetition.

Diffraction-unlimited imaging based on conventional optical devices

We propose a computational paradigm where off-the-shelf optical devices can be used to image objects in a scene well beyond their native optical resolution. By design, our approach is generic, does not require active illumination, and is applicable to several types of optical devices. It only requires the placement of a spatial light modulator some distance from the optical system. In this paper, we first introduce the acquisition strategy together with the reconstruction framework. We then conduct practical experiments with a webcam that confirm that this approach can image objects with substantially enhanced spatial resolution compared to the performance of the native optical device. We finally discuss potential applications, current limitations, and future research directions.

A binary sampling Res2net reconstruction network for single-pixel imaging

The traditional algorithm for compressive reconstruction has high computational complexity. In order to reduce the reconstruction time of compressive sensing, deep learning networks have proven to be an effective solution. In this paper, we have developed a single-pixel imaging system based on deep learning and designed the binary sampling Res2Net reconstruction network (Bsr2-Net) model suitable for binary matrix sampling. In the experiments, we compared the structural similarity, peak signal-to-noise ratio, and reconstruction time using different reconstruction methods. Experimental results show that the Bsr2-Net is superior to several deep learning networks recently reported and closes to the most advanced reconstruction algorithms.

Single pixel hyperspectral bioluminescence tomography based on compressive sensing

Photonics based imaging is a widely utilised technique for the study of biological functions within pre-clinical studies. Specifically, bioluminescence imaging is a sensitive non-invasive and non-contact optical imaging technique that is able to detect distributed (biologically informative) visible and near-infrared activated light sources within tissue, providing information about tissue function. Compressive sensing (CS) is a method of signal processing that works on the basis that a signal or image can be compressed without important information being lost. This work describes the development of a CS based hyperspectral Bioluminescence imaging system that is used to collect compressed fluence data from the external surface of an animal model, due to an internal source, providing lower acquisition times, higher spectral content and potentially better tomographic source localisation. The work demonstrates that hyperspectral surface fluence images of both block and mouse shaped phantom due to internal light sources could be obtained at 30% of the time and measurements it would take to collect the data using conventional raster scanning methods. Using hyperspectral data, tomographic reconstruction of internal light sources can be carried out using any desired number of wavelengths and spectral bandwidth. Reconstructed images of internal light sources using four wavelengths as obtained through CS are presented showing a localisation error of ∼3 mm. Additionally, tomographic images of dual-colored sources demonstrating multi-wavelength light sources being recovered are presented further highlighting the benefits of the hyperspectral system for utilising multi-colored biomarker applications.

Novel time-resolved camera based on compressed sensing

Time-resolved cameras with high temporal resolution (down to ps) enable a huge set of novel applications ranging from biomedicine and environmental science to material and device characterization. In this work, we propose, and experimentally validate, a novel detection scheme for time-resolved imaging based on a compressed sampling approach. The proposed scheme unifies into a single element all the required operations, i.e. space modulation, space integration and time-resolved detection, paving the way to dramatic cost reduction, performance improvement and ease of use.

Manganese dioxide nanosheets: from preparation to biomedical applications

Advancements in nanotechnology and molecular biology have promoted the development of a diverse range of models to intervene in various disorders (from diagnosis to treatment and even theranostics). Manganese dioxide nanosheets (MnO₂ NSs), a typical two-dimensional (2D) transition metal oxide of nanomaterial that possesses unique structure and distinct properties have been employed in multiple disciplines in recent decades, especially in the field of biomedicine, including biocatalysis, fluorescence sensing, magnetic resonance imaging and cargo-loading functionality. A brief overview of the different synthetic methodologies for MnO₂ NSs and their state-of-the-art biomedical applications is presented below, as well as the challenges and future perspectives of MnO₂ NSs.

Stokes shift microscopy by excitation and emission imaging

In this contribution, we present a new method, based on a tunable excitation laser source and a robust common path interferometer in the detection channel. Its purpose is to image spectral excitation and emission information on a monochrome complementary metal oxide semiconductor (CMOS) camera. This allows us to spatially obtain both excitation and emission spectra of the whole imaged area and create derived images such as red-green-blue (RGB), excitation and emission maxima, and Stokes shift images. Our presented method is a further development of hyperspectral imaging that usually is limited to recording spatially resolved emission spectra. Taking advantage of the full camera chip should speed up the acquisition versus line scan or pointwise hyperspectral imaging.

Deep-tissue optical imaging of near cellular-sized features

Detection of biological features at the cellular level with sufficient sensitivity in complex tissue remains a major challenge. To appreciate this challenge, this would require finding tens to hundreds of cells (a 0.1 mm tumor has ~125 cells), out of ~37 trillion cells in the human body. Near-infrared optical imaging holds promise for high-resolution, deep-tissue imaging, but is limited by autofluorescence and scattering. To date, the maximum reported depth using second-window near-infrared (NIR-II: 1000–1700 nm) fluorophores is 3.2 cm through tissue. Here, we design an NIR-II imaging system, “Detection of Optically Luminescent Probes using Hyperspectral and diffuse Imaging in Near-infrared” (DOLPHIN), that resolves these challenges. DOLPHIN achieves the following: (i) resolution of probes through up to 8 cm of tissue phantom; (ii) identification of spectral and scattering signatures of tissues without apriori knowledge of background or autofluorescence; and (iii) 3D reconstruction of live whole animals. Notably, we demonstrate noninvasive real-time tracking of a 0.1 mm-sized fluorophore through the gastrointestinal tract of a living mouse, which is beyond the detection limit of current imaging modalities.

Single-pixel imaging with sampling distributed over simplex vertices

We propose a method of reduction of experimental noise in single-pixel imaging by expressing the subsets of sampling patterns as linear combinations of vertices of a multidimensional regular simplex. This method also may be directly extended to complementary sampling. The modified measurement matrix contains nonnegative elements with patterns that may be directly displayed on intensity spatial light modulators. The measurement becomes theoretically independent of the ambient illumination, and in practice becomes more robust to the varying conditions of the experiment. We show how the optimal dimension of the simplex depends on the level of measurement noise. We present experimental results of single-pixel imaging using binarized sampling and real-time reconstruction with the Fourier domain regularized inversion method.

Net-FLICS: fast quantitative wide-field fluorescence lifetime imaging with compressed sensing - a deep learning approach

Macroscopic fluorescence lifetime imaging (MFLI) via compressed sensed (CS) measurements enables efficient and accurate quantification of molecular interactions in vivo over a large field of view (FOV). However, the current data-processing workflow is slow, complex and performs poorly under photon-starved conditions. In this paper, we propose Net-FLICS, a novel image reconstruction method based on a convolutional neural network (CNN), to directly reconstruct the intensity and lifetime images from raw time-resolved CS data. By carefully designing a large simulated dataset, Net-FLICS is successfully trained and achieves outstanding reconstruction performance on both in vitro and in vivo experimental data and even superior results at low photon count levels for lifetime quantification.