Real-time fluorescence lifetime actuation for cell sorting using a CMOS SPAD silicon photomultiplier

Compact and robust deep learning architecture for fluorescence lifetime imaging and FPGA implementation

This paper reports a bespoke adder-based deep learning network for time-domain fluorescence lifetime imaging (FLIM). By leveraging thel1-norm extraction method, we propose a 1D Fluorescence Lifetime AdderNet (FLAN) without multiplication-based convolutions to reduce the computational complexity. Further, we compressed fluorescence decays in temporal dimension using a log-scale merging technique to discard redundant temporal information derived as log-scaling FLAN (FLAN+LS). FLAN+LS achieves 0.11 and 0.23 compression ratios compared with FLAN and a conventional 1D convolutional neural network (1D CNN) while maintaining high accuracy in retrieving lifetimes. We extensively evaluated FLAN and FLAN+LS using synthetic and real data. A traditional fitting method and other non-fitting, high-accuracy algorithms were compared with our networks for synthetic data. Our networks attained a minor reconstruction error in different photon-count scenarios. For real data, we used fluorescent beads' data acquired by a confocal microscope to validate the effectiveness of real fluorophores, and our networks can differentiate beads with different lifetimes. Additionally, we implemented the network architecture on a field-programmable gate array (FPGA) with a post-quantization technique to shorten the bit-width, thereby improving computing efficiency. FLAN+LS on hardware achieves the highest computing efficiency compared to 1D CNN and FLAN. We also discussed the applicability of our network and hardware architecture for other time-resolved biomedical applications using photon-efficient, time-resolved sensors.

Dynamic fluorescence lifetime sensing with CMOS single-photon avalanche diode arrays and deep learning processors

Measuring fluorescence lifetimes of fast-moving cells or particles have broad applications in biomedical sciences. This paper presents a dynamic fluorescence lifetime sensing (DFLS) system based on the time-correlated single-photon counting (TCSPC) principle. It integrates a CMOS 192 × 128 single-photon avalanche diode (SPAD) array, offering an enormous photon-counting throughput without pile-up effects. We also proposed a quantized convolutional neural network (QCNN) algorithm and designed a field-programmable gate array embedded processor for fluorescence lifetime determinations. The processor uses a simple architecture, showing unparallel advantages in accuracy, analysis speed, and power consumption. It can resolve fluorescence lifetimes against disturbing noise. We evaluated the DFLS system using fluorescence dyes and fluorophore-tagged microspheres. The system can effectively measure fluorescence lifetimes within a single exposure period of the SPAD sensor, paving the way for portable time-resolved devices and shows potential in various applications.

Time-domain single photon-excited autofluorescence lifetime for label-free detection of T cell activation

Fluorescence lifetime imaging microscopy (FLIM) is a powerful technique, capable of label-free assessment of the metabolic state and function within single cells. The FLIM measurements of autofluorescence were recently shown to be sensitive to the functional state and subtype of T cells. Therefore, autofluorescence FLIM could improve cell manufacturing technologies for adoptive immunotherapy, which currently require a time-intensive process of cell labeling with fluorescent antibodies. However, current autofluorescence FLIM implementations are typically too slow, bulky, and prohibitively expensive for use in cell manufacturing pipelines. Here we report a single photon-excited confocal whole-cell autofluorescence system that uses fast field-programmable gate array-based time tagging electronics to achieve time-correlated single photon counting (TCSPC) of single-cell autofluorescence. The system includes simultaneous near-infrared bright-field imaging and is sensitive to variations in the fluorescence decay profile of the metabolic coenzyme NAD(P)H in human T cells due to the activation state. The classification of activated and quiescent T cells achieved high accuracy and precision (area under the receiver operating characteristic curve, AUC = 0.92). The lower-cost, higher acquisition speed, and resistance to pile-up effects at high photon flux compared to traditional multiphoton-excited FLIM and TCSPC implementations with similar SNR make this system attractive for integration into flow cytometry, sorting, and quality control in cell manufacturing.

A Review of New High-Throughput Methods Designed for Fluorescence Lifetime Sensing From Cells and Tissues

Though much of the interest in fluorescence in the past has been on measuring spectral qualities such as wavelength and intensity, there are two other highly useful intrinsic properties of fluorescence: lifetime (or decay) and anisotropy (or polarization). Each has its own set of unique advantages, limitations, and challenges in detection when it comes to use in biological studies. This review will focus on the property of fluorescence lifetime, providing a brief background on instrumentation and theory, and examine the recent advancements and applications of measuring lifetime in the fields of high-throughput fluorescence lifetime imaging microscopy (HT-FLIM) and time-resolved flow cytometry (TRFC). In addition, the crossover of these two methods and their outlooks will be discussed.

Sodium-Sensitive Contact Lens for Diagnostics of Ocular Pathologies

The ability to measure all the electrolyte concentrations in tears would be valuable in ophthalmology for research and diagnosis of dry eye disease (DED) and other ocular pathologies. However, tear samples are difficult to collect and analyze because the total volume is small and the chemical composition changes rapidly. Measurements of electrolytes in tears is challenging because typical clinical assays for proteins and other biomarkers cannot be used to detect ion concentrations tears. Here, we report the contact lens which is sensitive to sodium ion (Na+), one of the dominant electrolytes in tears. The Na ions in tears is diagnostic for DED. Three sodium-sensitive fluorophores (SG-C16, SG-LPE and SG-PL) were synthesized by derivatizing the sodium green with 1-hexadecyl amine, 1-oleoyl-2-hydroxy-sn-glycero-3-phosphoethanolamine or poly-L-lysine, respectively. These probes were bound to modern silicone hydrogel (SiHG) contact lens, Biofinity from Cooper Vision. Doped lenses were tested for sodium ion dependent spectral properties of probes within the contact lens. The probes displayed changes in intensity and lifetime in response to Na+ concentration, were completely reversible, no significant probe wash-out from the lenses, were not affected by proteins in tears and were not removed after repeated washing. These results are the first step to our long-term goal, which is a lens sensitive to all the electrolytes in tears. We presented design, synthesis and implementation of three new sodium sensitive probes within a silicon hydrogel lens. Contact lenses to measure the other electrolytes in tears can be developed using the same approach by synthesis and testing of new ion-sensitive fluorophores.

Design of a Real-Time Breakdown Voltage and On-Chip Temperature Monitoring System for Single Photon Avalanche Diodes

The design and implementation of a real-time breakdown voltage and on-chip temperature monitoring system for single photon avalanche diodes (SPADs) is described in this work. In the system, an on-chip shaded (active area of the detector covered by a metal layer) SPAD is used to provide a dark count rate for the breakdown voltage and temperature calculation. A bias circuit was designed to provide a bias voltage scanning for the shaded SPAD. A microcontroller records the pulses from the anode of the shaded SPAD and calculates its real-time dark count rate. An algorithm was developed for the microcontroller to calculate the SPAD’s breakdown voltage and the on-chip temperature in real time. Experimental results show that the system is capable of measuring the SPAD’s breakdown voltage with a mismatch of less than 1.2%. Results also show that the system can provide real-time on-chip temperature monitoring for the range of −10 to 50 °C with errors of less than 1.7 °C. The system proposed can be used for the real-time SPAD’s breakdown voltage and temperature estimation for dual-SPADs or SPAD arrays chip where identical detectors are fabricated on the same chip and one or more dummy SPADs are shaded. With the breakdown voltage and the on-chip temperature monitoring, intelligent control logic can be developed to optimize the performance of the SPAD-based photon counting system by adjusting the parameters such as excess bias voltage and dead-time. This is particularly useful for SPAD photon counting systems used in complex working environments such as the applications in 3D LIDAR imaging for geodesy, geology, geomorphology, forestry, atmospheric physics and autonomous vehicles.

Fluorescence lifetime imaging with a megapixel SPAD camera and neural network lifetime estimation

Fluorescence lifetime imaging microscopy (FLIM) is a key technology that provides direct insight into cell metabolism, cell dynamics and protein activity. However, determining the lifetimes of different fluorescent proteins requires the detection of a relatively large number of photons, hence slowing down total acquisition times. Moreover, there are many cases, for example in studies of cell collectives, where wide-field imaging is desired. We report scan-less wide-field FLIM based on a 0.5 MP resolution, time-gated Single Photon Avalanche Diode (SPAD) camera, with acquisition rates up to 1 Hz. Fluorescence lifetime estimation is performed via a pre-trained artificial neural network with 1000-fold improvement in processing times compared to standard least squares fitting techniques. We utilised our system to image HT1080-human fibrosarcoma cell line as well as Convallaria. The results show promise for real-time FLIM and a viable route towards multi-megapixel fluorescence lifetime images, with a proof-of-principle mosaic image shown with 3.6 MP.

Realization of a time-correlated photon counting technique for fluorescence analysis

An ultralow level light detection module, the time-correlated photon counter, is proposed and evaluated for fluorescence analysis. The time-correlated photon counter employs a silicon photomultiplier as a photon counting sensor in conjunction with a Poisson statistics algorithm and a double time windows technique, and therefore it can accurately count the photon number. The time-correlated photon counter is compatible with the time-correlated single photon counting technique and can record the arrival time of very faint light signals. This low-cost and compact instrument was used to analyze the intensity and lifetime of fluorescein isothiocyanate; a limit of detection of 16 pg/ml with a large linear dynamic range from 2.86 pg/ml to 0.5 µg/ml was obtained, and the lifetime of fluorescein isothiocyanate was measured to be 3.758 ns, which agrees well with the results of a sophisticated commercial fluorescence analysis instrument. The time-correlated photon counter may be useful in applications such as point-of-care testing.

High-speed microparticle isolation unlimited by Poisson statistics

High-speed isolation of microparticles (e.g., microplastics, heavy metal particles, microbes, cells) from heterogeneous populations is the key element of high-throughput sorting instruments for chemical, biological, industrial and medical applications. Unfortunately, the performance of continuous microparticle isolation or so-called sorting is fundamentally limited by the trade-off between throughput, purity, and yield. For example, at a given throughput, high-purity sorting needs to sacrifice yield, or vice versa. This is due to Poisson statistics of events (i.e., microparticles, microparticle clusters, microparticle debris) in which the interval between successive events is stochastic and can be very short. Here we demonstrate an on-chip microparticle sorter with an ultrashort switching window in both time (10 μs) and space (10 μm) at a high flow speed of 1 m s^-1, thereby overcoming the Poisson trade-off. This is made possible by using femtosecond laser pulses that can produce highly localized transient cavitation bubbles in a microchannel to kick target microparticles from an acoustically focused, densely aligned, bumper-to-bumper stream of microparticles. Our method is important for rare-microparticle sorting applications where both high purity and high yield are required to avoid missing rare microparticles.

Liquid-core waveguide TCSPC sensor for high-accuracy fluorescence lifetime analysis

Liquid-core waveguide (LCW) has many advantages such as the elimination of optical artifacts typically exhibited in systems employing lenses and filters. However, due to the effect of temporal dispersion, LCWs are typically employed in steady-state fluorescence detection microsystems rather than in fluorescence lifetime measurement (FLM) systems. In this paper, we present a compact liquid-core waveguide time-correlated single-photon counting (LCW-TCSPC) sensor for FLM. The propagation of excitation within the LCW is analyzed both analytically and in simulations, with results in agreement with experimental characterization. Results reveal an optimal region within the LCW for highly accurate FLM. The proposed prototype achieves excellent excitation rejection and low temporal dispersion as a result of optimization of the propagation length of the excitation within the LCW. The prototype achieves a detection limit of 5 nM for Coumarin 6 in dimethyl sulfoxide with < 3% lifetime error. The techniques proposed for analyzing the LCW for TCSPC based FLM and prototype demonstration pave the way for developing high-performance fluorescence lifetime measurement for microfluidics and point-of-care applications. Graphical abstract A compact liquid-core waveguide time-correlated single-photon counting (LCW-TCSPC) sensor for fluorescence lifetime measurement (FLM) is presented. Results reveal an optimal propagation length region within the LCW for highly accurate FLM. The prototype achieves a detection limit of 5 nM for Coumarin 6 in dimethyl sulfoxide with < 3% lifetime error.

Response to Comment on “Ghost cytometry”

Di Carlo et al . comment that our original results were insufficient to prove that the ghost cytometry technique is performing a morphologic analysis of cells in flow. We emphasize that the technique is primarily intended to acquire and classify morphological information of cells in a computationally efficient manner without reconstructing images. We provide additional supporting information, including images reconstructed from the compressive waveforms and a discussion of current and future throughput potentials.

Measurement Matrix Construction for Large-area Single Photon Compressive Imaging

We have developed a single photon compressive imaging system based on single photon counting technology and compressed sensing theory, using a photomultiplier tube (PMT) photon counting head as the bucket detector. This system can realize ultra-weak light imaging with the imaging area up to the entire digital micromirror device (DMD) working region. The measurement matrix in this system is required to be binary due to the two working states of the micromirror corresponding to two controlled elements. And it has a great impact on the performance of the imaging system, because it involves modulation of the optical signal and image reconstruction. Three kinds of binary matrix including sparse binary random matrix, m sequence matrix and true random number matrix are constructed. The properties of these matrices are analyzed theoretically with the uncertainty principle. The parameters of measurement matrix including sparsity ratio, compressive sampling ratio and reconstruction time are verified in the experimental system. The experimental results show that, the increase of sparsity ratio and compressive sampling ratio can improve the reconstruction quality. However, when the increase is up to a certain value, the reconstruction quality tends to be saturated. Compared to the other two types of measurement matrices, the m sequence matrix has better performance in image reconstruction.

Single-molecule techniques in biophysics: a review of the progress in methods and applications

Single-molecule biophysics has transformed our understanding of biology, but also of the physics of life. More exotic than simple soft matter, biomatter lives far from thermal equilibrium, covering multiple lengths from the nanoscale of single molecules to up to several orders of magnitude higher in cells, tissues and organisms. Biomolecules are often characterized by underlying instability: multiple metastable free energy states exist, separated by levels of just a few multiples of the thermal energy scale k _B T, where k _B is the Boltzmann constant and T absolute temperature, implying complex inter-conversion kinetics in the relatively hot, wet environment of active biological matter. A key benefit of single-molecule biophysics techniques is their ability to probe heterogeneity of free energy states across a molecular population, too challenging in general for conventional ensemble average approaches. Parallel developments in experimental and computational techniques have catalysed the birth of multiplexed, correlative techniques to tackle previously intractable biological questions. Experimentally, progress has been driven by improvements in sensitivity and speed of detectors, and the stability and efficiency of light sources, probes and microfluidics. We discuss the motivation and requirements for these recent experiments, including the underpinning mathematics. These methods are broadly divided into tools which detect molecules and those which manipulate them. For the former we discuss the progress of super-resolution microscopy, transformative for addressing many longstanding questions in the life sciences, and for the latter we include progress in 'force spectroscopy' techniques that mechanically perturb molecules. We also consider in silico progress of single-molecule computational physics, and how simulation and experimentation may be drawn together to give a more complete understanding. Increasingly, combinatorial techniques are now used, including correlative atomic force microscopy and fluorescence imaging, to probe questions closer to native physiological behaviour. We identify the trade-offs, limitations and applications of these techniques, and discuss exciting new directions.

Recent Advances in Fluorescence Lifetime Analytical Microsystems: Contact Optics and CMOS Time-Resolved Electronics

Fluorescence spectroscopy has become a prominent research tool with wide applications in medical diagnostics and bio-imaging. However, the realization of combined high-performance, portable, and low-cost spectroscopic sensors still remains a challenge, which has limited the technique to the laboratories. A fluorescence lifetime measurement seeks to obtain the characteristic lifetime from the fluorescence decay profile. Time-correlated single photon counting (TCSPC) and time-gated techniques are two key variations of time-resolved measurements. However, commercial time-resolved analysis systems typically contain complex optics and discrete electronic components, which lead to bulkiness and a high cost. These two limitations can be significantly mitigated using contact sensing and complementary metal-oxide-semiconductor (CMOS) implementation. Contact sensing simplifies the optics, whereas CMOS technology enables on-chip, arrayed detection and signal processing, significantly reducing size and power consumption. This paper examines recent advances in contact sensing and CMOS time-resolved circuits for the realization of fully integrated fluorescence lifetime measurement microsystems. The high level of performance from recently reported prototypes suggests that the CMOS-based contact sensing microsystems are emerging as sound technologies for application-specific, low-cost, and portable time-resolved diagnostic devices.

Contact lens to measure individual ion concentrations in tears and applications to dry eye disease

Dry eye disease (DED) affects millions of individuals in the United States and worldwide, and the incidence is increasing with an aging population. There is widespread agreement that the measurement of total tear osmolarity is the most reliable test, but this procedure provides only the total ionic strength and does not provide the concentration of each ionic species in tears. Here, we describe an approach to determine the individual ion concentrations in tears using modern silicone hydrogel (SiHG) contact lenses. We made pH (or H₃O⁺, hydronium cation,/OH^-, hydroxyl ion) and chloride ion (two of the important electrolytes in tear fluid) sensitive SiHG contact lenses. We attached hydrophobic C18 chains to water-soluble fluorescent probes for pH and chloride. The resulting hydrophobic ion sensitive fluorophores (H-ISF) bind strongly to SiHG lenses and could not be washed out with aqueous solutions. Both H-ISFs provide measurements which are independent of total intensity by use of wavelength-ratiometric measurements for pH or lifetime-based sensing for chloride. Our approach can be extended to fabricate a contact lens which provides measurements of the six dominant ionic species in tears. This capability will be valuable for research into the biochemical processes causing DED, which may improve the ability to diagnose the various types of DED.

Towards sensitive, high-throughput, biomolecular assays based on fluorescence lifetime

Time-resolved fluorescence detection for robust sensing of biomolecular interactions is developed by implementing time-correlated single photon counting in high-throughput conditions. Droplet microfluidics is used as a promising platform for the very fast handling of low-volume samples. We illustrate the potential of this very sensitive and cost-effective technology in the context of an enzymatic activity assay based on fluorescently-labeled biomolecules. Fluorescence lifetime detection by time-correlated single photon counting is shown to enable reliable discrimination between positive and negative control samples at a throughput as high as several hundred samples per second.

Note: Rapid measurement of fluorescence lifetimes using SiPM detection and waveform sampling

In fluorescence spectroscopy and imaging, fluorescence lifetime measurement-assessing the average time fluorophores spend in their excited state before returning to their ground state-offers a number of advantages over quantifying fluorescence intensities that include resistance to photo-bleaching and independence from fluorophore concentration, excitation intensity, and measurement methodology. Despite growing interest, fluorescence lifetime techniques frequently mandate relatively complex instrumentation, slow data acquisition rates, and significant data analyses. In this work, we demonstrate the feasibility of measuring fluorescence lifetimes using off-the-shelf analog silicon photomultipliers and switched-capacitor array waveform sampling techniques, with precision matching that of much larger and more elaborate commercial instruments.

Time-resolved spectroscopy at 19,000 lines per second using a CMOS SPAD line array enables advanced biophotonics applications

A SPAD-based line sensor fabricated in 130 nm CMOS technology capable of acquiring time-resolved fluorescence spectra (TRFS) in 8.3 milliseconds is presented. To the best of our knowledge, this is the fastest time correlated single photon counting (TCSPC) TRFS acquisition reported to date. The line sensor is an upgrade to our prior work and incorporates: i) parallelized interface from sensor to surrounding circuitry enabling high line rate to the PC (19,000 lines/s) and ii) novel time-gating architecture where detected photons in the OFF region are rejected digitally after the output stage of the SPAD. The time-gating architecture was chosen to avoid electrical transients on the SPAD high voltage supplies when gating is achieved by excess bias modulation. The time-gate has an adjustable location and time window width allowing the user to focus on time-events of interest. On-chip integrated center-of-mass (CMM) calculations provide efficient acquisition of photon arrivals and direct lifetime estimation of fluorescence decays. Furthermore, any of the SPC, TCSPC and on-chip CMM modes can be used in conjunction with the time-gating. The higher readout rate and versatile architecture greatly empower the user and will allow widespread applications across many techniques and disciplines. Here we focused on 3 examples of TRFS and time-gated Raman spectroscopy: i) kinetics of chlorophyll A fluorescence from an intact leaf; ii) kinetics of a thrombin biosensor FRET probe from quenched to fluorescence states; iii) ex vivo mouse lung tissue autofluorescence TRFS; iv) time-gated Raman spectroscopy of toluene at 3056 cm^-1 peak. To the best of our knowledge, we detect spectrally for the first time the fast rise in fluorescence lifetime of chlorophyll A in a measurement over single fluorescent transient.