Artificial confocal microscopy for deep label-free imaging

Perspectives on label-free microscopy of heterogeneous and dynamic biological systems

Significance

Advancements in label-free microscopy could provide real-time, non-invasive imaging with unique sources of contrast and automated standardized analysis to characterize heterogeneous and dynamic biological processes. These tools would overcome challenges with widely used methods that are destructive (e.g., histology, flow cytometry) or lack cellular resolution (e.g., plate-based assays, whole animal bioluminescence imaging).

Aim

This perspective aims to (1) justify the need for label-free microscopy to track heterogeneous cellular functions over time and space within unperturbed systems and (2) recommend improvements regarding instrumentation, image analysis, and image interpretation to address these needs.

Approach

Three key research areas (cancer research, autoimmune disease, and tissue and cell engineering) are considered to support the need for label-free microscopy to characterize heterogeneity and dynamics within biological systems. Based on the strengths (e.g., multiple sources of molecular contrast, non-invasive monitoring) and weaknesses (e.g., imaging depth, image interpretation) of several label-free microscopy modalities, improvements for future imaging systems are recommended.

Conclusion

Improvements in instrumentation including strategies that increase resolution and imaging speed, standardization and centralization of image analysis tools, and robust data validation and interpretation will expand the applications of label-free microscopy to study heterogeneous and dynamic biological systems.

Quantitative phase imaging based on holography: trends and new perspectives

In 1948, Dennis Gabor proposed the concept of holography, providing a pioneering solution to a quantitative description of the optical wavefront. After 75 years of development, holographic imaging has become a powerful tool for optical wavefront measurement and quantitative phase imaging. The emergence of this technology has given fresh energy to physics, biology, and materials science. Digital holography (DH) possesses the quantitative advantages of wide-field, non-contact, precise, and dynamic measurement capability for complex-waves. DH has unique capabilities for the propagation of optical fields by measuring light scattering with phase information. It offers quantitative visualization of the refractive index and thickness distribution of weak absorption samples, which plays a vital role in the pathophysiology of various diseases and the characterization of various materials. It provides a possibility to bridge the gap between the imaging and scattering disciplines. The propagation of wavefront is described by the complex amplitude. The complex-value in the complex-domain is reconstructed from the intensity-value measurement by camera in the real-domain. Here, we regard the process of holographic recording and reconstruction as a transformation between complex-domain and real-domain, and discuss the mathematics and physical principles of reconstruction. We review the DH in underlying principles, technical approaches, and the breadth of applications. We conclude with emerging challenges and opportunities based on combining holographic imaging with other methodologies that expand the scope and utility of holographic imaging even further. The multidisciplinary nature brings technology and application experts together in label-free cell biology, analytical chemistry, clinical sciences, wavefront sensing, and semiconductor production.

In silico labeling in cell biology: Potential and limitations

In silico labeling is the computational cross-modality image translation where the output modality is a subcellular marker that is not specifically encoded in the input image, for example, in silico localization of organelles from transmitted light images. In principle, in silico labeling has the potential to facilitate rapid live imaging of multiple organelles with reduced photobleaching and phototoxicity, a technology enabling a major leap toward understanding the cell as an integrated complex system. However, five years have passed since feasibility was attained, without any demonstration of using in silico labeling to uncover new biological insight. In here, we discuss the current state of in silico labeling, the limitations preventing it from becoming a practical tool, and how we can overcome these limitations to reach its full potential.

CellSNAP: a fast, accurate algorithm for 3D cell segmentation in quantitative phase imaging

Significance

Three-dimensional quantitative phase imaging (QPI) has rapidly emerged as a complementary tool to fluorescence imaging, as it provides an objective measure of cell morphology and dynamics, free of variability due to contrast agents. It has opened up new directions of investigation by providing systematic and correlative analysis of various cellular parameters without limitations of photobleaching and phototoxicity. While current QPI systems allow the rapid acquisition of tomographic images, the pipeline to analyze these raw three-dimensional (3D) tomograms is not well-developed. We focus on a critical, yet often underappreciated, step of the analysis pipeline that of 3D cell segmentation from the acquired tomograms.

Aim

We report the CellSNAP (Cell Segmentation via Novel Algorithm for Phase Imaging) algorithm for the 3D segmentation of QPI images.

Approach

The cell segmentation algorithm mimics the gemstone extraction process, initiating with a coarse 3D extrusion from a two-dimensional (2D) segmented mask to outline the cell structure. A 2D image is generated, and a segmentation algorithm identifies the boundary in the x - y plane. Leveraging cell continuity in consecutive z -stacks, a refined 3D segmentation, akin to fine chiseling in gemstone carving, completes the process.

Results

The CellSNAP algorithm outstrips the current gold standard in terms of speed, robustness, and implementation, achieving cell segmentation under 2 s per cell on a single-core processor. The implementation of CellSNAP can easily be parallelized on a multi-core system for further speed improvements. For the cases where segmentation is possible with the existing standard method, our algorithm displays an average difference of 5% for dry mass and 8% for volume measurements. We also show that CellSNAP can handle challenging image datasets where cells are clumped and marred by interferogram drifts, which pose major difficulties for all QPI-focused AI-based segmentation tools.

Conclusion

Our proposed method is less memory intensive and significantly faster than existing methods. The method can be easily implemented on a student laptop. Since the approach is rule-based, there is no need to collect a lot of imaging data and manually annotate them to perform machine learning based training of the model. We envision our work will lead to broader adoption of QPI imaging for high-throughput analysis, which has, in part, been stymied by a lack of suitable image segmentation tools.

Perspective on quantitative phase imaging to improve precision cancer medicine

Significance

Quantitative phase imaging (QPI) offers a label-free approach to non-invasively characterize cellular processes by exploiting their refractive index based intrinsic contrast. QPI captures this contrast by translating refractive index associated phase shifts into intensity-based quantifiable data with nanoscale sensitivity. It holds significant potential for advancing precision cancer medicine by providing quantitative characterization of the biophysical properties of cells and tissue in their natural states.

Aim

This perspective aims to discuss the potential of QPI to increase our understanding of cancer development and its response to therapeutics. It also explores new developments in QPI methods towards advancing personalized cancer therapy and early detection.

Approach

We begin by detailing the technical advancements of QPI, examining its implementations across transmission and reflection geometries and phase retrieval methods, both interferometric and non-interferometric. The focus then shifts to QPI's applications in cancer research, including dynamic cell mass imaging for drug response assessment, cancer risk stratification, and in-vivo tissue imaging.

Results

QPI has emerged as a crucial tool in precision cancer medicine, offering insights into tumor biology and treatment efficacy. Its sensitivity to detecting nanoscale changes holds promise for enhancing cancer diagnostics, risk assessment, and prognostication. The future of QPI is envisioned in its integration with artificial intelligence, morpho-dynamics, and spatial biology, broadening its impact in cancer research.

Conclusions

QPI presents significant potential in advancing precision cancer medicine and redefining our approach to cancer diagnosis, monitoring, and treatment. Future directions include harnessing high-throughput dynamic imaging, 3D QPI for realistic tumor models, and combining artificial intelligence with multi-omics data to extend QPI's capabilities. As a result, QPI stands at the forefront of cancer research and clinical application in cancer care.

EVATOM: an optical, label-free, machine learning assisted embryo health assessment tool

The combination of a good quality embryo and proper maternal health factors promise higher chances of a successful in vitro fertilization (IVF) procedure leading to clinical pregnancy and live birth. Of these two factors, selection of a good embryo is a controllable aspect. The current gold standard in clinical practice is visual assessment of an embryo based on its morphological appearance by trained embryologists. More recently, machine learning has been incorporated into embryo selection "packages". Here, we report EVATOM: a machine-learning assisted embryo health assessment tool utilizing an optical quantitative phase imaging technique called artificial confocal microscopy (ACM). We present a label-free nucleus detection method with, to the best of our knowledge, novel quantitative embryo health biomarkers. Two viability assessment models are presented for grading embryos into two classes: healthy/intermediate (H/I) or sick (S) class. The models achieve a weighted F1 score of 1.0 and 0.99 respectively on the in-distribution test set of 72 fixed embryos and a weighted F1 score of 0.9 and 0.95 respectively on the out-of-distribution test dataset of 19 time-instances from 8 live embryos.

Coherent light scattering from cellular dynamics in living tissues

This review examines the biological physics of intracellular transport probed by the coherent optics of dynamic light scattering from optically thick living tissues. Cells and their constituents are in constant motion, composed of a broad range of speeds spanning many orders of magnitude that reflect the wide array of functions and mechanisms that maintain cellular health. From the organelle scale of tens of nanometers and upward in size, the motion inside living tissue is actively driven rather than thermal, propelled by the hydrolysis of bioenergetic molecules and the forces of molecular motors. Active transport can mimic the random walks of thermal Brownian motion, but mean-squared displacements are far from thermal equilibrium and can display anomalous diffusion through Lévy or fractional Brownian walks. Despite the average isotropic three-dimensional environment of cells and tissues, active cellular or intracellular transport of single light-scattering objects is often pseudo-one-dimensional, for instance as organelle displacement persists along cytoskeletal tracks or as membranes displace along the normal to cell surfaces, albeit isotropically oriented in three dimensions. Coherent light scattering is a natural tool to characterize such tissue dynamics because persistent directed transport induces Doppler shifts in the scattered light. The many frequency-shifted partial waves from the complex and dynamic media interfere to produce dynamic speckle that reveals tissue-scale processes through speckle contrast imaging and fluctuation spectroscopy. Low-coherence interferometry, dynamic optical coherence tomography, diffusing-wave spectroscopy, diffuse-correlation spectroscopy, differential dynamic microscopy and digital holography offer coherent detection methods that shed light on intracellular processes. In health-care applications, altered states of cellular health and disease display altered cellular motions that imprint on the statistical fluctuations of the scattered light. For instance, the efficacy of medical therapeutics can be monitored by measuring the changes they induce in the Doppler spectra of livingex vivocancer biopsies.

Beyond fluorescence: advances in computational label-free full specificity in 3D quantitative phase microscopy

Despite remarkable progresses in quantitative phase imaging (QPI) microscopes, their wide acceptance is limited due to the lack of specificity compared with the well-established fluorescence microscopy. In fact, the absence of fluorescent tag prevents to identify subcellular structures in single cells, making challenging the interpretation of label-free 2D and 3D phase-contrast data. Great effort has been made by many groups worldwide to address and overcome such limitation. Different computational methods have been proposed and many more are currently under investigation to achieve label-free microscopic imaging at single-cell level to recognize and quantify different subcellular compartments. This route promises to bridge the gap between QPI and FM for real-world applications.

Automated Cell Lineage Reconstruction using Label-Free 4D Microscopy

Here we describe embGAN, a deep learning pipeline that addresses the challenge of automated cell detection and tracking in label-free 3D time lapse imaging. embGAN requires no manual data annotation for training, learns robust detections that exhibits a high degree of scale invariance and generalizes well to images acquired in multiple labs on multiple instruments.

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

Trustworthy in silico cell labeling via ensemble-based image translation

Artificial intelligence (AI) image translation has been a valuable tool for processing image data in biological and medical research. To apply such a tool in mission-critical applications, including drug screening, toxicity study, and clinical diagnostics, it is essential to ensure that the AI prediction is trustworthy. Here, we demonstrate that an ensemble learning method can quantify the uncertainty of AI image translation. We tested the uncertainty evaluation using experimentally acquired images of mesenchymal stromal cells. We find that the ensemble method reports a prediction standard deviation that correlates with the prediction error, estimating the prediction uncertainty. We show that this uncertainty is in agreement with the prediction error and Pearson correlation coefficient. We further show that the ensemble method can detect out-of-distribution input images by reporting increased uncertainty. Altogether, these results suggest that the ensemble-estimated uncertainty can be a useful indicator for identifying erroneous AI image translations.

Live-cell imaging in the deep learning era

Live imaging is a powerful tool, enabling scientists to observe living organisms in real time. In particular, when combined with fluorescence microscopy, live imaging allows the monitoring of cellular components with high sensitivity and specificity. Yet, due to critical challenges (i.e., drift, phototoxicity, dataset size), implementing live imaging and analyzing the resulting datasets is rarely straightforward. Over the past years, the development of bioimage analysis tools, including deep learning, is changing how we perform live imaging. Here we briefly cover important computational methods aiding live imaging and carrying out key tasks such as drift correction, denoising, super-resolution imaging, artificial labeling, tracking, and time series analysis. We also cover recent advances in self-driving microscopy.

Polarization-multiplexed metasurface enabled tri-functional imaging

Diffraction-limited focusing imaging, edge-enhanced imaging, and long depth of focus imaging offer crucial technical capabilities for applications such as biological microscopy and surface topography detection. To conveniently and quickly realize the microscopy imaging of different functions, the multifunctional integrated system of microscopy imaging has become an increasingly important research direction. However, conventional microscopes necessitate bulky optical components to switch between these functionalities, suffering from the system's complexity and unstability. Hence, solving the problem of integrating multiple functions within an optical system is a pressing need. In this work, we present an approach using a polarization-multiplexed tri-functional metasurface, capable of realizing the aforementioned imaging functions simply by changing the polarization state of the input and output light, enhancing the system structure's compactness and flexibility. This work offers a new avenue for multifunctional imaging, with potential applications in biomedicine and microscopy imaging.

Multipass wide-field phase imager

Advances in optical imaging always look for an increase in sensitivity and resolution among other practicability aspects. Within the same scope, in this work we report a versatile interference contrast imaging technique, with high phase sensitivity and a large field-of-view of several mm2. Sensitivity is increased through the use of a self-imaging non-resonant cavity, which causes photons to probe the sample in multiple rounds before being detected, where the configuration can be transmissive or reflective. Phase profiles can be resolved individually for each round thanks to a specially designed single-photon camera with time-of-flight capabilities and true pixels-off gating. Measurement noise is reduced by novel data processing combining the retrieved sample profiles from multiple rounds. Our protocol is especially useful under extremely low light conditions as required by biological or photo-sensitive samples. Results demonstrate more than a four-fold reduction in phase measurement noise, compared to single round imaging, and values close to the predicted sensitivity in case of the best possible cavity configuration, where all photons are maintained until n rounds. We also find good agreement with the theoretical predictions for low number of rounds, where experimental imperfections would play a minor role. The absence of a laser or cavity lock-in mechanism makes the technique an easy to use inspection tool.

YOLOv5-FPN: A Robust Framework for Multi-Sized Cell Counting in Fluorescence Images

Cell counting in fluorescence microscopy is an essential task in biomedical research for analyzing cellular dynamics and studying disease progression. Traditional methods for cell counting involve manual counting or threshold-based segmentation, which are time-consuming and prone to human error. Recently, deep learning-based object detection methods have shown promising results in automating cell counting tasks. However, the existing methods mainly focus on segmentation-based techniques that require a large amount of labeled data and extensive computational resources. In this paper, we propose a novel approach to detect and count multiple-size cells in a fluorescence image slide using You Only Look Once version 5 (YOLOv5) with a feature pyramid network (FPN). Our proposed method can efficiently detect multiple cells with different sizes in a single image, eliminating the need for pixel-level segmentation. We show that our method outperforms state-of-the-art segmentation-based approaches in terms of accuracy and computational efficiency. The experimental results on publicly available datasets demonstrate that our proposed approach achieves an average precision of 0.8 and a processing time of 43.9 ms per image. Our approach addresses the research gap in the literature by providing a more efficient and accurate method for cell counting in fluorescence microscopy that requires less computational resources and labeled data.

Deep learning-enhanced fluorescence microscopy via confocal physical imaging model

Confocal microscopy is one of the most widely used tools for high-resolution cellular, tissue imaging and industrial inspection. Micrograph reconstruction based on deep learning has become an effective tool for modern microscopy imaging techniques. While most deep learning methods neglect the imaging process mechanism, which requires a lot of work to solve the multi-scale image pairs aliasing problem. We show that these limitations can be mitigated via an image degradation model based on Richards-Wolf vectorial diffraction integral and confocal imaging theory. The low-resolution images required for network training are generated by model degradation from their high-resolution counterparts, thereby eliminating the need for accurate image alignment. The image degradation model ensures the generalization and fidelity of the confocal images. By combining the residual neural network with a lightweight feature attention module with degradation model of confocal microscopy ensures high fidelity and generalization. Experiments on different measured data report that compared with the two deconvolution algorithms, non-negative least squares algorithm and Richardson-Lucy algorithm, the structural similarity index between the network output image and the real image reaches a high level above 0.82, and the peak signal-to-noise ratio can be improved by more than 0.6 dB. It also shows good applicability in different deep learning networks.

Influence of Yokukansan on the refractive index of neuroblastoma cells

Yokukansan (YKS) is a traditional Japanese herbal medicine that is increasingly being studied for its effects on neurodegenerative diseases. In our study, we presented a novel methodology for a multimodal analysis of the effects of YKS on nerve cells. The measurements of 3D refractive index distribution and its changes performed by holographic tomography were supported with an investigation by Raman micro-spectroscopy and fluorescence microscopy to gather complementary morphological and chemical information about cells and YKS influence. It was shown that at the concentrations tested, YKS inhibits proliferation, possibly involving reactive oxygen species. Also substantial changes in the cell RI after few hours of YKS exposure were detected, followed by longer-term changes in cell lipid composition and chromatin state.