Single-cell cytometry via multiplexed fluorescence prediction by label-free reflectance microscopy

Information-Distilled Generative Label-Free Morphological Profiling Encodes Cellular Heterogeneity

Image-based cytometry faces challenges due to technical variations arising from different experimental batches and conditions, such as differences in instrument configurations or image acquisition protocols, impeding genuine biological interpretation of cell morphology. Existing solutions, often necessitating extensive pre-existing data knowledge or control samples across batches, have proved limited, especially with complex cell image data. To overcome this, "Cyto-Morphology Adversarial Distillation" (CytoMAD), a self-supervised multi-task learning strategy that distills biologically relevant cellular morphological information from batch variations, is introduced to enable integrated analysis across multiple data batches without complex data assumptions or extensive manual annotation. Unique to CytoMAD is its "morphology distillation", symbiotically paired with deep-learning image-contrast translation-offering additional interpretable insights into label-free cell morphology. The versatile efficacy of CytoMAD is demonstrated in augmenting the power of biophysical imaging cytometry. It allows integrated label-free classification of human lung cancer cell types and accurately recapitulates their progressive drug responses, even when trained without the drug concentration information. CytoMAD  also allows joint analysis of tumor biophysical cellular heterogeneity, linked to epithelial-mesenchymal plasticity, that standard fluorescence markers overlook. CytoMAD can substantiate the wide adoption of biophysical cytometry for cost-effective diagnosis and screening.

In silico labeling in cell biology: Potential and limitations

In silico labeling is the computational cross-modality image translation where the output modality is a subcellular marker that is not specifically encoded in the input image, for example, in silico localization of organelles from transmitted light images. In principle, in silico labeling has the potential to facilitate rapid live imaging of multiple organelles with reduced photobleaching and phototoxicity, a technology enabling a major leap toward understanding the cell as an integrated complex system. However, five years have passed since feasibility was attained, without any demonstration of using in silico labeling to uncover new biological insight. In here, we discuss the current state of in silico labeling, the limitations preventing it from becoming a practical tool, and how we can overcome these limitations to reach its full potential.

Perspective on quantitative phase imaging to improve precision cancer medicine

Significance

Quantitative phase imaging (QPI) offers a label-free approach to non-invasively characterize cellular processes by exploiting their refractive index based intrinsic contrast. QPI captures this contrast by translating refractive index associated phase shifts into intensity-based quantifiable data with nanoscale sensitivity. It holds significant potential for advancing precision cancer medicine by providing quantitative characterization of the biophysical properties of cells and tissue in their natural states.

Aim

This perspective aims to discuss the potential of QPI to increase our understanding of cancer development and its response to therapeutics. It also explores new developments in QPI methods towards advancing personalized cancer therapy and early detection.

Approach

We begin by detailing the technical advancements of QPI, examining its implementations across transmission and reflection geometries and phase retrieval methods, both interferometric and non-interferometric. The focus then shifts to QPI's applications in cancer research, including dynamic cell mass imaging for drug response assessment, cancer risk stratification, and in-vivo tissue imaging.

Results

QPI has emerged as a crucial tool in precision cancer medicine, offering insights into tumor biology and treatment efficacy. Its sensitivity to detecting nanoscale changes holds promise for enhancing cancer diagnostics, risk assessment, and prognostication. The future of QPI is envisioned in its integration with artificial intelligence, morpho-dynamics, and spatial biology, broadening its impact in cancer research.

Conclusions

QPI presents significant potential in advancing precision cancer medicine and redefining our approach to cancer diagnosis, monitoring, and treatment. Future directions include harnessing high-throughput dynamic imaging, 3D QPI for realistic tumor models, and combining artificial intelligence with multi-omics data to extend QPI's capabilities. As a result, QPI stands at the forefront of cancer research and clinical application in cancer care.

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

Efficient Supervised Pretraining of Swin-Transformer for Virtual Staining of Microscopy Images

Fluorescence staining is an important technique in life science for labeling cellular constituents. However, it also suffers from being time-consuming, having difficulty in simultaneous labeling, etc. Thus, virtual staining, which does not rely on chemical labeling, has been introduced. Recently, deep learning models such as transformers have been applied to virtual staining tasks. However, their performance relies on large-scale pretraining, hindering their development in the field. To reduce the reliance on large amounts of computation and data, we construct a Swin-transformer model and propose an efficient supervised pretraining method based on the masked autoencoder (MAE). Specifically, we adopt downsampling and grid sampling to mask 75% of pixels and reduce the number of tokens. The pretraining time of our method is only 1/16 compared with the original MAE. We also design a supervised proxy task to predict stained images with multiple styles instead of masked pixels. Additionally, most virtual staining approaches are based on private datasets and evaluated by different metrics, making a fair comparison difficult. Therefore, we develop a standard benchmark based on three public datasets and build a baseline for the convenience of future researchers. We conduct extensive experiments on three benchmark datasets, and the experimental results show the proposed method achieves the best performance both quantitatively and qualitatively. In addition, ablation studies are conducted, and experimental results illustrate the effectiveness of the proposed pretraining method. The benchmark and code are available at https://github.com/birkhoffkiki/CAS-Transformer.

CellVisioner: A Generalizable Cell Virtual Staining Toolbox based on Few-Shot Transfer Learning for Mechanobiological Analysis

Visualizing cellular structures especially the cytoskeleton and the nucleus is crucial for understanding mechanobiology, but traditional fluorescence staining has inherent limitations such as phototoxicity and photobleaching. Virtual staining techniques provide an alternative approach to addressing these issues but often require substantial amount of user training data. In this study, we develop a generalizable cell virtual staining toolbox (termed CellVisioner) based on few-shot transfer learning that requires substantially reduced user training data. CellVisioner can virtually stain F-actin and nuclei for various types of cells and extract single-cell parameters relevant to mechanobiology research. Taking the label-free single-cell images as input, CellVisioner can predict cell mechanobiological status (e.g., Yes-associated protein nuclear/cytoplasmic ratio) and perform long-term monitoring for living cells. We envision that CellVisioner would be a powerful tool to facilitate on-site mechanobiological research.

Roadmap on Label-Free Super-Resolution Imaging

Label-free super-resolution (LFSR) imaging relies on light-scattering processes in nanoscale objects without a need for fluorescent (FL) staining required in super-resolved FL microscopy. The objectives of this Roadmap are to present a comprehensive vision of the developments, the state-of-the-art in this field, and to discuss the resolution boundaries and hurdles which need to be overcome to break the classical diffraction limit of the LFSR imaging. The scope of this Roadmap spans from the advanced interference detection techniques, where the diffraction-limited lateral resolution is combined with unsurpassed axial and temporal resolution, to techniques with true lateral super-resolution capability which are based on understanding resolution as an information science problem, on using novel structured illumination, near-field scanning, and nonlinear optics approaches, and on designing superlenses based on nanoplasmonics, metamaterials, transformation optics, and microsphere-assisted approaches. To this end, this Roadmap brings under the same umbrella researchers from the physics and biomedical optics communities in which such studies have often been developing separately. The ultimate intent of this paper is to create a vision for the current and future developments of LFSR imaging based on its physical mechanisms and to create a great opening for the series of articles in this field.

Virtual Fluorescence Translation for Biological Tissue by Conditional Generative Adversarial Network

Fluorescence labeling and imaging provide an opportunity to observe the structure of biological tissues, playing a crucial role in the field of histopathology. However, when labeling and imaging biological tissues, there are still some challenges, e.g., time-consuming tissue preparation steps, expensive reagents, and signal bias due to photobleaching. To overcome these limitations, we present a deep-learning-based method for fluorescence translation of tissue sections, which is achieved by conditional generative adversarial network (cGAN). Experimental results from mouse kidney tissues demonstrate that the proposed method can predict the other types of fluorescence images from one raw fluorescence image, and implement the virtual multi-label fluorescent staining by merging the generated different fluorescence images as well. Moreover, this proposed method can also effectively reduce the time-consuming and laborious preparation in imaging processes, and further saves the cost and time.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43657-023-00094-1.

Deep learning-enabled virtual histological staining of biological samples

Histological staining is the gold standard for tissue examination in clinical pathology and life-science research, which visualizes the tissue and cellular structures using chromatic dyes or fluorescence labels to aid the microscopic assessment of tissue. However, the current histological staining workflow requires tedious sample preparation steps, specialized laboratory infrastructure, and trained histotechnologists, making it expensive, time-consuming, and not accessible in resource-limited settings. Deep learning techniques created new opportunities to revolutionize staining methods by digitally generating histological stains using trained neural networks, providing rapid, cost-effective, and accurate alternatives to standard chemical staining methods. These techniques, broadly referred to as virtual staining, were extensively explored by multiple research groups and demonstrated to be successful in generating various types of histological stains from label-free microscopic images of unstained samples; similar approaches were also used for transforming images of an already stained tissue sample into another type of stain, performing virtual stain-to-stain transformations. In this Review, we provide a comprehensive overview of the recent research advances in deep learning-enabled virtual histological staining techniques. The basic concepts and the typical workflow of virtual staining are introduced, followed by a discussion of representative works and their technical innovations. We also share our perspectives on the future of this emerging field, aiming to inspire readers from diverse scientific fields to further expand the scope of deep learning-enabled virtual histological staining techniques and their applications.

Optofluidic imaging meets deep learning: from merging to emerging

Propelled by the striking advances in optical microscopy and deep learning (DL), the role of imaging in lab-on-a-chip has dramatically been transformed from a silo inspection tool to a quantitative "smart" engine. A suite of advanced optical microscopes now enables imaging over a range of spatial scales (from molecules to organisms) and temporal window (from microseconds to hours). On the other hand, the staggering diversity of DL algorithms has revolutionized image processing and analysis at the scale and complexity that were once inconceivable. Recognizing these exciting but overwhelming developments, we provide a timely review of their latest trends in the context of lab-on-a-chip imaging, or coined optofluidic imaging. More importantly, here we discuss the strengths and caveats of how to adopt, reinvent, and integrate these imaging techniques and DL algorithms in order to tailor different lab-on-a-chip applications. In particular, we highlight three areas where the latest advances in lab-on-a-chip imaging and DL can form unique synergisms: image formation, image analytics and intelligent image-guided autonomous lab-on-a-chip. Despite the on-going challenges, we anticipate that they will represent the next frontiers in lab-on-a-chip imaging that will spearhead new capabilities in advancing analytical chemistry research, accelerating biological discovery, and empowering new intelligent clinical applications.

Label-free cell segmentation of diverse lymphoid tissues in 2D and 3D

Unlocking and quantifying fundamental biological processes through tissue microscopy requires accurate, in situ segmentation of all cells imaged. Currently, achieving this is complex and requires exogenous fluorescent labels that occupy significant spectral bandwidth, increasing the duration and complexity of imaging experiments while limiting the number of channels remaining to address the study's objectives. We demonstrate that the excitation light reflected during routine confocal microscopy contains sufficient information to achieve accurate, label-free cell segmentation in 2D and 3D. This is achieved using a simple convolutional neural network trained to predict the probability that reflected light pixels belong to either nucleus, cytoskeleton, or background classifications. We demonstrate the approach across diverse lymphoid tissues and provide video tutorials demonstrating deployment in Python and MATLAB or via standalone software for Windows.

Ghost translation: an end-to-end ghost imaging approach based on the transformer network

Artificial intelligence has recently been widely used in computational imaging. The deep neural network (DNN) improves the signal-to-noise ratio of the retrieved images, whose quality is otherwise corrupted due to the low sampling ratio or noisy environments. This work proposes a new computational imaging scheme based on the sequence transduction mechanism with the transformer network. The simulation database assists the network in achieving signal translation ability. The experimental single-pixel detector's signal will be 'translated' into a 2D image in an end-to-end manner. High-quality images with no background noise can be retrieved at a sampling ratio as low as 2%. The illumination patterns can be either well-designed speckle patterns for sub-Nyquist imaging or random speckle patterns. Moreover, our method is robust to noise interference. This translation mechanism opens a new direction for DNN-assisted ghost imaging and can be used in various computational imaging scenarios.

Virtual labeling of mitochondria in living cells using correlative imaging and physics-guided deep learning

Mitochondria play a crucial role in cellular metabolism. This paper presents a novel method to visualize mitochondria in living cells without the use of fluorescent markers. We propose a physics-guided deep learning approach for obtaining virtually labeled micrographs of mitochondria from bright-field images. We integrate a microscope's point spread function in the learning of an adversarial neural network for improving virtual labeling. We show results (average Pearson correlation 0.86) significantly better than what was achieved by state-of-the-art (0.71) for virtual labeling of mitochondria. We also provide new insights into the virtual labeling problem and suggest additional metrics for quality assessment. The results show that our virtual labeling approach is a powerful way of segmenting and tracking individual mitochondria in bright-field images, results previously achievable only for fluorescently labeled mitochondria.

Label-free prediction of cell painting from brightfield images

Cell Painting is a high-content image-based assay applied in drug discovery to predict bioactivity, assess toxicity and understand mechanisms of action of chemical and genetic perturbations. We investigate label-free Cell Painting by predicting the five fluorescent Cell Painting channels from brightfield input. We train and validate two deep learning models with a dataset representing 17 batches, and we evaluate on batches treated with compounds from a phenotypic set. The mean Pearson correlation coefficient of the predicted images across all channels is 0.84. Without incorporating features into the model training, we achieved a mean correlation of 0.45 with ground truth features extracted using a segmentation-based feature extraction pipeline. Additionally, we identified 30 features which correlated greater than 0.8 to the ground truth. Toxicity analysis on the label-free Cell Painting resulted a sensitivity of 62.5% and specificity of 99.3% on images from unseen batches. We provide a breakdown of the feature profiles by channel and feature type to understand the potential and limitations of label-free morphological profiling. We demonstrate that label-free Cell Painting has the potential to be used for downstream analyses and could allow for repurposing imaging channels for other non-generic fluorescent stains of more targeted biological interest.

A dominant function of LynB kinase in preventing autoimmunity

Here, we report that the LynB splice variant of the Src-family kinase Lyn exerts a dominant immunosuppressive function in vivo, whereas the LynA isoform is uniquely required to restrain autoimmunity in female mice. We used CRISPR-Cas9 gene editing to constrain lyn splicing and expression, generating single-isoform LynA knockout (LynAKO) or LynBKO mice. Autoimmune disease in total LynKO mice is characterized by production of antinuclear antibodies, glomerulonephritis, impaired B cell development, and overabundance of activated B cells and proinflammatory myeloid cells. Expression of LynA or LynB alone uncoupled the developmental phenotype from the autoimmune disease: B cell transitional populations were restored, but myeloid cells and differentiated B cells were dysregulated. These changes were isoform-specific, sexually dimorphic, and distinct from the complete LynKO. Despite the apparent differences in disease etiology and penetrance, loss of either LynA or LynB had the potential to induce severe autoimmune disease with parallels to human systemic lupus erythematosus (SLE).

Label-Free Dynamic Imaging of Chromatin in Live Cell Nuclei by High-Speed Scattering-Based Interference Microscopy

Chromatin is a DNA-protein complex that is densely packed in the cell nucleus. The nanoscale chromatin compaction plays critical roles in the modulation of cell nuclear processes. However, little is known about the spatiotemporal dynamics of chromatin compaction states because it remains difficult to quantitatively measure the chromatin compaction level in live cells. Here, we demonstrate a strategy, referenced as DYNAMICS imaging, for mapping chromatin organization in live cell nuclei by analyzing the dynamic scattering signal of molecular fluctuations. Highly sensitive optical interference microscopy, coherent brightfield (COBRI) microscopy, is implemented to detect the linear scattering of unlabeled chromatin at a high speed. A theoretical model is established to determine the local chromatin density from the statistical fluctuation of the measured scattering signal. DYNAMICS imaging allows us to reconstruct a speckle-free nucleus map that is highly correlated to the fluorescence chromatin image. Moreover, together with calibration based on nanoparticle colloids, we show that the DYNAMICS signal is sensitive to the chromatin compaction level at the nanoscale. We confirm the effectiveness of DYNAMICS imaging in detecting the condensation and decondensation of chromatin induced by chemical drug treatments. Importantly, the stable scattering signal supports a continuous observation of the chromatin condensation and decondensation processes for more than 1 h. Using this technique, we detect transient and nanoscopic chromatin condensation events occurring on a time scale of a few seconds. Label-free DYNAMICS imaging offers the opportunity to investigate chromatin conformational dynamics and to explore their significance in various gene activities.

Label-Free and In Situ Identification of Cells via Combinational Machine Learning Models

Cell identification and counting in living and coculture systems are crucial in cell interaction studies, but current methods primarily rely on complicated and time-consuming staining techniques. Here, a label-free method to precisely recognize, identify, and instantly count cells in situ in coculture systems via combinational machine learning models s presented. A convolutional neural network (CNN) model is first used to generate virtual images of cell nuclei based on unlabeled phase-contrast images. Coordinates of all the cells are then returned according to the virtual nucleus images using two clustering algorithms. Finally, phase-contrast images of single cells are cropped based on the coordinates and sent into another CNN model for cell-type identification. This combinational approach is highly automatic and efficient, which requires few to no manual annotations of images in the training phase. It shows practical performance in different cell culture conditions including cell ratios, densities, and substrate materials, having great potential in real-time cell tracking and analyzing.

Computational Methods for Single-Cell Imaging and Omics Data Integration

Integrating single cell omics and single cell imaging allows for a more effective characterisation of the underlying mechanisms that drive a phenotype at the tissue level, creating a comprehensive profile at the cellular level. Although the use of imaging data is well established in biomedical research, its primary application has been to observe phenotypes at the tissue or organ level, often using medical imaging techniques such as MRI, CT, and PET. These imaging technologies complement omics-based data in biomedical research because they are helpful for identifying associations between genotype and phenotype, along with functional changes occurring at the tissue level. Single cell imaging can act as an intermediary between these levels. Meanwhile new technologies continue to arrive that can be used to interrogate the genome of single cells and its related omics datasets. As these two areas, single cell imaging and single cell omics, each advance independently with the development of novel techniques, the opportunity to integrate these data types becomes more and more attractive. This review outlines some of the technologies and methods currently available for generating, processing, and analysing single-cell omics- and imaging data, and how they could be integrated to further our understanding of complex biological phenomena like ageing. We include an emphasis on machine learning algorithms because of their ability to identify complex patterns in large multidimensional data.

Light sheet based volume flow cytometry (VFC) for rapid volume reconstruction and parameter estimation on the go

Optical imaging is paramount for disease diagnosis and to access its progression over time. The proposed optical flow imaging (VFC/iLIFE) is a powerful technique that adds new capabilities (3D volume visualization, organelle-level resolution, and multi-organelle screening) to the existing system. Unlike state-of-the-art point-illumination-based biomedical imaging techniques, the sheet-based VFC technique is capable of single-shot sectional visualization, high throughput interrogation, real-time parameter estimation, and instant volume reconstruction with organelle-level resolution of live specimens. The specimen flow system was realized on a multichannel (Y-type) microfluidic chip that enables visualization of organelle distribution in several cells in-parallel at a relatively high flow-rate (2000 nl/min). The calibration of VFC system requires the study of point emitters (fluorescent beads) at physiologically relevant flow-rates (500-2000 nl/min) for determining flow-induced optical aberration in the system point spread function (PSF). Subsequently, the recorded raw images and volumes were computationally deconvolved with flow-variant PSF to reconstruct the cell volume. High throughput investigation of the mitochondrial network in HeLa cancer cell was carried out at sub-cellular resolution in real-time and critical parameters (mitochondria count and size distribution, morphology, entropy, and cell strain statistics) were determined on-the-go. These parameters determine the physiological state of cells, and the changes over-time, revealing the metastatic progression of diseases. Overall, the developed VFC system enables real-time monitoring of sub-cellular organelle organization at a high-throughput with high-content capacity.

Epi-illumination dark-field microscopy enables direct visualization of unlabeled small organisms with high spatial and temporal resolution

Dark-field microscopy is known to offer both high resolution and direct visualization of thin samples. However, its performance and optimization on thick samples is under-explored and so far, only meso-scale information from whole organisms has been demonstrated. In this work, we carefully investigate the difference between trans- and epi-illumination configurations. Our findings suggest that the epi-illumination configuration is superior in both contrast and fidelity compared to trans-illumination, while having the added advantage of experimental simplicity and an "open top" for experimental intervention. Guided by the theoretical analysis, we constructed an epi-illumination dark-field microscope with measured lateral and axial resolutions of 260 nm and 520 nm, respectively. Subcellular structures in whole organisms were directly visualized without the need for image reconstruction, and further confirmed via simultaneous fluorescence imaging. With an imaging speed of 20 to 50 fps, we visualize fast dynamic processes such as cell division and pharyngeal pumping in Caenorhabditis elegans.

Label-free multiplexed microtomography of endogenous subcellular dynamics using generalizable deep learning

Simultaneous imaging of various facets of intact biological systems across multiple spatiotemporal scales is a long-standing goal in biology and medicine, for which progress is hindered by limits of conventional imaging modalities. Here we propose using the refractive index (RI), an intrinsic quantity governing light-matter interaction, as a means for such measurement. We show that major endogenous subcellular structures, which are conventionally accessed via exogenous fluorescence labelling, are encoded in three-dimensional (3D) RI tomograms. We decode this information in a data-driven manner, with a deep learning-based model that infers multiple 3D fluorescence tomograms from RI measurements of the corresponding subcellular targets, thereby achieving multiplexed microtomography. This approach, called RI2FL for refractive index to fluorescence, inherits the advantages of both high-specificity fluorescence imaging and label-free RI imaging. Importantly, full 3D modelling of absolute and unbiased RI improves generalization, such that the approach is applicable to a broad range of new samples without retraining to facilitate immediate applicability. The performance, reliability and scalability of this technology are extensively characterized, and its various applications within single-cell profiling at unprecedented scales (which can generate new experimentally testable hypotheses) are demonstrated.

An Integrated Photoelectrochemical Nanotool for Intracellular Drug Delivery and Evaluation of Treatment Effect

With reduced background and high sensitivity, photoelectrochemistry (PEC) may be applied as an intracellular nanotool and open a new technological direction of single-cell study. Nevertheless, the present palette of single-cell tools lacks such a PEC-oriented solution. Here a dual-functional photocathodic single-cell nanotool capable of direct electroosmotic intracellular drug delivery and evaluation of oxidative stress is devised by engineering a target-specific organic molecule/NiO/Ni film at the tip of a nanopipette. Specifically, the organic molecule probe serves simultaneously as the biorecognition element and sensitizer to synergize with p-type NiO. Upon intracellular delivery at picoliter level, the oxidative stress effect will cause structural change of the organic probe, switching its optical absorption and altering the cathodic response. This work has revealed the potential of PEC single-cell nanotool and extended the boundary of current single-cell electroanalysis.

Adiels C, Romeo S, Midtvedt D, Volpe G. Extracting quantitative biological information from bright-field cell images using deep learning

Quantitative analysis of cell structures is essential for biomedical and pharmaceutical research. The standard imaging approach relies on fluorescence microscopy, where cell structures of interest are labeled by chemical staining techniques. However, these techniques are often invasive and sometimes even toxic to the cells, in addition to being time consuming, labor intensive, and expensive. Here, we introduce an alternative deep-learning-powered approach based on the analysis of bright-field images by a conditional generative adversarial neural network (cGAN). We show that this is a robust and fast-converging approach to generate virtually stained images from the bright-field images and, in subsequent downstream analyses, to quantify the properties of cell structures. Specifically, we train a cGAN to virtually stain lipid droplets, cytoplasm, and nuclei using bright-field images of human stem-cell-derived fat cells (adipocytes), which are of particular interest for nanomedicine and vaccine development. Subsequently, we use these virtually stained images to extract quantitative measures about these cell structures. Generating virtually stained fluorescence images is less invasive, less expensive, and more reproducible than standard chemical staining; furthermore, it frees up the fluorescence microscopy channels for other analytical probes, thus increasing the amount of information that can be extracted from each cell. To make this deep-learning-powered approach readily available for other users, we provide a Python software package, which can be easily personalized and optimized for specific virtual-staining and cell-profiling applications.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Interpretable deep learning uncovers cellular properties in label-free live cell images that are predictive of highly metastatic melanoma

Deep learning has emerged as the technique of choice for identifying hidden patterns in cell imaging data but is often criticized as "black box." Here, we employ a generative neural network in combination with supervised machine learning to classify patient-derived melanoma xenografts as "efficient" or "inefficient" metastatic, validate predictions regarding melanoma cell lines with unknown metastatic efficiency in mouse xenografts, and use the network to generate in silico cell images that amplify the critical predictive cell properties. These exaggerated images unveiled pseudopodial extensions and increased light scattering as hallmark properties of metastatic cells. We validated this interpretation using live cells spontaneously transitioning between states indicative of low and high metastatic efficiency. This study illustrates how the application of artificial intelligence can support the identification of cellular properties that are predictive of complex phenotypes and integrated cell functions but are too subtle to be identified in the raw imagery by a human expert. A record of this paper's transparent peer review process is included in the supplemental information. VIDEO ABSTRACT.

Image3C, a multimodal image-based and label-independent integrative method for single-cell analysis

Image-based cell classification has become a common tool to identify phenotypic changes in cell populations. However, this methodology is limited to organisms possessing well-characterized species-specific reagents (e.g., antibodies) that allow cell identification, clustering, and convolutional neural network (CNN) training. In the absence of such reagents, the power of image-based classification has remained mostly off-limits to many research organisms. We have developed an image-based classification methodology we named Image3C (Image-Cytometry Cell Classification) that does not require species-specific reagents nor pre-existing knowledge about the sample. Image3C combines image-based flow cytometry with an unbiased, high-throughput cell clustering pipeline and CNN integration. Image3C exploits intrinsic cellular features and non-species-specific dyes to perform de novo cell composition analysis and detect changes between different conditions. Therefore, Image3C expands the use of image-based analyses of cell population composition to research organisms in which detailed cellular phenotypes are unknown or for which species-specific reagents are not available.

Image-based cell phenotyping with deep learning

A cell's phenotype is the culmination of several cellular processes through a complex network of molecular interactions that ultimately result in a unique morphological signature. Visual cell phenotyping is the characterization and quantification of these observable cellular traits in images. Recently, cellular phenotyping has undergone a massive overhaul in terms of scale, resolution, and throughput, which is attributable to advances across electronic, optical, and chemical technologies for imaging cells. Coupled with the rapid acceleration of deep learning-based computational tools, these advances have opened up new avenues for innovation across a wide variety of high-throughput cell biology applications. Here, we review applications wherein deep learning is powering the recognition, profiling, and prediction of visual phenotypes to answer important biological questions. As the complexity and scale of imaging assays increase, deep learning offers computational solutions to elucidate the details of previously unexplored cellular phenotypes.

Data science in cell imaging

Cell imaging has entered the 'Big Data' era. New technologies in light microscopy and molecular biology have led to an explosion in high-content, dynamic and multidimensional imaging data. Similar to the 'omics' fields two decades ago, our current ability to process, visualize, integrate and mine this new generation of cell imaging data is becoming a critical bottleneck in advancing cell biology. Computation, traditionally used to quantitatively test specific hypotheses, must now also enable iterative hypothesis generation and testing by deciphering hidden biologically meaningful patterns in complex, dynamic or high-dimensional cell image data. Data science is uniquely positioned to aid in this process. In this Perspective, we survey the rapidly expanding new field of data science in cell imaging. Specifically, we highlight how data science tools are used within current image analysis pipelines, propose a computation-first approach to derive new hypotheses from cell image data, identify challenges and describe the next frontiers where we believe data science will make an impact. We also outline steps to ensure broad access to these powerful tools - democratizing infrastructure availability, developing sensitive, robust and usable tools, and promoting interdisciplinary training to both familiarize biologists with data science and expose data scientists to cell imaging.