Content-aware image restoration: pushing the limits of fluorescence microscopy

Joint segmentation and image reconstruction with error prediction in photoacoustic imaging using deep learning

Deep learning has been used to improve photoacoustic (PA) image reconstruction. One major challenge is that errors cannot be quantified to validate predictions when ground truth is unknown. Validation is key to quantitative applications, especially using limited-bandwidth ultrasonic linear detector arrays. Here, we propose a hybrid Bayesian convolutional neural network (Hybrid-BCNN) to jointly predict PA image and segmentation with error (uncertainty) predictions. Each output pixel represents a probability distribution where error can be quantified. The Hybrid-BCNN was trained with simulated PA data and applied to both simulations and experiments. Due to the sparsity of PA images, segmentation focuses Hybrid-BCNN on minimizing the loss function in regions with PA signals for better predictions. The results show that accurate PA segmentations and images are obtained, and error predictions are highly statistically correlated to actual errors. To leverage error predictions, confidence processing created PA images above a specific confidence level.

This Microtubule Does Not Exist: Super-Resolution Microscopy Image Generation by a Diffusion Model

Generative models, such as diffusion models, have made significant advancements in recent years, enabling the synthesis of high-quality realistic data across various domains. Here, the adaptation and training of a diffusion model on super-resolution microscopy images are explored. It is shown that the generated images resemble experimental images, and that the generation process does not exhibit a large degree of memorization from existing images in the training set. To demonstrate the usefulness of the generative model for data augmentation, the performance of a deep learning-based single-image super-resolution (SISR) method trained using generated high-resolution data is compared against training using experimental images alone, or images generated by mathematical modeling. Using a few experimental images, the reconstruction quality and the spatial resolution of the reconstructed images are improved, showcasing the potential of diffusion model image generation for overcoming the limitations accompanying the collection and annotation of microscopy images. Finally, the pipeline is made publicly available, runnable online, and user-friendly to enable researchers to generate their own synthetic microscopy data. This work demonstrates the potential contribution of generative diffusion models for microscopy tasks and paves the way for their future application in this field.

TEX264 drives selective autophagy of DNA lesions to promote DNA repair and cell survival

DNA repair and autophagy are distinct biological processes vital for cell survival. Although autophagy helps maintain genome stability, there is no evidence of its direct role in the repair of DNA lesions. We discovered that lysosomes process topoisomerase 1 cleavage complexes (TOP1cc) DNA lesions in vertebrates. Selective degradation of TOP1cc by autophagy directs DNA damage repair and cell survival at clinically relevant doses of topoisomerase 1 inhibitors. TOP1cc are exported from the nucleus to lysosomes through a transient alteration of the nuclear envelope and independent of the proteasome. Mechanistically, the autophagy receptor TEX264 acts as a TOP1cc sensor at DNA replication forks, triggering TOP1cc processing by the p97 ATPase and mediating the delivery of TOP1cc to lysosomes in an MRE11-nuclease- and ATR-kinase-dependent manner. We found an evolutionarily conserved role for selective autophagy in DNA repair that enables cell survival, protects genome stability, and is clinically relevant for colorectal cancer patients.

Mitochondrial Structure, Dynamics, and Physiology: Light Microscopy to Disentangle the Network

Mitochondria serve as energetic and signaling hubs of the cell: This function results from the complex interplay between their structure, function, dynamics, interactions, and molecular organization. The ability to observe and quantify these properties often represents the puzzle piece critical for deciphering the mechanisms behind mitochondrial function and dysfunction. Fluorescence microscopy addresses this critical need and has become increasingly powerful with the advent of superresolution methods and context-sensitive fluorescent probes. In this review, we delve into advanced light microscopy methods and analyses for studying mitochondrial ultrastructure, dynamics, and physiology, and highlight notable discoveries they enabled.

Self-inspired learning for denoising live-cell super-resolution microscopy

Every collected photon is precious in live-cell super-resolution (SR) microscopy. Here, we describe a data-efficient, deep learning-based denoising solution to improve diverse SR imaging modalities. The method, SN2N, is a Self-inspired Noise2Noise module with self-supervised data generation and self-constrained learning process. SN2N is fully competitive with supervised learning methods and circumvents the need for large training set and clean ground truth, requiring only a single noisy frame for training. We show that SN2N improves photon efficiency by one-to-two orders of magnitude and is compatible with multiple imaging modalities for volumetric, multicolor, time-lapse SR microscopy. We further integrated SN2N into different SR reconstruction algorithms to effectively mitigate image artifacts. We anticipate SN2N will enable improved live-SR imaging and inspire further advances.

Cell dynamics revealed by microscopy advances

Cell biology emerges from spatiotemporally coordinated molecular processes. Recent advances in live-cell microscopy, fueled by a surge in optical, molecular, and computational technologies, have enabled dynamic observations from single molecules to whole organisms. Despite technological leaps, there is still an untapped opportunity to fully leverage their capabilities toward biological insight. We highlight how single-molecule imaging has transformed our understanding of biological processes, with a focus on chromatin organization and transcription in the nucleus. We describe how this was enabled by the close integration of new imaging techniques with analysis tools and discuss the challenges to make a comparable impact at larger scales from organelles to organisms. By highlighting recent successful examples, we describe an outlook of ever-increasing data and the need for seamless integration between dataset visualization and quantification to realize the full potential warranted by advances in new imaging technologies.

A dual-mode, image-enhanced, miniaturized microscopy system for incubator-compatible monitoring of live cells

Imaging live cells under stable culture conditions is essential to investigate cell physiological activities and proliferation. To achieve this goal, typically, a specialized incubation chamber that creates desired culture conditions needs to be incorporated into a microscopy system to perform cell monitoring. However, such imaging systems are generally large and costly, hampering their wide applications. Recent advances in the field of miniaturized microscopy systems have enabled incubator cell monitoring, providing a hospitable environment for live cells. Although these systems are more cost-effective, they are usually limited in imaging modalities and spatial temporal resolution. Here, we present a dual-mode, image-enhanced, miniaturized microscopy system (termed MiniCube) for direct monitoring of live cells inside incubators. MiniCube enables both bright field imaging and fluorescence imaging with single-cell spatial resolution and sub-second temporal resolution. Moreover, this system can also perform cell monitoring inside the incubator with tunable time scales ranging from a few seconds to days. Meanwhile, automatic cell segmentation and image enhancement are realized by the proposed data analysis pipeline of this system, and the signal-to-noise ratio (SNR) of acquired data is significantly improved using a deep learning based image denoising algorithm. Image data can be acquired with 5 times lower light exposure while maintaining comparable SNR. The versatility of this miniaturized microscopy system lends itself to various applications in biology studies, providing a practical platform and method for studying live cell dynamics within the incubator.

Arkitekt: streaming analysis and real-time workflows for microscopy

Quantitative microscopy workflows have evolved dramatically over the past years, progressively becoming more complex with the emergence of deep learning. Long-standing challenges such as three-dimensional segmentation of complex microscopy data can finally be addressed, and new imaging modalities are breaking records in both resolution and acquisition speed, generating gigabytes if not terabytes of data per day. With this shift in bioimage workflows comes an increasing need for efficient orchestration and data management, necessitating multitool interoperability and the ability to span dedicated computing resources. However, existing solutions are still limited in their flexibility and scalability and are usually restricted to offline analysis. Here we introduce Arkitekt, an open-source middleman between users and bioimage apps that enables complex quantitative microscopy workflows in real time. It allows the orchestration of popular bioimage software locally or remotely in a reliable and efficient manner. It includes visualization and analysis modules, but also mechanisms to execute source code and pilot acquisition software, making 'smart microscopy' a reality.

Neural space-time model for dynamic multi-shot imaging

Computational imaging reconstructions from multiple measurements that are captured sequentially often suffer from motion artifacts if the scene is dynamic. We propose a neural space-time model (NSTM) that jointly estimates the scene and its motion dynamics, without data priors or pre-training. Hence, we can both remove motion artifacts and resolve sample dynamics from the same set of raw measurements used for the conventional reconstruction. We demonstrate NSTM in three computational imaging systems: differential phase-contrast microscopy, three-dimensional structured illumination microscopy and rolling-shutter DiffuserCam. We show that NSTM can recover subcellular motion dynamics and thus reduce the misinterpretation of living systems caused by motion artifacts.

STORM image denoising and information extraction

Stochastic optical reconstruction microscopy (STORM) is extensively utilized in the fields of cell and molecular biology as a super-resolution imaging technique for visualizing cells and molecules. Nonetheless, the imaging process of STORM is frequently susceptible to noise, which can significantly impact the subsequent image analysis. Moreover, there is currently a lack of a comprehensive automated processing approach for analyzing protein aggregation states from a large number of STORM images. This paper initially applies our previously proposed denoising algorithm, UNet-Att, in STORM image denoising. This algorithm was constructed based on attention mechanism and multi-scale features, showcasing a remarkably efficient performance in denoising. Subsequently, we propose a collection of automated image processing algorithms for the ultimate feature extractions and data analyses of the STORM images. The information extraction workflow effectively integrates automated methods of image denoising, objective image segmentation and binarization, and object information extraction, and a novel image information clustering algorithm specifically developed for the morphological analysis of the objects in the STORM images. This automated workflow significantly improves the efficiency of the effective data analysis for large-scale original STORM images.

Visualizing PIEZO1 Localization and Activity in hiPSC-Derived Single Cells and Organoids with HaloTag Technology

PIEZO1 is critical to numerous physiological processes, transducing diverse mechanical stimuli into electrical and chemical signals. Recent studies underscore the importance of visualizing endogenous PIEZO1 activity and localization to understand its functional roles. To enable physiologically and clinically relevant studies on human PIEZO1, we genetically engineered human induced pluripotent stem cells (hiPSCs) to express a HaloTag fused to endogenous PIEZO1. Combined with advanced imaging, our chemogenetic platform allows precise visualization of PIEZO1 localization dynamics in various cell types. Furthermore, the PIEZO1-HaloTag hiPSC technology facilitates the non-invasive monitoring of channel activity across diverse cell types using Ca2+-sensitive HaloTag ligands, achieving temporal resolution approaching that of patch clamp electrophysiology. Finally, we used lightsheet imaging of hiPSC-derived neural organoids to achieve molecular scale imaging of PIEZO1 in three-dimensional tissue organoids. Our advances offer a novel platform for studying PIEZO1 mechanotransduction in human cells and tissues, with potential for elucidating disease mechanisms and targeted therapeutic development.

Computational Methods for Image Analysis in Craniofacial Development and Disease

Observation is at the center of all biological sciences. Advances in imaging technologies are therefore essential to derive novel biological insights to better understand the complex workings of living systems. Recent high-throughput sequencing and imaging techniques are allowing researchers to simultaneously address complex molecular variations spatially and temporarily in tissues and organs. The availability of increasingly large dataset sizes has allowed for the evolution of robust deep learning models, designed to interrogate biomedical imaging data. These models are emerging as transformative tools in diagnostic medicine. Combined, these advances allow for dynamic, quantitative, and predictive observations of entire organisms and tissues. Here, we address 3 main tasks of bioimage analysis, image restoration, segmentation, and tracking and discuss new computational tools allowing for 3-dimensional spatial genomics maps. Finally, we demonstrate how these advances have been applied in studies of craniofacial development and oral disease pathogenesis.

Fast Multiphoton Microscopic Imaging Joint Image Super-Resolution for Automated Gleason Grading of Prostate Cancers

Gleason grading system is dependable for quantifying prostate cancer. This paper introduces a fast multiphoton microscopic imaging method via deep learning for automatic Gleason grading. Due to the contradiction between multiphoton microscopy (MPM) imaging speed and quality, a deep learning architecture (SwinIR) is used for image super-resolution to address this issue. The quality of low-resolution image is improved, which increased the acquisition speed from 7.55 s per frame to 0.24 s per frame. A classification network (Swin Transformer) was introduced for automated Gleason grading. The classification accuracy and Macro-F1 achieved by training on high-resolution images are respectively 90.9% and 90.9%. For training on super-resolution images, the classification accuracy and Macro-F1 are respectively 89.9% and 89.9%. It shows that super-resolution image can provide a comparable performance to high-resolution image. Our results suggested that MPM joint image super-resolution and automatic classification methods hold the potential to be a real-time clinical diagnostic tool for prostate cancer diagnosis.

Deep intravital brain tumor imaging enabled by tailored three-photon microscopy and analysis

Intravital 2P-microscopy enables the longitudinal study of brain tumor biology in superficial mouse cortex layers. Intravital microscopy of the white matter, an important route of glioblastoma invasion and recurrence, has not been feasible, due to low signal-to-noise ratios and insufficient spatiotemporal resolution. Here, we present an intravital microscopy and artificial intelligence-based analysis workflow (Deep3P) that enables longitudinal deep imaging of glioblastoma up to a depth of 1.2 mm. We find that perivascular invasion is the preferred invasion route into the corpus callosum and uncover two vascular mechanisms of glioblastoma migration in the white matter. Furthermore, we observe morphological changes after white matter infiltration, a potential basis of an imaging biomarker during early glioblastoma colonization. Taken together, Deep3P allows for a non-invasive intravital investigation of brain tumor biology and its tumor microenvironment at subcortical depths explored, opening up opportunities for studying the neuroscience of brain tumors and other model systems.

A hierarchically annotated dataset drives tangled filament recognition in digital neuron reconstruction

Reconstructing neuronal morphology is vital for classifying neurons and mapping brain connectivity. However, it remains a significant challenge due to its complex structure, dense distribution, and low image contrast. In particular, AI-assisted methods often yield numerous errors that require extensive manual intervention. Therefore, reconstructing hundreds of neurons is already a daunting task for general research projects. A key issue is the lack of specialized training for challenging regions due to inadequate data and training methods. This study extracted 2,800 challenging neuronal blocks and categorized them into multiple density levels. Furthermore, we enhanced images using an axial continuity-based network that improved three-dimensional voxel resolution while reducing the difficulty of neuron recognition. Comparing the pre- and post-enhancement results in automatic algorithms using fluorescence micro-optical sectioning tomography (fMOST) data, we observed a significant increase in the recall rate. Our study not only enhances the throughput of reconstruction but also provides a fundamental dataset for tangled neuron reconstruction.

3D live imaging and phenotyping of CAR-T cell mediated-cytotoxicity using high-throughput Bessel oblique plane microscopy

Clarification of the cytotoxic function of T cells is crucial for understanding human immune responses and immunotherapy procedures. Here, we report a high-throughput Bessel oblique plane microscopy (HBOPM) platform capable of 3D live imaging and phenotyping of chimeric antigen receptor (CAR)-modified T-cell cytotoxicity against cancer cells. The HBOPM platform has the following characteristics: an isotropic subcellular resolution of 320 nm, large-scale scouting over 400 interacting cell pairs, long-term observation across 5 hours, and quantitative analysis of the Terabyte-scale 3D, multichannel, time-lapse image datasets. Using this advanced microscopy platform, several key subcellular events in CAR-T cells are captured and comprehensively analyzed; these events include the instantaneous formation of immune synapses and the sustained changes in the microtubing morphology. Furthermore, we identify the actin retrograde flow speed, the actin depletion coefficient, the microtubule polarization and the contact area of the CAR-T/target cell conjugates as essential parameters strongly correlated with CAR-T-cell cytotoxic function. Our approach will be useful for establishing criteria for quantifying T-cell function in individual patients for all T-cell-based immunotherapies.

Convolutional neural network transformer (CNNT) for fluorescence microscopy image denoising with improved generalization and fast adaptation

Deep neural networks can improve the quality of fluorescence microscopy images. Previous methods, based on Convolutional Neural Networks (CNNs), require time-consuming training of individual models for each experiment, impairing their applicability and generalization. In this study, we propose a novel imaging-transformer based model, Convolutional Neural Network Transformer (CNNT), that outperforms CNN based networks for image denoising. We train a general CNNT based backbone model from pairwise high-low Signal-to-Noise Ratio (SNR) image volumes, gathered from a single type of fluorescence microscope, an instant Structured Illumination Microscope. Fast adaptation to new microscopes is achieved by fine-tuning the backbone on only 5-10 image volume pairs per new experiment. Results show that the CNNT backbone and fine-tuning scheme significantly reduces training time and improves image quality, outperforming models trained using only CNNs such as 3D-RCAN and Noise2Fast. We show three examples of efficacy of this approach in wide-field, two-photon, and confocal fluorescence microscopy.

Single-detector multiplex imaging flow cytometry for cancer cell classification with deep learning

Imaging flow cytometry, which combines the advantages of flow cytometry and microscopy, has emerged as a powerful tool for cell analysis in various biomedical fields such as cancer detection. In this study, we develop multiplex imaging flow cytometry (mIFC) by employing a spatial wavelength division multiplexing technique. Our mIFC can simultaneously obtain brightfield and multi-color fluorescence images of individual cells in flow, which are excited by a metal halide lamp and measured by a single detector. Statistical analysis results of multiplex imaging experiments with resolution test lens, magnification test lens, and fluorescent microspheres validate the operation of the mIFC with good imaging channel consistency and micron-scale differentiation capabilities. A deep learning method is designed for multiplex image processing that consists of three deep learning networks (U-net, very deep super resolution, and visual geometry group 19). It is demonstrated that the cluster of differentiation 24 (CD24) imaging channel is more sensitive than the brightfield, nucleus, or cancer antigen 125 (CA125) imaging channel in classifying the three types of ovarian cell lines (IOSE80 normal cell, A2780, and OVCAR3 cancer cells). An average accuracy rate of 97.1% is achieved for the classification of these three types of cells by deep learning analysis when all four imaging channels are considered. Our single-detector mIFC is promising for the development of future imaging flow cytometers and for the automatic single-cell analysis with deep learning in various biomedical fields.

In silico labeling in cell biology: Potential and limitations

In silico labeling is the computational cross-modality image translation where the output modality is a subcellular marker that is not specifically encoded in the input image, for example, in silico localization of organelles from transmitted light images. In principle, in silico labeling has the potential to facilitate rapid live imaging of multiple organelles with reduced photobleaching and phototoxicity, a technology enabling a major leap toward understanding the cell as an integrated complex system. However, five years have passed since feasibility was attained, without any demonstration of using in silico labeling to uncover new biological insight. In here, we discuss the current state of in silico labeling, the limitations preventing it from becoming a practical tool, and how we can overcome these limitations to reach its full potential.

Artificial intelligence for high content imaging in drug discovery

Artificial intelligence (AI) and high-content imaging (HCI) are contributing to advancements in drug discovery, propelled by the recent progress in deep neural networks. This review highlights AI's role in analysis of HCI data from fixed and live-cell imaging, enabling novel label-free and multi-channel fluorescent screening methods, and improving compound profiling. HCI experiments are rapid and cost-effective, facilitating large data set accumulation for AI model training. However, the success of AI in drug discovery also depends on high-quality data, reproducible experiments, and robust validation to ensure model performance. Despite challenges like the need for annotated compounds and managing vast image data, AI's potential in phenotypic screening and drug profiling is significant. Future improvements in AI, including increased interpretability and integration of multiple modalities, are expected to solidify AI and HCI's role in drug discovery.

Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration

Fluorescence microscopy-based image restoration has received widespread attention in the life sciences and has led to significant progress, benefiting from deep learning technology. However, most current task-specific methods have limited generalizability to different fluorescence microscopy-based image restoration problems. Here, we seek to improve generalizability and explore the potential of applying a pretrained foundation model to fluorescence microscopy-based image restoration. We provide a universal fluorescence microscopy-based image restoration (UniFMIR) model to address different restoration problems, and show that UniFMIR offers higher image restoration precision, better generalization and increased versatility. Demonstrations on five tasks and 14 datasets covering a wide range of microscopy imaging modalities and biological samples demonstrate that the pretrained UniFMIR can effectively transfer knowledge to a specific situation via fine-tuning, uncover clear nanoscale biomolecular structures and facilitate high-quality imaging. This work has the potential to inspire and trigger new research highlights for fluorescence microscopy-based image restoration.

The mammalian actin elongation factor ENAH/MENA contributes to autophagosome formation via its actin regulatory function

Macroautophagy/autophagy is a catabolic process crucial for degrading cytosolic components and damaged organelles to maintain cellular homeostasis, enabling cells to survive in extreme extracellular environments. ENAH/MENA, a member of the Ena/VASP protein family, functions as a highly efficient actin elongation factor. In this study, our objective was to explore the role of ENAH in the autophagy process. Initially, we demonstrated that depleting ENAH in cancer cells inhibits autophagosome formation. Subsequently, we observed ENAH's colocalization with MAP1LC3/LC3 during tumor cell starvation, dependent on actin cytoskeleton polymerization and the interaction between ENAH and BECN1 (beclin 1). Additionally, mammalian ATG9A formed a ring-like structure around ENAH-LC3 puncta during starvation, relying on actin cytoskeleton polymerization. Furthermore, ENAH's EVH1 and EVH2 domains were found to be indispensable for its colocalization with LC3 and BECN1, while the PRD domain played a crucial role in the formation of the ATG9A ring. Finally, our study revealed ENAH-led actin comet tails in autophagosome trafficking. In conclusion, our findings provide initial insights into the regulatory role of the mammalian actin elongation factor ENAH in autophagy.Abbreviations: 3-MA 3-methyladenine; ABPs actin-binding proteins; ATG autophagy related; ATG9A autophagy related 9A; Baf A1 bafilomycin A1; CM complete medium; CytERM endoplasmic reticulum signal-anchor membrane protein; Cyto D cytochalasin D; EBSS Earl's balanced salt solution; ENAH/MENA ENAH actin regulator; EVH1 Ena/VASP homology 1 domain; EVH2 Ena/VASP homology 2 domain; GAPDH glyceraldehyde-3-phosphate dehydrogenase; Lat B latrunculin B; LC3-I unlipidated form of LC3; LC3-II phosphatidylethanolamine-conjugated form of LC3; MAP1LC3/LC3 microtubule associated protein 1 light chain 3; mEGFP monomeric enhanced green fluorescent protein; mTagBFP2 monomeric Tag blue fluorescent protein 2; OSER organized smooth endoplasmic reticulum; PRD proline-rich domain; PtdIns3K class III phosphatidylinositol 3-kinase; WM wortmannin.

Epidermal turnover in the planarian Schmidtea mediterranea involves basal cell extrusion and intestinal digestion

Planarian flatworms undergo continuous internal turnover, wherein old cells are replaced by the division progeny of adult pluripotent stem cells (neoblasts). How cell turnover is carried out at the organismal level remains an intriguing question in planarians and other systems. While previous studies have predominantly focused on neoblast proliferation, little is known about the processes that mediate cell loss during tissue homeostasis. Here, we use the planarian epidermis as a model to study the mechanisms of cell removal. We established a covalent dye-labeling assay and image analysis pipeline to quantify the cell turnover rate in the planarian epidermis. Our findings indicate that the ventral epidermis is highly dynamic and epidermal cells undergo internalization via basal extrusion, followed by a relocation toward the intestine and ultimately digestion by intestinal phagocytes. Overall, our study reveals a complex homeostatic process of cell clearance that may generally allow planarians to catabolize their own cells.

Piezo regulates epithelial topology and promotes precision in organ size control

Mechanosensitive Piezo channels regulate cell division, cell extrusion, and cell death. However, systems-level functions of Piezo in regulating organogenesis remain poorly understood. Here, we demonstrate that Piezo controls epithelial cell topology to ensure precise organ growth by integrating live-imaging experiments with pharmacological and genetic perturbations and computational modeling. Notably, the knockout or knockdown of Piezo increases bilateral asymmetry in wing size. Piezo's multifaceted functions can be deconstructed as either autonomous or non-autonomous based on a comparison between tissue-compartment-level perturbations or between genetic perturbation populations at the whole-tissue level. A computational model that posits cell proliferation and apoptosis regulation through modulation of the cutoff tension required for Piezo channel activation explains key cell and tissue phenotypes arising from perturbations of Piezo expression levels. Our findings demonstrate that Piezo promotes robustness in regulating epithelial topology and is necessary for precise organ size control.

A cellular tilting mechanism important for dynamic tissue shape changes and cell differentiation in Drosophila

Dynamic changes in three-dimensional cell shape are important for tissue form and function. In the developing Drosophila eye, photoreceptor differentiation requires the progression across the tissue of an epithelial fold known as the morphogenetic furrow. Morphogenetic furrow progression involves apical cell constriction and movement of apical cell edges. Here, we show that cells progressing through the morphogenetic furrow move their basal edges in opposite direction to their apical edges, resulting in a cellular tilting movement. We further demonstrate that cells generate, at their basal side, oriented, force-generating protrusions. Knockdown of the protein kinase Src42A or photoactivation of a dominant-negative form of the small GTPase Rac1 reduces protrusion formation. Impaired protrusion formation stalls basal cell movement and slows down morphogenetic furrow progression and photoreceptor differentiation. This work identifies a cellular tilting mechanism important for the generation of dynamic tissue shape changes and cell differentiation.

Deep learning-based aberration compensation improves contrast and resolution in fluorescence microscopy

Optical aberrations hinder fluorescence microscopy of thick samples, reducing image signal, contrast, and resolution. Here we introduce a deep learning-based strategy for aberration compensation, improving image quality without slowing image acquisition, applying additional dose, or introducing more optics into the imaging path. Our method (i) introduces synthetic aberrations to images acquired on the shallow side of image stacks, making them resemble those acquired deeper into the volume and (ii) trains neural networks to reverse the effect of these aberrations. We use simulations and experiments to show that applying the trained 'de-aberration' networks outperforms alternative methods, providing restoration on par with adaptive optics techniques; and subsequently apply the networks to diverse datasets captured with confocal, light-sheet, multi-photon, and super-resolution microscopy. In all cases, the improved quality of the restored data facilitates qualitative image inspection and improves downstream image quantitation, including orientational analysis of blood vessels in mouse tissue and improved membrane and nuclear segmentation in C. elegans embryos.

BirthSeq, a new method to isolate and analyze dated cells in different vertebrates

Embryonic development is a complex and dynamic process that unfolds over time and involves the production and diversification of increasing numbers of cells. The impact of developmental time on the formation of the central nervous system is well documented, with evidence showing that time plays a crucial role in establishing the identity of neuronal subtypes. However, the study of how time translates into genetic instructions driving cell fate is limited by the scarcity of suitable experimental tools. We introduce BirthSeq, a new method for isolating and analyzing cells based on their birth date. This innovative technique allows for in vivo labeling of cells, isolation via fluorescence-activated cell sorting, and analysis using high-throughput techniques. We calibrated the BirthSeq method for developmental organs across three vertebrate species (mouse, chick and gecko), and utilized it for single-cell RNA sequencing and novel spatially resolved transcriptomic approaches in mouse and chick, respectively. Overall, BirthSeq provides a versatile tool for studying virtually any tissue in different vertebrate organisms, aiding developmental biology research by targeting cells and their temporal cues.

Deep 3D histology powered by tissue clearing, omics and AI

To comprehensively understand tissue and organism physiology and pathophysiology, it is essential to create complete three-dimensional (3D) cellular maps. These maps require structural data, such as the 3D configuration and positioning of tissues and cells, and molecular data on the constitution of each cell, spanning from the DNA sequence to protein expression. While single-cell transcriptomics is illuminating the cellular and molecular diversity across species and tissues, the 3D spatial context of these molecular data is often overlooked. Here, I discuss emerging 3D tissue histology techniques that add the missing third spatial dimension to biomedical research. Through innovations in tissue-clearing chemistry, labeling and volumetric imaging that enhance 3D reconstructions and their synergy with molecular techniques, these technologies will provide detailed blueprints of entire organs or organisms at the cellular level. Machine learning, especially deep learning, will be essential for extracting meaningful insights from the vast data. Further development of integrated structural, molecular and computational methods will unlock the full potential of next-generation 3D histology.

Inspiring a convergent engineering approach to measure and model the tissue microenvironment

Understanding the molecular and physical complexity of the tissue microenvironment (TiME) in the context of its spatiotemporal organization has remained an enduring challenge. Recent advances in engineering and data science are now promising the ability to study the structure, functions, and dynamics of the TiME in unprecedented detail; however, many advances still occur in silos that rarely integrate information to study the TiME in its full detail. This review provides an integrative overview of the engineering principles underlying chemical, optical, electrical, mechanical, and computational science to probe, sense, model, and fabricate the TiME. In individual sections, we first summarize the underlying principles, capabilities, and scope of emerging technologies, the breakthrough discoveries enabled by each technology and recent, promising innovations. We provide perspectives on the potential of these advances in answering critical questions about the TiME and its role in various disease and developmental processes. Finally, we present an integrative view that appreciates the major scientific and educational aspects in the study of the TiME.

SmartEM: machine-learning guided electron microscopy

Connectomics provides essential nanometer-resolution, synapse-level maps of neural circuits to understand brain activity and behavior. However, few researchers have access to the high-throughput electron microscopes necessary to generate enough data for whole circuit or brain reconstruction. To date, machine-learning methods have been used after the collection of images by electron microscopy (EM) to accelerate and improve neuronal segmentation, synapse reconstruction and other data analysis. With the computational improvements in processing EM images, acquiring EM images has now become the rate-limiting step. Here, in order to speed up EM imaging, we integrate machine-learning into real-time image acquisition in a singlebeam scanning electron microscope. This SmartEM approach allows an electron microscope to perform intelligent, data-aware imaging of specimens. SmartEM allocates the proper imaging time for each region of interest - scanning all pixels equally rapidly, then re-scanning small subareas more slowly where a higher quality signal is required to achieve accurate segmentability, in significantly less time. We demonstrate that this pipeline achieves a 7-fold acceleration of image acquisition time for connectomics using a commercial single-beam SEM. We apply SmartEM to reconstruct a portion of mouse cortex with the same accuracy as traditional microscopy but in less time.

Advanced Design of Soft Robots with Artificial Intelligence

A comprehensive review focused on the whole systems of the soft robotics with artificial intelligence, which can feel, think, react and interact with humans, is presented. The design strategies concerning about various aspects of the soft robotics, like component materials, device structures, prepared technologies, integrated method, and potential applications, are summarized. A broad outlook on the future considerations for the soft robots is proposed.

In recent years, breakthrough has been made in the field of artificial intelligence (AI), which has also revolutionized the industry of robotics. Soft robots featured with high-level safety, less weight, lower power consumption have always been one of the research hotspots. Recently, multifunctional sensors for perception of soft robotics have been rapidly developed, while more algorithms and models of machine learning with high accuracy have been optimized and proposed. Designs of soft robots with AI have also been advanced ranging from multimodal sensing, human–machine interaction to effective actuation in robotic systems. Nonetheless, comprehensive reviews concerning the new developments and strategies for the ingenious design of the soft robotic systems equipped with AI are rare. Here, the new development is systematically reviewed in the field of soft robots with AI. First, background and mechanisms of soft robotic systems are briefed, after which development focused on how to endow the soft robots with AI, including the aspects of feeling, thought and reaction, is illustrated. Next, applications of soft robots with AI are systematically summarized and discussed together with advanced strategies proposed for performance enhancement. Design thoughts for future intelligent soft robotics are pointed out. Finally, some perspectives are put forward.

RESPAN: an accurate, unbiased and automated pipeline for analysis of dendritic morphology and dendritic spine mapping

Accurate and unbiased reconstructions of neuronal morphology, including quantification of dendritic spine morphology and distribution, are widely used in neuroscience but remain a major roadblock for large-scale analysis. Traditionally, spine analysis has required labor-intensive manual annotation, which is prone to human error and impractical for large 3D datasets. Previous automated tools for reconstructing neuronal morphology and quantitative dendritic spine analysis face challenges in generating accurate results and, following close inspection, often require extensive manual correction. While recent tools leveraging deep learning approaches have substantially increased accuracy, they lack functionality and useful outputs, necessitating additional tools to perform a complete analysis and limiting their utility. In this paper, we describe Restoration Enhanced SPine And Neuron (RESPAN) analysis, a new comprehensive pipeline developed as an open-source, easily deployable solution that harnesses recent advances in deep learning and GPU processing. Our approach demonstrates high accuracy and robustness, validated extensively across a range of imaging modalities for automated dendrite and spine mapping. It also offers extensive visual and tabulated data outputs, including detailed morphological and spatial metrics, dendritic spine classification, and 3D renderings. Additionally, RESPAN includes tools for validating results, ensuring scientific rigor and reproducibility.

Depth-enhanced high-throughput microscopy by compact PSF engineering

High-throughput microscopy is vital for screening applications, where three-dimensional (3D) cellular models play a key role. However, due to defocus susceptibility, current 3D high-throughput microscopes require axial scanning, which lowers throughput and increases photobleaching and photodamage. Point spread function (PSF) engineering is an optical method that enables various 3D imaging capabilities, yet it has not been implemented in high-throughput microscopy due to the cumbersome optical extension it typically requires. Here we demonstrate compact PSF engineering in the objective lens, which allows us to enhance the imaging depth of field and, combined with deep learning, recover 3D information using single snapshots. Beyond the applications shown here, this work showcases the usefulness of high-throughput microscopy in obtaining training data for deep learning-based algorithms, applicable to a variety of microscopy modalities.

Connecto-informatics at the mesoscale: current advances in image processing and analysis for mapping the brain connectivity

Mapping neural connections within the brain has been a fundamental goal in neuroscience to understand better its functions and changes that follow aging and diseases. Developments in imaging technology, such as microscopy and labeling tools, have allowed researchers to visualize this connectivity through high-resolution brain-wide imaging. With this, image processing and analysis have become more crucial. However, despite the wealth of neural images generated, access to an integrated image processing and analysis pipeline to process these data is challenging due to scattered information on available tools and methods. To map the neural connections, registration to atlases and feature extraction through segmentation and signal detection are necessary. In this review, our goal is to provide an updated overview of recent advances in these image-processing methods, with a particular focus on fluorescent images of the mouse brain. Our goal is to outline a pathway toward an integrated image-processing pipeline tailored for connecto-informatics. An integrated workflow of these image processing will facilitate researchers' approach to mapping brain connectivity to better understand complex brain networks and their underlying brain functions. By highlighting the image-processing tools available for fluroscent imaging of the mouse brain, this review will contribute to a deeper grasp of connecto-informatics, paving the way for better comprehension of brain connectivity and its implications.

Automated classification of cellular expression in multiplexed imaging data with Nimbus

Multiplexed imaging offers a powerful approach to characterize the spatial topography of tissues in both health and disease. To analyze such data, the specific combination of markers that are present in each cell must be enumerated to enable accurate phenotyping, a process that often relies on unsupervised clustering. We constructed the Pan-Multiplex (Pan-M) dataset containing 197 million distinct annotations of marker expression across 15 different cell types. We used Pan-M to create Nimbus, a deep learning model to predict marker positivity from multiplexed image data. Nimbus is a pre-trained model that uses the underlying images to classify marker expression across distinct cell types, from different tissues, acquired using different microscope platforms, without requiring any retraining. We demonstrate that Nimbus predictions capture the underlying staining patterns of the full diversity of markers present in Pan-M. We then show how Nimbus predictions can be integrated with downstream clustering algorithms to robustly identify cell subtypes in image data. We have open-sourced Nimbus and Pan-M to enable community use at https://github.com/angelolab/Nimbus-Inference.

Cytokine polarized, alternatively activated bone marrow neutrophils drive axon regeneration

The adult central nervous system (CNS) possesses a limited capacity for self-repair. Severed CNS axons typically fail to regrow. There is an unmet need for treatments designed to enhance neuronal viability, facilitate axon regeneration and ultimately restore lost neurological functions to individuals affected by traumatic CNS injury, multiple sclerosis, stroke and other neurological disorders. Here we demonstrate that both mouse and human bone marrow neutrophils, when polarized with a combination of recombinant interleukin-4 (IL-4) and granulocyte colony-stimulating factor (G-CSF), upregulate alternative activation markers and produce an array of growth factors, thereby gaining the capacity to promote neurite outgrowth. Moreover, adoptive transfer of IL-4/G-CSF-polarized bone marrow neutrophils into experimental models of CNS injury triggered substantial axon regeneration within the optic nerve and spinal cord. These findings have far-reaching implications for the future development of autologous myeloid cell-based therapies that may bring us closer to effective solutions for reversing CNS damage.

A comparative study of CARE 2D and N2V 2D for tissue-specific denoising in second harmonic generation imaging

This study explored the application of deep learning in second harmonic generation (SHG) microscopy, a rapidly growing area. This study focuses on the impact of glycerol concentration on image noise in SHG microscopy and compares two image restoration techniques: Noise-to-Void 2D (N2V 2D, no reference image restoration) and content-aware image restoration (CARE 2D, full reference image restoration). We demonstrated that N2V 2D effectively restored the images affected by high glycerol concentrations. To reduce sample exposure and damage, this study further addresses low-power SHG imaging by reducing the laser power by 70% using deep learning techniques. CARE 2D excels in preserving detailed structures, whereas N2V 2D maintains natural muscle structure. This study highlights the strengths and limitations of these models in specific SHG microscopy applications, offering valuable insights and potential advancements in the field .

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

Diffusion-based deep learning method for augmenting ultrastructural imaging and volume electron microscopy

Electron microscopy (EM) revolutionized the way to visualize cellular ultrastructure. Volume EM (vEM) has further broadened its three-dimensional nanoscale imaging capacity. However, intrinsic trade-offs between imaging speed and quality of EM restrict the attainable imaging area and volume. Isotropic imaging with vEM for large biological volumes remains unachievable. Here, we developed EMDiffuse, a suite of algorithms designed to enhance EM and vEM capabilities, leveraging the cutting-edge image generation diffusion model. EMDiffuse generates realistic predictions with high resolution ultrastructural details and exhibits robust transferability by taking only one pair of images of 3 megapixels to fine-tune in denoising and super-resolution tasks. EMDiffuse also demonstrated proficiency in the isotropic vEM reconstruction task, generating isotropic volume even in the absence of isotropic training data. We demonstrated the robustness of EMDiffuse by generating isotropic volumes from seven public datasets obtained from different vEM techniques and instruments. The generated isotropic volume enables accurate three-dimensional nanoscale ultrastructure analysis. EMDiffuse also features self-assessment functionalities on predictions' reliability. We envision EMDiffuse to pave the way for investigations of the intricate subcellular nanoscale ultrastructure within large volumes of biological systems.

Super-resolution of biomedical volumes with 2D supervision

Volumetric biomedical microscopy has the potential to increase the diagnostic information extracted from clinical tissue specimens and improve the diagnostic accuracy of both human pathologists and computational pathology models. Unfortunately, barriers to integrating 3-dimensional (3D) volumetric microscopy into clinical medicine include long imaging times, poor depth/z-axis resolution, and an insufficient amount of high-quality volumetric data. Leveraging the abundance of high-resolution 2D microscopy data, we introduce masked slice diffusion for super-resolution (MSDSR), which exploits the inherent equivalence in the data-generating distribution across all spatial dimensions of biological specimens. This intrinsic characteristic allows for super-resolution models trained on high-resolution images from one plane (e.g., XY) to effectively generalize to others (XZ, YZ), overcoming the traditional dependency on orientation. We focus on the application of MSDSR to stimulated Raman histology (SRH), an optical imaging modality for biological specimen analysis and intraoperative diagnosis, characterized by its rapid acquisition of high-resolution 2D images but slow and costly optical z-sectioning. To evaluate MSDSR's efficacy, we introduce a new performance metric, SliceFID, and demonstrate MSDSR's superior performance over baseline models through extensive evaluations. Our findings reveal that MSDSR not only significantly enhances the quality and resolution of 3D volumetric data, but also addresses major obstacles hindering the broader application of 3D volumetric microscopy in clinical diagnostics and biomedical research.

Advancements and Practical Considerations for Biophysical Research: Navigating the Challenges and Future of Super-resolution Microscopy

The introduction of super-resolution microscopy (SRM) has significantly advanced our understanding of cellular and molecular dynamics, offering a detailed view previously beyond our reach. Implementing SRM in biophysical research, however, presents numerous challenges. This review addresses the crucial aspects of utilizing SRM effectively, from selecting appropriate fluorophores and preparing samples to analyzing complex data sets. We explore recent technological advancements and methodological improvements that enhance the capabilities of SRM. Emphasizing the integration of SRM with other analytical methods, we aim to overcome inherent limitations and expand the scope of biological insights achievable. By providing a comprehensive guide for choosing the most suitable SRM methods based on specific research objectives, we aim to empower researchers to explore complex biological processes with enhanced precision and clarity, thereby advancing the frontiers of biophysical research.

Zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy

Computational super-resolution methods, including conventional analytical algorithms and deep learning models, have substantially improved optical microscopy. Among them, supervised deep neural networks have demonstrated outstanding performance, however, demanding abundant high-quality training data, which are laborious and even impractical to acquire due to the high dynamics of living cells. Here, we develop zero-shot deconvolution networks (ZS-DeconvNet) that instantly enhance the resolution of microscope images by more than 1.5-fold over the diffraction limit with 10-fold lower fluorescence than ordinary super-resolution imaging conditions, in an unsupervised manner without the need for either ground truths or additional data acquisition. We demonstrate the versatile applicability of ZS-DeconvNet on multiple imaging modalities, including total internal reflection fluorescence microscopy, three-dimensional wide-field microscopy, confocal microscopy, two-photon microscopy, lattice light-sheet microscopy, and multimodal structured illumination microscopy, which enables multi-color, long-term, super-resolution 2D/3D imaging of subcellular bioprocesses from mitotic single cells to multicellular embryos of mouse and C. elegans.

Expanding the Horizons of Machine Learning in Nanomaterials to Chiral Nanostructures

Machine learning holds significant research potential in the field of nanotechnology, enabling nanomaterial structure and property predictions, facilitating materials design and discovery, and reducing the need for time-consuming and labor-intensive experiments and simulations. In contrast to their achiral counterparts, the application of machine learning for chiral nanomaterials is still in its infancy, with a limited number of publications to date. This is despite the great potential of machine learning to advance the development of new sustainable chiral materials with high values of optical activity, circularly polarized luminescence, and enantioselectivity, as well as for the analysis of structural chirality by electron microscopy. In this review, an analysis of machine learning methods used for studying achiral nanomaterials is provided, subsequently offering guidance on adapting and extending this work to chiral nanomaterials. An overview of chiral nanomaterials within the framework of synthesis-structure-property-application relationships is presented and insights on how to leverage machine learning for the study of these highly complex relationships are provided. Some key recent publications are reviewed and discussed on the application of machine learning for chiral nanomaterials. Finally, the review captures the key achievements, ongoing challenges, and the prospective outlook for this very important research field.

Modular segmentation, spatial analysis and visualization of volume electron microscopy datasets

Volume electron microscopy is the method of choice for the in situ interrogation of cellular ultrastructure at the nanometer scale, and with the increase in large raw image datasets generated, improving computational strategies for image segmentation and spatial analysis is necessary. Here we describe a practical and annotation-efficient pipeline for organelle-specific segmentation, spatial analysis and visualization of large volume electron microscopy datasets using freely available, user-friendly software tools that can be run on a single standard workstation. The procedures are aimed at researchers in the life sciences with modest computational expertise, who use volume electron microscopy and need to generate three-dimensional (3D) segmentation labels for different types of cell organelles while minimizing manual annotation efforts, to analyze the spatial interactions between organelle instances and to visualize the 3D segmentation results. We provide detailed guidelines for choosing well-suited segmentation tools for specific cell organelles, and to bridge compatibility issues between freely available open-source tools, we distribute the critical steps as easily installable Album solutions for deep learning segmentation, spatial analysis and 3D rendering. Our detailed description can serve as a reference for similar projects requiring particular strategies for single- or multiple-organelle analysis, which can be achieved with computational resources commonly available to single-user setups.

Self-Supervised Joint Learning for pCLE Image Denoising

Probe-based confocal laser endoscopy (pCLE) has emerged as a powerful tool for disease diagnosis, yet it faces challenges such as the formation of hexagonal patterns in images due to the inherent characteristics of fiber bundles. Recent advancements in deep learning offer promise in image denoising, but the acquisition of clean-noisy image pairs for training networks across all potential scenarios can be prohibitively costly. Few studies have explored training denoising networks on such pairs. Here, we propose an innovative self-supervised denoising method. Our approach integrates noise prediction networks, image quality assessment networks, and denoising networks in a collaborative, jointly trained manner. Compared to prior self-supervised denoising methods, our approach yields superior results on pCLE images and fluorescence microscopy images. In summary, our novel self-supervised denoising technique enhances image quality in pCLE diagnosis by leveraging the synergy of noise prediction, image quality assessment, and denoising networks, surpassing previous methods on both pCLE and fluorescence microscopy images.

Label-Free Techniques for Probing Biomolecular Condensates

Biomolecular condensates play important roles in a wide array of fundamental biological processes, such as cellular compartmentalization, cellular regulation, and other biochemical reactions. Since their discovery and first observations, an extensive and expansive library of tools has been developed to investigate various aspects and properties, encompassing structural and compositional information, material properties, and their evolution throughout the life cycle from formation to eventual dissolution. This Review presents an overview of the expanded set of tools and methods that researchers use to probe the properties of biomolecular condensates across diverse scales of length, concentration, stiffness, and time. In particular, we review recent years' exciting development of label-free techniques and methodologies. We broadly organize the set of tools into 3 categories: (1) imaging-based techniques, such as transmitted-light microscopy (TLM) and Brillouin microscopy (BM), (2) force spectroscopy techniques, such as atomic force microscopy (AFM) and the optical tweezer (OT), and (3) microfluidic platforms and emerging technologies. We point out the tools' key opportunities, challenges, and future perspectives and analyze their correlative potential as well as compatibility with other techniques. Additionally, we review emerging techniques, namely, differential dynamic microscopy (DDM) and interferometric scattering microscopy (iSCAT), that have huge potential for future applications in studying biomolecular condensates. Finally, we highlight how some of these techniques can be translated for diagnostics and therapy purposes. We hope this Review serves as a useful guide for new researchers in this field and aids in advancing the development of new biophysical tools to study biomolecular condensates.

Endogenous tagging using split mNeonGreen in human iPSCs for live imaging studies

Endogenous tags have become invaluable tools to visualize and study native proteins in live cells. However, generating human cell lines carrying endogenous tags is difficult due to the low efficiency of homology-directed repair. Recently, an engineered split mNeonGreen protein was used to generate a large-scale endogenous tag library in HEK293 cells. Using split mNeonGreen for large-scale endogenous tagging in human iPSCs would open the door to studying protein function in healthy cells and across differentiated cell types. We engineered an iPS cell line to express the large fragment of the split mNeonGreen protein (mNG21-10) and showed that it enables fast and efficient endogenous tagging of proteins with the short fragment (mNG211). We also demonstrate that neural network-based image restoration enables live imaging studies of highly dynamic cellular processes such as cytokinesis in iPSCs. This work represents the first step towards a genome-wide endogenous tag library in human stem cells.

Receptor transfer between immune cells by autoantibody-enhanced, CD32-driven trogocytosis is hijacked by HIV-1 to infect resting CD4 T cells

Immune cell phenotyping frequently detects lineage-unrelated receptors. Here, we report that surface receptors can be transferred from primary macrophages to CD4 T cells and identify the Fcγ receptor CD32 as driver and cargo of this trogocytotic transfer. Filamentous CD32+ nanoprotrusions deposit distinct plasma membrane patches onto target T cells. Transferred receptors confer cell migration and adhesion properties, and macrophage-derived membrane patches render resting CD4 T cells susceptible to infection by serving as hotspots for HIV-1 binding. Antibodies that recognize T cell epitopes enhance CD32-mediated trogocytosis. Such autoreactive anti-HIV-1 envelope antibodies can be found in the blood of HIV-1 patients and, consistently, the percentage of CD32+ CD4 T cells is increased in their blood. This CD32-mediated, antigen-independent cell communication mode transiently expands the receptor repertoire and functionality of immune cells. HIV-1 hijacks this mechanism by triggering the generation of trogocytosis-promoting autoantibodies to gain access to immune cells critical to its persistence.

Cellular architecture of evolving neuroinflammatory lesions and multiple sclerosis pathology

Multiple sclerosis (MS) is a neurological disease characterized by multifocal lesions and smoldering pathology. Although single-cell analyses provided insights into cytopathology, evolving cellular processes underlying MS remain poorly understood. We investigated the cellular dynamics of MS by modeling temporal and regional rates of disease progression in mouse experimental autoimmune encephalomyelitis (EAE). By performing single-cell spatial expression profiling using in situ sequencing (ISS), we annotated disease neighborhoods and found centrifugal evolution of active lesions. We demonstrated that disease-associated (DA)-glia arise independently of lesions and are dynamically induced and resolved over the disease course. Single-cell spatial mapping of human archival MS spinal cords confirmed the differential distribution of homeostatic and DA-glia, enabled deconvolution of active and inactive lesions into sub-compartments, and identified new lesion areas. By establishing a spatial resource of mouse and human MS neuropathology at a single-cell resolution, our study unveils the intricate cellular dynamics underlying MS.

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Deep learning algorithms have been widely used in microscopic image translation. The corresponding data-driven models can be trained by supervised or unsupervised learning depending on the availability of paired data. However, general cases are where the data are only roughly paired such that supervised learning could be invalid due to data unalignment, and unsupervised learning would be less ideal as the roughly paired information is not utilized. In this work, we propose a unified framework (U-Frame) that unifies supervised and unsupervised learning by introducing a tolerance size that can be adjusted automatically according to the degree of data misalignment. Together with the implementation of a global sampling rule, we demonstrate that U-Frame consistently outperforms both supervised and unsupervised learning in all levels of data misalignments (even for perfectly aligned image pairs) in a myriad of image translation applications, including pseudo-optical sectioning, virtual histological staining (with clinical evaluations for cancer diagnosis), improvement of signal-to-noise ratio or resolution, and prediction of fluorescent labels, potentially serving as new standard for image translation.