Evaluation and development of deep neural networks for image super-resolution in optical microscopy

Neural space-time model for dynamic multi-shot imaging

Computational imaging reconstructions from multiple measurements that are captured sequentially often suffer from motion artifacts if the scene is dynamic. We propose a neural space-time model (NSTM) that jointly estimates the scene and its motion dynamics, without data priors or pre-training. Hence, we can both remove motion artifacts and resolve sample dynamics from the same set of raw measurements used for the conventional reconstruction. We demonstrate NSTM in three computational imaging systems: differential phase-contrast microscopy, three-dimensional structured illumination microscopy and rolling-shutter DiffuserCam. We show that NSTM can recover subcellular motion dynamics and thus reduce the misinterpretation of living systems caused by motion artifacts.

Dynamic phosphorylation of FOXA1 by Aurora B guides post-mitotic gene reactivation

FOXA1 serves as a crucial pioneer transcription factor during developmental processes and plays a pivotal role as a mitotic bookmarking factor to perpetuate gene expression profiles and maintain cellular identity. During mitosis, the majority of FOXA1 dissociates from specific DNA binding sites and redistributes to non-specific binding sites; however, the regulatory mechanisms governing molecular dynamics and activity of FOXA1 remain elusive. Here, we show that mitotic kinase Aurora B specifies the different DNA binding modes of FOXA1 and guides FOXA1 biomolecular condensation in mitosis. Mechanistically, Aurora B kinase phosphorylates FOXA1 at Serine 221 (S221) to liberate the specific, but not the non-specific, DNA binding. Interestingly, the phosphorylation of S221 attenuates the FOXA1 condensation that requires specific DNA binding. Importantly, perturbation of the dynamic phosphorylation impairs accurate gene reactivation and cell proliferation, suggesting that reversible mitotic protein phosphorylation emerges as a fundamental mechanism for the spatiotemporal control of mitotic bookmarking.

Image restoration in frequency space using complex-valued CNNs

Real-valued convolutional neural networks (RV-CNNs) in the spatial domain have outperformed classical approaches in many image restoration tasks such as image denoising and super-resolution. Fourier analysis of the results produced by these spatial domain models reveals the limitations of these models in properly processing the full frequency spectrum. This lack of complete spectral information can result in missing textural and structural elements. To address this limitation, we explore the potential of complex-valued convolutional neural networks (CV-CNNs) for image restoration tasks. CV-CNNs have shown remarkable performance in tasks such as image classification and segmentation. However, CV-CNNs for image restoration problems in the frequency domain have not been fully investigated to address the aforementioned issues. Here, we propose several novel CV-CNN-based models equipped with complex-valued attention gates for image denoising and super-resolution in the frequency domains. We also show that our CV-CNN-based models outperform their real-valued counterparts for denoising super-resolution structured illumination microscopy (SR-SIM) and conventional image datasets. Furthermore, the experimental results show that our proposed CV-CNN-based models preserve the frequency spectrum better than their real-valued counterparts in the denoising task. Based on these findings, we conclude that CV-CNN-based methods provide a plausible and beneficial deep learning approach for image restoration in the frequency domain.

Self-inspired learning for denoising live-cell super-resolution microscopy

Every collected photon is precious in live-cell super-resolution (SR) microscopy. Here, we describe a data-efficient, deep learning-based denoising solution to improve diverse SR imaging modalities. The method, SN2N, is a Self-inspired Noise2Noise module with self-supervised data generation and self-constrained learning process. SN2N is fully competitive with supervised learning methods and circumvents the need for large training set and clean ground truth, requiring only a single noisy frame for training. We show that SN2N improves photon efficiency by one-to-two orders of magnitude and is compatible with multiple imaging modalities for volumetric, multicolor, time-lapse SR microscopy. We further integrated SN2N into different SR reconstruction algorithms to effectively mitigate image artifacts. We anticipate SN2N will enable improved live-SR imaging and inspire further advances.

Peering into the mind: unraveling schizophrenia's secrets using models

Schizophrenia (SCZ) is a complex mental disorder characterized by a range of symptoms, including positive and negative symptoms, as well as cognitive impairments. Despite the extensive research, the underlying neurobiology of SCZ remain elusive. To overcome this challenge, the use of diverse laboratory modeling techniques, encompassing cellular and animal models, and innovative approaches like induced pluripotent stem cell (iPSC)-derived neuronal cultures or brain organoids and genetically engineered animal models, has been crucial. Immortalized cellular models provide controlled environments for investigating the molecular and neurochemical pathways involved in neuronal function, while iPSCs and brain organoids, derived from patient-specific sources, offer significant advantage in translational research by facilitating direct comparisons of cellular phenotypes between patient-derived neurons and healthy-control neurons. Animal models can recapitulate the different psychopathological aspects that should be modeled, offering valuable insights into the neurobiology of SCZ. In addition, invertebrates' models are genetically tractable and offer a powerful approach to dissect the core genetic underpinnings of SCZ, while vertebrate models, especially mammals, with their more complex nervous systems and behavioral repertoire, provide a closer approximation of the human condition to study SCZ-related traits. This narrative review provides a comprehensive overview of the diverse modeling approaches, critically evaluating their strengths and limitations. By synthesizing knowledge from these models, this review offers a valuable source for researchers, clinicians, and stakeholders alike. Integrating findings across these different models may allow us to build a more holistic picture of SCZ pathophysiology, facilitating the exploration of new research avenues and informed decision-making for interventions.

Deep learning permits imaging of multiple structures with the same fluorophores

Fluorescence microscopy, which employs fluorescent tags to label and observe cellular structures and their dynamics, is a powerful tool for life sciences. However, due to the spectral overlap between different dyes, a limited number of structures can be separately labeled and imaged for live-cell applications. In addition, the conventional sequential channel imaging procedure is quite time consuming, as it needs to switch either different lasers or filters. Here, we propose a novel double-structure network (DBSN) that consists of multiple connected models, which can extract six distinct subcellular structures from three raw images with only two separate fluorescent labels. DBSN combines the intensity-balance model to compensate for uneven fluorescent labels for different structures and the structure-separation model to extract multiple different structures with the same fluorescent labels. Therefore, DBSN breaks the bottleneck of the existing technologies and holds immense potential applications in the field of cell biology.

Fast, faster, and the fastest structured illumination microscopy

Parallel acquisition-readout structured-illumination microscopy (PAR-SIM) was designed for high-speed raw data acquisition. By utilizing an xy-scan galvo mirror set, the raw data is projected onto different areas of the camera, enabling a fundamentally stupendous information spatial-temporal flux.

Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration

Fluorescence microscopy-based image restoration has received widespread attention in the life sciences and has led to significant progress, benefiting from deep learning technology. However, most current task-specific methods have limited generalizability to different fluorescence microscopy-based image restoration problems. Here, we seek to improve generalizability and explore the potential of applying a pretrained foundation model to fluorescence microscopy-based image restoration. We provide a universal fluorescence microscopy-based image restoration (UniFMIR) model to address different restoration problems, and show that UniFMIR offers higher image restoration precision, better generalization and increased versatility. Demonstrations on five tasks and 14 datasets covering a wide range of microscopy imaging modalities and biological samples demonstrate that the pretrained UniFMIR can effectively transfer knowledge to a specific situation via fine-tuning, uncover clear nanoscale biomolecular structures and facilitate high-quality imaging. This work has the potential to inspire and trigger new research highlights for fluorescence microscopy-based image restoration.

Orientation selectivity mapping in the visual cortex

The orientation map is one of the most well-studied functional maps of the visual cortex. However, results from the literature are of different qualities. Clear boundaries among different orientation domains and blurred uncertain distinctions were shown in different studies. These unclear imaging results will lead to an inaccuracy in depicting cortical structures, and the lack of consideration in experimental design will also lead to biased depictions of the cortical features. How we accurately define orientation domains will impact the entire field of research. In this study, we test how spatial frequency (SF), stimulus size, location, chromatic, and data processing methods affect the orientation functional maps (including a large area of dorsal V4, and parts of dorsal V1) acquired by intrinsic signal optical imaging. Our results indicate that, for large imaging fields, large grating stimuli with mixed SF components should be considered to acquire the orientation map. A diffusion model image enhancement based on the difference map could further improve the map quality. In addition, the similar outcomes of achromatic and chromatic gratings indicate two alternative types of afferents from LGN, pooling in V1 to generate cue-invariant orientation selectivity.

Approaching Maximum Resolution in Structured Illumination Microscopy via Accurate Noise Modeling

Biological images captured by microscopes are characterized by heterogeneous signal-to-noise ratios (SNRs) due to spatially varying photon emission across the field of view convoluted with camera noise. State-of-the-art unsupervised structured illumination microscopy (SIM) reconstruction algorithms, commonly implemented in the Fourier domain, do not accurately model this noise and suffer from high-frequency artifacts, user-dependent choices of smoothness constraints making assumptions on biological features, and unphysical negative values in the recovered fluorescence intensity map. On the other hand, supervised methods rely on large datasets for training, and often require retraining for new sample structures. Consequently, achieving high contrast near the maximum theoretical resolution in an unsupervised, physically principled, manner remains an open problem. Here, we propose Bayesian-SIM (B-SIM), an unsupervised Bayesian framework to quantitatively reconstruct SIM data, rectifying these shortcomings by accurately incorporating known noise sources in the spatial domain. To accelerate the reconstruction process, we use the finite extent of the point-spread-function to devise a parallelized Monte Carlo strategy involving chunking and restitching of the inferred fluorescence intensity. We benchmark our framework on both simulated and experimental images, and demonstrate improved contrast permitting feature recovery at up to 25% shorter length scales over state-of-the-art methods at both high- and low-SNR. B-SIM enables unsupervised, quantitative, physically accurate reconstruction without the need for labeled training data, democratizing high-quality SIM reconstruction and expands the capabilities of live-cell SIM to lower SNR, potentially revealing biological features in previously inaccessible regimes.

Model-free robust motion control for biological optical microscopy using time-delay estimation with an adaptive RBFNN compensator

The field of large numerical aperture microscopy has witnessed significant advancements in spatial and temporal resolution, as well as improvements in optical microscope imaging quality. However, these advancements have concurrently raised the demand for enhanced precision, extended range, and increased load-bearing capacity in objective motion carrier (OMC). To address this challenge, this study introduces an innovative OMC that employs a ball screw mechanism as its primary driving component. Furthermore, a robust nonlinear motion control strategy has been developed, which integrates fast nonsingular terminal sliding mode, experimental estimation techniques, and adaptive radial basis neural network, to mitigate the impact of nonlinear friction within the ball screw mechanism on motion precision. The stability of the closed-loop control system has been rigorously demonstrated through Lyapunov theory. Compared with other enhanced sliding mode control strategies, the maximum error and root mean square error of this controller are improved by 33% and 34% respectively. The implementation of the novel OMC has enabled the establishment of a high-resolution bio-optical microscope, which has proven its effectiveness in the microscopic imaging of retinal organoids.

A comparative study of CARE 2D and N2V 2D for tissue-specific denoising in second harmonic generation imaging

This study explored the application of deep learning in second harmonic generation (SHG) microscopy, a rapidly growing area. This study focuses on the impact of glycerol concentration on image noise in SHG microscopy and compares two image restoration techniques: Noise-to-Void 2D (N2V 2D, no reference image restoration) and content-aware image restoration (CARE 2D, full reference image restoration). We demonstrated that N2V 2D effectively restored the images affected by high glycerol concentrations. To reduce sample exposure and damage, this study further addresses low-power SHG imaging by reducing the laser power by 70% using deep learning techniques. CARE 2D excels in preserving detailed structures, whereas N2V 2D maintains natural muscle structure. This study highlights the strengths and limitations of these models in specific SHG microscopy applications, offering valuable insights and potential advancements in the field .

Enhanced optical imaging and fluorescent labeling for visualizing drug molecules within living organisms

The visualization of drugs in living systems has become key techniques in modern therapeutics. Recent advancements in optical imaging technologies and molecular design strategies have revolutionized drug visualization. At the subcellular level, super-resolution microscopy has allowed exploration of the molecular landscape within individual cells and the cellular response to drugs. Moving beyond subcellular imaging, researchers have integrated multiple modes, like optical near-infrared II imaging, to study the complex spatiotemporal interactions between drugs and their surroundings. By combining these visualization approaches, researchers gain supplementary information on physiological parameters, metabolic activity, and tissue composition, leading to a comprehensive understanding of drug behavior. This review focuses on cutting-edge technologies in drug visualization, particularly fluorescence imaging, and the main types of fluorescent molecules used. Additionally, we discuss current challenges and prospects in targeted drug research, emphasizing the importance of multidisciplinary cooperation in advancing drug visualization. With the integration of advanced imaging technology and molecular design, drug visualization has the potential to redefine our understanding of pharmacology, enabling the analysis of drug micro-dynamics in subcellular environments from new perspectives and deepening pharmacological research to the levels of the cell and organelles.

DGCBG-Net: A dual-branch network with global cross-modal interaction and boundary guidance for tumor segmentation in PET/CT images

BACKGROUND AND OBJECTIVES

Automatic tumor segmentation plays a crucial role in cancer diagnosis and treatment planning. Computed tomography (CT) and positron emission tomography (PET) are extensively employed for their complementary medical information. However, existing methods ignore bilateral cross-modal interaction of global features during feature extraction, and they underutilize multi-stage tumor boundary features.

METHODS

To address these limitations, we propose a dual-branch tumor segmentation network based on global cross-modal interaction and boundary guidance in PET/CT images (DGCBG-Net). DGCBG-Net consists of 1) a global cross-modal interaction module that extracts global contextual information from PET/CT images and promotes bilateral cross-modal interaction of global feature; 2) a shared multi-path downsampling module that learns complementary features from PET/CT modalities to mitigate the impact of misleading features and decrease the loss of discriminative features during downsampling; 3) a boundary prior-guided branch that extracts potential boundary features from CT images at multiple stages, assisting the semantic segmentation branch in improving the accuracy of tumor boundary segmentation.

RESULTS

Extensive experiments are conducted on STS and Hecktor 2022 datasets to evaluate the proposed method. The average Dice scores of our DGCB-Net on the two datasets are 80.33% and 79.29%, with average IOU scores of 67.64% and 70.18%. DGCB-Net outperformed the current state-of-the-art methods with a 1.77% higher Dice score and a 2.12% higher IOU score.

CONCLUSIONS

Extensive experimental results demonstrate that DGCBG-Net outperforms existing segmentation methods, and is competitive to state-of-arts.

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

Diffusion-based deep learning method for augmenting ultrastructural imaging and volume electron microscopy

Electron microscopy (EM) revolutionized the way to visualize cellular ultrastructure. Volume EM (vEM) has further broadened its three-dimensional nanoscale imaging capacity. However, intrinsic trade-offs between imaging speed and quality of EM restrict the attainable imaging area and volume. Isotropic imaging with vEM for large biological volumes remains unachievable. Here, we developed EMDiffuse, a suite of algorithms designed to enhance EM and vEM capabilities, leveraging the cutting-edge image generation diffusion model. EMDiffuse generates realistic predictions with high resolution ultrastructural details and exhibits robust transferability by taking only one pair of images of 3 megapixels to fine-tune in denoising and super-resolution tasks. EMDiffuse also demonstrated proficiency in the isotropic vEM reconstruction task, generating isotropic volume even in the absence of isotropic training data. We demonstrated the robustness of EMDiffuse by generating isotropic volumes from seven public datasets obtained from different vEM techniques and instruments. The generated isotropic volume enables accurate three-dimensional nanoscale ultrastructure analysis. EMDiffuse also features self-assessment functionalities on predictions' reliability. We envision EMDiffuse to pave the way for investigations of the intricate subcellular nanoscale ultrastructure within large volumes of biological systems.

Light field image super-resolution based on dual learning and deep Fourier channel attention

Light field (LF) imaging has gained significant attention in the field of computational imaging due to its unique capability to capture both spatial and angular information of a scene. In recent years, super-resolution (SR) techniques based on deep learning have shown considerable advantages in enhancing LF image resolution. However, the inherent challenges of obtaining rich structural information and reconstructing complex texture details persist, particularly in scenarios where spatial and angular information are intricately interwoven. This Letter introduces a novel, to the best of our knowledge, approach for Disentangling LF Image SR Network (DLISN) by leveraging the synergy of dual learning and Fourier channel attention (FCA) mechanisms. Dual learning strategies are employed to enhance reconstruction results, addressing limitations in model generalization caused by the difficulty in acquiring paired datasets in real-world LF scenarios. The integration of FCA facilitates the extraction of high-frequency information associated with different structures, contributing to improved spatial resolution. Experimental results consistently demonstrate superior performance in enhancing the resolution of LF images.

A Residual Dense Attention Generative Adversarial Network for Microscopic Image Super-Resolution

With the development of deep learning, the Super-Resolution (SR) reconstruction of microscopic images has improved significantly. However, the scarcity of microscopic images for training, the underutilization of hierarchical features in original Low-Resolution (LR) images, and the high-frequency noise unrelated with the image structure generated during the reconstruction process are still challenges in the Single Image Super-Resolution (SISR) field. Faced with these issues, we first collected sufficient microscopic images through Motic, a company engaged in the design and production of optical and digital microscopes, to establish a dataset. Secondly, we proposed a Residual Dense Attention Generative Adversarial Network (RDAGAN). The network comprises a generator, an image discriminator, and a feature discriminator. The generator includes a Residual Dense Block (RDB) and a Convolutional Block Attention Module (CBAM), focusing on extracting the hierarchical features of the original LR image. Simultaneously, the added feature discriminator enables the network to generate high-frequency features pertinent to the image's structure. Finally, we conducted experimental analysis and compared our model with six classic models. Compared with the best model, our model improved PSNR and SSIM by about 1.5 dB and 0.2, respectively.

Ultra-high spatio-temporal resolution imaging with parallel acquisition-readout structured illumination microscopy (PAR-SIM)

Structured illumination microscopy (SIM) has emerged as a promising super-resolution fluorescence imaging technique, offering diverse configurations and computational strategies to mitigate phototoxicity during real-time imaging of biological specimens. Traditional efforts to enhance system frame rates have concentrated on processing algorithms, like rolling reconstruction or reduced frame reconstruction, or on investments in costly sCMOS cameras with accelerated row readout rates. In this article, we introduce an approach to elevate SIM frame rates and region of interest (ROI) coverage at the hardware level, without necessitating an upsurge in camera expenses or intricate algorithms. Here, parallel acquisition-readout SIM (PAR-SIM) achieves the highest imaging speed for fluorescence imaging at currently available detector sensitivity. By using the full frame-width of the detector through synchronizing the pattern generation and image exposure-readout process, we have achieved a fundamentally stupendous information spatial-temporal flux of 132.9 MPixels · s-1, 9.6-fold that of the latest techniques, with the lowest SNR of -2.11 dB and 100 nm resolution. PAR-SIM demonstrates its proficiency in successfully reconstructing diverse cellular organelles in dual excitations, even under conditions of low signal due to ultra-short exposure times. Notably, mitochondrial dynamic tubulation and ongoing membrane fusion processes have been captured in live COS-7 cell, recorded with PAR-SIM at an impressive 408 Hz. We posit that this novel parallel exposure-readout mode not only augments SIM pattern modulation for superior frame rates but also holds the potential to benefit other complex imaging systems with a strategic controlling approach.

Visualization of cristae and mtDNA interactions via STED nanoscopy using a low saturation power probe

Mitochondria are crucial organelles closely associated with cellular metabolism and function. Mitochondrial DNA (mtDNA) encodes a variety of transcripts and proteins essential for cellular function. However, the interaction between the inner membrane (IM) and mtDNA remains elusive due to the limitations in spatiotemporal resolution offered by conventional microscopy and the absence of suitable in vivo probes specifically targeting the IM. Here, we have developed a novel fluorescence probe called HBmito Crimson, characterized by exceptional photostability, fluorogenicity within lipid membranes, and low saturation power. We successfully achieved over 500 frames of low-power stimulated emission depletion microscopy (STED) imaging to visualize the IM dynamics, with a spatial resolution of 40 nm. By utilizing dual-color imaging of the IM and mtDNA, it has been uncovered that mtDNA tends to habitat at mitochondrial tips or branch points, exhibiting an overall spatially uniform distribution. Notably, the dynamics of mitochondria are intricately associated with the positioning of mtDNA, and fusion consistently occurs in close proximity to mtDNA to minimize pressure during cristae remodeling. In healthy cells, >66% of the mitochondria are Class III (i.e., mitochondria >5 μm or with >12 cristae), while it dropped to <18% in ferroptosis. Mitochondrial dynamics, orchestrated by cristae remodeling, foster the even distribution of mtDNA. Conversely, in conditions of apoptosis and ferroptosis where the cristae structure is compromised, mtDNA distribution becomes irregular. These findings, achieved with unprecedented spatiotemporal resolution, reveal the intricate interplay between cristae and mtDNA and provide insights into the driving forces behind mtDNA distribution.

Zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy

Computational super-resolution methods, including conventional analytical algorithms and deep learning models, have substantially improved optical microscopy. Among them, supervised deep neural networks have demonstrated outstanding performance, however, demanding abundant high-quality training data, which are laborious and even impractical to acquire due to the high dynamics of living cells. Here, we develop zero-shot deconvolution networks (ZS-DeconvNet) that instantly enhance the resolution of microscope images by more than 1.5-fold over the diffraction limit with 10-fold lower fluorescence than ordinary super-resolution imaging conditions, in an unsupervised manner without the need for either ground truths or additional data acquisition. We demonstrate the versatile applicability of ZS-DeconvNet on multiple imaging modalities, including total internal reflection fluorescence microscopy, three-dimensional wide-field microscopy, confocal microscopy, two-photon microscopy, lattice light-sheet microscopy, and multimodal structured illumination microscopy, which enables multi-color, long-term, super-resolution 2D/3D imaging of subcellular bioprocesses from mitotic single cells to multicellular embryos of mouse and C. elegans.

Multi-attention fusion transformer for single-image super-resolution

Recently, Transformer-based methods have gained prominence in image super-resolution (SR) tasks, addressing the challenge of long-range dependence through the incorporation of cross-layer connectivity and local attention mechanisms. However, the analysis of these networks using local attribution maps has revealed significant limitations in leveraging the spatial extent of input information. To unlock the inherent potential of Transformer in image SR, we propose the Multi-Attention Fusion Transformer (MAFT), a novel model designed to integrate multiple attention mechanisms with the objective of expanding the number and range of pixels activated during image reconstruction. This integration enhances the effective utilization of input information space. At the core of our model lies the Multi-attention Adaptive Integration Groups, which facilitate the transition from dense local attention to sparse global attention through the introduction of Local Attention Aggregation and Global Attention Aggregation blocks with alternating connections, effectively broadening the network's receptive field. The effectiveness of our proposed algorithm has been validated through comprehensive quantitative and qualitative evaluation experiments conducted on benchmark datasets. Compared to state-of-the-art methods (e.g. HAT), the proposed MAFT achieves 0.09 dB gains on Urban100 dataset for × 4 SR task while containing 32.55% and 38.01% fewer parameters and FLOPs, respectively.

Dual-Channel in Spatial-Frequency Domain CycleGAN for perceptual enhancement of transcranial cortical vascular structure and function

Observing cortical vascular structures and functions using laser speckle contrast imaging (LSCI) at high resolution plays a crucial role in understanding cerebral pathologies. Usually, open-skull window techniques have been applied to reduce scattering of skull and enhance image quality. However, craniotomy surgeries inevitably induce inflammation, which may obstruct observations in certain scenarios. In contrast, image enhancement algorithms provide popular tools for improving the signal-to-noise ratio (SNR) of LSCI. The current methods were less than satisfactory through intact skulls because the transcranial cortical images were of poor quality. Moreover, existing algorithms do not guarantee the accuracy of dynamic blood flow mappings. In this study, we develop an unsupervised deep learning method, named Dual-Channel in Spatial-Frequency Domain CycleGAN (SF-CycleGAN), to enhance the perceptual quality of cortical blood flow imaging by LSCI. SF-CycleGAN enabled convenient, non-invasive, and effective cortical vascular structure observation and accurate dynamic blood flow mappings without craniotomy surgeries to visualize biodynamics in an undisturbed biological environment. Our experimental results showed that SF-CycleGAN achieved a SNR at least 4.13 dB higher than that of other unsupervised methods, imaged the complete vascular morphology, and enabled the functional observation of small cortical vessels. Additionally, the proposed method showed remarkable robustness and could be generalized to various imaging configurations and image modalities, including fluorescence images, without retraining.

Cellular nucleus image-based smarter microscope system for single cell analysis

Cell imaging technology is undoubtedly a powerful tool for studying single-cell heterogeneity due to its non-invasive and visual advantages. It covers microscope hardware, software, and image analysis techniques, which are hindered by low throughput owing to abundant hands-on time and expertise. Herein, a cellular nucleus image-based smarter microscope system for single-cell analysis is reported to achieve high-throughput analysis and high-content detection of cells. By combining the hardware of an automatic fluorescence microscope and multi-object recognition/acquisition software, we have achieved more advanced process automation with the assistance of Robotic Process Automation (RPA), which realizes a high-throughput collection of single-cell images. Automated acquisition of single-cell images has benefits beyond ease and throughout and can lead to uniform standard and higher quality images. We further constructed a single-cell image database-based convolutional neural network (Efficient Convolutional Neural Network, E-CNN) exceeding 20618 single-cell nucleus images. Computational analysis of large and complex data sets enhances the content and efficiency of single-cell analysis with the assistance of Artificial Intelligence (AI), which breaks through the super-resolution microscope's hardware limitation, such as specialized light sources with specific wavelengths, advanced optical components, and high-performance graphics cards. Our system can identify single-cell nucleus images that cannot be artificially distinguished with an accuracy of 95.3%. Overall, we build an ordinary microscope into a high-throughput analysis and high-content smarter microscope system, making it a candidate tool for Imaging cytology.

Deep learning empowers photothermal microscopy with super-resolution capabilities

In the past two decades, photothermal microscopy (PTM) has achieved sensitivity at the level of a single particle or molecule and has found applications in the fields of material science and biology. PTM is a far-field imaging method; its resolution is restricted by the diffraction limits. In our previous work, the modulated difference PTM (MDPTM) was proposed to improve the lateral resolution, but its resolution improvement was seriously constrained by information loss and artifacts. In this Letter, a deep learning approach of the cycle generative adversarial network (Cycle GAN) is employed for further improving the resolution of PTM, called DMDPTM. The point spread functions (PSFs) of both PTM and MDPTM are optimized and act as the second generator of Cycle GAN. Besides, the relationship between the sample's volume and the photothermal signal is utilized during dataset construction. The images of both PTM and MDPTM are utilized as the inputs of the Cycle GAN to incorporate more information. In the simulation, DMDPTM quantitatively distinguishes a distance of 60 nm between two nanoparticles (each with a diameter of 60 nm), demonstrating a 4.4-fold resolution enhancement over the conventional PTM. Experimentally, the super-resolution capability of DMDPTM is verified by restored images of Au nanoparticles, achieving the resolution of 114 nm. Finally, the DMDPTM is successfully employed for the imaging of carbon nanotubes. Therefore, the DMDPTM will serve as a powerful tool to improve the lateral resolution of PTM.

SpiDe-Sr: blind super-resolution network for precise cell segmentation and clustering in spatial proteomics imaging

Spatial proteomics elucidates cellular biochemical changes with unprecedented topological level. Imaging mass cytometry (IMC) is a high-dimensional single-cell resolution platform for targeted spatial proteomics. However, the precision of subsequent clinical analysis is constrained by imaging noise and resolution. Here, we propose SpiDe-Sr, a super-resolution network embedded with a denoising module for IMC spatial resolution enhancement. SpiDe-Sr effectively resists noise and improves resolution by 4 times. We demonstrate SpiDe-Sr respectively with cells, mouse and human tissues, resulting 18.95%/27.27%/21.16% increase in peak signal-to-noise ratio and 15.95%/31.63%/15.52% increase in cell extraction accuracy. We further apply SpiDe-Sr to study the tumor microenvironment of a 20-patient clinical breast cancer cohort with 269,556 single cells, and discover the invasion of Gram-negative bacteria is positively correlated with carcinogenesis markers and negatively correlated with immunological markers. Additionally, SpiDe-Sr is also compatible with fluorescence microscopy imaging, suggesting SpiDe-Sr an alternative tool for microscopy image super-resolution.

A convolutional neural network-based method for the generation of super-resolution 3D models from clinical CT images

BACKGROUND AND OBJECTIVE

The accurate evaluation of bone mechanical properties is essential for predicting fracture risk based on clinical computed tomography (CT) images. However, blurring and noise in clinical CT images can compromise the accuracy of these predictions, leading to incorrect diagnoses. Although previous studies have explored enhancing trabecular bone CT images to super-resolution (SR), none of these studies have examined the possibility of using clinical CT images from different instruments, typically of lower resolution, as a basis for analysis. Additionally, previous studies rely on 2D SR images, which may not be sufficient for accurate mechanical property evaluation, due to the complex nature of the 3D trabecular bone structures. The objective of this study was to address these limitations.

METHODS

A workflow was developed that utilizes convolutional neural networks to generate SR 3D models across different clinical CT instruments. The morphological and finite-element-derived mechanical properties of these SR models were compared with ground truth models obtained from micro-CT scans.

RESULTS

A significant improvement in analysis accuracy was demonstrated, where the new SR models increased the accuracy by up to 700 % compared with the low-resolution data, i.e. clinical CT images. Additionally, we found that the mixture of different CT image datasets may improve the SR model performance.

CONCLUSIONS

SR images, generated by convolutional neural networks, outperformed clinical CT images in the determination of morphological and mechanical properties. The developed workflow could be implemented for fracture risk prediction, potentially leading to improved diagnoses and subsequent clinical decision making.

SEMORE: SEgmentation and MORphological fingErprinting by machine learning automates super-resolution data analysis

The morphology of protein assemblies impacts their behaviour and contributes to beneficial and aberrant cellular responses. While single-molecule localization microscopy provides the required spatial resolution to investigate these assemblies, the lack of universal robust analytical tools to extract and quantify underlying structures limits this powerful technique. Here we present SEMORE, a semi-automatic machine learning framework for universal, system- and input-dependent, analysis of super-resolution data. SEMORE implements a multi-layered density-based clustering module to dissect biological assemblies and a morphology fingerprinting module for quantification by multiple geometric and kinetics-based descriptors. We demonstrate SEMORE on simulations and diverse raw super-resolution data: time-resolved insulin aggregates, and published data of dSTORM imaging of nuclear pore complexes, fibroblast growth receptor 1, sptPALM of Syntaxin 1a and dynamic live-cell PALM of ryanodine receptors. SEMORE extracts and quantifies all protein assemblies, their temporal morphology evolution and provides quantitative insights, e.g. classification of heterogeneous insulin aggregation pathways and NPC geometry in minutes. SEMORE is a general analysis platform for super-resolution data, and being a time-aware framework can also support the rise of 4D super-resolution data.

Harnessing artificial intelligence to reduce phototoxicity in live imaging

Fluorescence microscopy is essential for studying living cells, tissues and organisms. However, the fluorescent light that switches on fluorescent molecules also harms the samples, jeopardizing the validity of results - particularly in techniques such as super-resolution microscopy, which demands extended illumination. Artificial intelligence (AI)-enabled software capable of denoising, image restoration, temporal interpolation or cross-modal style transfer has great potential to rescue live imaging data and limit photodamage. Yet we believe the focus should be on maintaining light-induced damage at levels that preserve natural cell behaviour. In this Opinion piece, we argue that a shift in role for AIs is needed - AI should be used to extract rich insights from gentle imaging rather than recover compromised data from harsh illumination. Although AI can enhance imaging, our ultimate goal should be to uncover biological truths, not just retrieve data. It is essential to prioritize minimizing photodamage over merely pushing technical limits. Our approach is aimed towards gentle acquisition and observation of undisturbed living systems, aligning with the essence of live-cell fluorescence microscopy.

Content-aware frame interpolation (CAFI): deep learning-based temporal super-resolution for fast bioimaging

The development of high-resolution microscopes has made it possible to investigate cellular processes in 3D and over time. However, observing fast cellular dynamics remains challenging because of photobleaching and phototoxicity. Here we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo and Depth-Aware Video Frame Interpolation, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series post-acquisition. We show that CAFI is capable of understanding the motion context of biological structures and can perform better than standard interpolation methods. We benchmark CAFI's performance on 12 different datasets, obtained from four different microscopy modalities, and demonstrate its capabilities for single-particle tracking and nuclear segmentation. CAFI potentially allows for reduced light exposure and phototoxicity on the sample for improved long-term live-cell imaging. The models and the training and testing data are available via the ZeroCostDL4Mic platform.

A versatile Wavelet-Enhanced CNN-Transformer for improved fluorescence microscopy image restoration

Fluorescence microscopes are indispensable tools for the life science research community. Nevertheless, the presence of optical component limitations, coupled with the maximum photon budget that the specimen can tolerate, inevitably leads to a decline in imaging quality and a lack of useful signals. Therefore, image restoration becomes essential for ensuring high-quality and accurate analyses. This paper presents the Wavelet-Enhanced Convolutional-Transformer (WECT), a novel deep learning technique developed specifically for the purpose of reducing noise in microscopy images and attaining super-resolution. Unlike traditional approaches, WECT integrates wavelet transform and inverse-transform for multi-resolution image decomposition and reconstruction, resulting in an expanded receptive field for the network without compromising information integrity. Subsequently, multiple consecutive parallel CNN-Transformer modules are utilized to collaboratively model local and global dependencies, thus facilitating the extraction of more comprehensive and diversified deep features. In addition, the incorporation of generative adversarial networks (GANs) into WECT enhances its capacity to generate high perceptual quality microscopic images. Extensive experiments have demonstrated that the WECT framework outperforms current state-of-the-art restoration methods on real fluorescence microscopy data under various imaging modalities and conditions, in terms of quantitative and qualitative analysis.

Antimicrobial resistance crisis: could artificial intelligence be the solution?

Antimicrobial resistance is a global public health threat, and the World Health Organization (WHO) has announced a priority list of the most threatening pathogens against which novel antibiotics need to be developed. The discovery and introduction of novel antibiotics are time-consuming and expensive. According to WHO's report of antibacterial agents in clinical development, only 18 novel antibiotics have been approved since 2014. Therefore, novel antibiotics are critically needed. Artificial intelligence (AI) has been rapidly applied to drug development since its recent technical breakthrough and has dramatically improved the efficiency of the discovery of novel antibiotics. Here, we first summarized recently marketed novel antibiotics, and antibiotic candidates in clinical development. In addition, we systematically reviewed the involvement of AI in antibacterial drug development and utilization, including small molecules, antimicrobial peptides, phage therapy, essential oils, as well as resistance mechanism prediction, and antibiotic stewardship.

Evaluation of Swin Transformer and knowledge transfer for denoising of super-resolution structured illumination microscopy data

BACKGROUND

Convolutional neural network (CNN)-based methods have shown excellent performance in denoising and reconstruction of super-resolved structured illumination microscopy (SR-SIM) data. Therefore, CNN-based architectures have been the focus of existing studies. However, Swin Transformer, an alternative and recently proposed deep learning-based image restoration architecture, has not been fully investigated for denoising SR-SIM images. Furthermore, it has not been fully explored how well transfer learning strategies work for denoising SR-SIM images with different noise characteristics and recorded cell structures for these different types of deep learning-based methods. Currently, the scarcity of publicly available SR-SIM datasets limits the exploration of the performance and generalization capabilities of deep learning methods.

RESULTS

In this work, we present SwinT-fairSIM, a novel method based on the Swin Transformer for restoring SR-SIM images with a low signal-to-noise ratio. The experimental results show that SwinT-fairSIM outperforms previous CNN-based denoising methods. Furthermore, as a second contribution, two types of transfer learning-namely, direct transfer and fine-tuning-were benchmarked in combination with SwinT-fairSIM and CNN-based methods for denoising SR-SIM data. Direct transfer did not prove to be a viable strategy, but fine-tuning produced results comparable to conventional training from scratch while saving computational time and potentially reducing the amount of training data required. As a third contribution, we publish four datasets of raw SIM images and already reconstructed SR-SIM images. These datasets cover two different types of cell structures, tubulin filaments and vesicle structures. Different noise levels are available for the tubulin filaments.

CONCLUSION

The SwinT-fairSIM method is well suited for denoising SR-SIM images. By fine-tuning, already trained models can be easily adapted to different noise characteristics and cell structures. Furthermore, the provided datasets are structured in a way that the research community can readily use them for research on denoising, super-resolution, and transfer learning strategies.

Perception-oriented generative adversarial network for retinal fundus image super-resolution

Retinal fundus imaging is a crucial diagnostic tool in ophthalmology, enabling the early detection and monitoring of various ocular diseases. However, capturing high-resolution fundus images often presents challenges due to factors such as defocusing and diffraction in the digital imaging process, limited shutter speed, sensor unit density, and random noise in the image sensor or during image transmission. Super-resolution techniques offer a promising solution to overcome these limitations and enhance the visual details in retinal fundus images. Since the retina has rich texture details, the super-resolution images often introduce artifacts into texture details and lose some fine retinal vessel structures. To improve the perceptual quality of the retinal fundus image, a generative adversarial network that consists of a generator and a discriminator is proposed. The proposed generator mainly comprises 23 multi-scale feature extraction blocks, an image segmentation network, and 23 residual-in-residual dense blocks. These components are employed to extract features at different scales, acquire the retinal vessel grayscale image, and extract retinal vascular features, respectively. The generator has two branches that are mainly responsible for extracting global features and vascular features, respectively. The extracted features from the two branches are fused to better restore the super-resolution image. The proposed generator can restore more details and more accurate fine vessel structures in retinal images. The improved discriminator is proposed by introducing our designed attention modules to help the generator yield clearer super-resolution images. Additionally, an artifact loss function is also introduced to enhance the generative adversarial network, enabling more accurate measurement of the disparity between the high-resolution image and the restored image. Experimental results show that the generated images obtained by our proposed method have a better perceptual quality than the state-of-the-art image super-resolution methods.

Physics-informed deep neural network for image denoising

Image enhancement deep neural networks (DNN) can improve signal to noise ratio or resolution of optically collected visual information. The literature reports a variety of approaches with varying effectiveness. All these algorithms rely on arbitrary data (the pixels' count-rate) normalization, making their performance strngly affected by dataset or user-specific data pre-manipulation. We developed a DNN algorithm capable to enhance images signal-to-noise surpassing previous algorithms. Our model stems from the nature of the photon detection process which is characterized by an inherently Poissonian statistics. Our algorithm is thus driven by distance between probability functions instead than relying on the sole count-rate, producing high performance results especially in high-dynamic-range images. Moreover, it does not require any arbitrary image renormalization other than the transformation of the camera's count-rate into photon-number.

Compound weighted fusion evaluation and optimization of intelligent tracking algorithm in radar seeker

This paper designs a hierarchical weighted fusion evaluation/optimization scheme for the radar seeker neural network (NN) tracking algorithm. The first weighted fusion of closed-loop performance index is carried out to exclude the hardware influence on algorithm evaluation. Then, according to different tracking scenarios, the tracking index is divided into different periods; a single period score is given by a linear-nonlinear hybrid scoring mechanism. Furthermore, in a single index, the internal scores of different time periods are weighted and fused for the second time to obtain the index overall score. Finally, the third weighted fusion of the multi-index scores obtains the comprehensive score of the algorithm. We design the parameter evaluation case sets and repeat the aforementioned compound weighting; hence the case with the highest comprehensive score is obtained. Finally, the algorithm is optimized by the highest-score case. The experiment using fuzzy NN radar seeker verifies the effectiveness of the method.

Video-rate 3D imaging of living cells using Fourier view-channel-depth light field microscopy

Interrogation of subcellular biological dynamics occurring in a living cell often requires noninvasive imaging of the fragile cell with high spatiotemporal resolution across all three dimensions. It thereby poses big challenges to modern fluorescence microscopy implementations because the limited photon budget in a live-cell imaging task makes the achievable performance of conventional microscopy approaches compromise between their spatial resolution, volumetric imaging speed, and phototoxicity. Here, we incorporate a two-stage view-channel-depth (VCD) deep-learning reconstruction strategy with a Fourier light-field microscope based on diffractive optical element to realize fast 3D super-resolution reconstructions of intracellular dynamics from single diffraction-limited 2D light-filed measurements. This VCD-enabled Fourier light-filed imaging approach (F-VCD), achieves video-rate (50 volumes per second) 3D imaging of intracellular dynamics at a high spatiotemporal resolution of ~180 nm × 180 nm × 400 nm and strong noise-resistant capability, with which light field images with a signal-to-noise ratio (SNR) down to -1.62 dB could be well reconstructed. With this approach, we successfully demonstrate the 4D imaging of intracellular organelle dynamics, e.g., mitochondria fission and fusion, with ~5000 times of observation.

Live-cell imaging in the deep learning era

Live imaging is a powerful tool, enabling scientists to observe living organisms in real time. In particular, when combined with fluorescence microscopy, live imaging allows the monitoring of cellular components with high sensitivity and specificity. Yet, due to critical challenges (i.e., drift, phototoxicity, dataset size), implementing live imaging and analyzing the resulting datasets is rarely straightforward. Over the past years, the development of bioimage analysis tools, including deep learning, is changing how we perform live imaging. Here we briefly cover important computational methods aiding live imaging and carrying out key tasks such as drift correction, denoising, super-resolution imaging, artificial labeling, tracking, and time series analysis. We also cover recent advances in self-driving microscopy.

Roadmap on Label-Free Super-Resolution Imaging

Label-free super-resolution (LFSR) imaging relies on light-scattering processes in nanoscale objects without a need for fluorescent (FL) staining required in super-resolved FL microscopy. The objectives of this Roadmap are to present a comprehensive vision of the developments, the state-of-the-art in this field, and to discuss the resolution boundaries and hurdles which need to be overcome to break the classical diffraction limit of the LFSR imaging. The scope of this Roadmap spans from the advanced interference detection techniques, where the diffraction-limited lateral resolution is combined with unsurpassed axial and temporal resolution, to techniques with true lateral super-resolution capability which are based on understanding resolution as an information science problem, on using novel structured illumination, near-field scanning, and nonlinear optics approaches, and on designing superlenses based on nanoplasmonics, metamaterials, transformation optics, and microsphere-assisted approaches. To this end, this Roadmap brings under the same umbrella researchers from the physics and biomedical optics communities in which such studies have often been developing separately. The ultimate intent of this paper is to create a vision for the current and future developments of LFSR imaging based on its physical mechanisms and to create a great opening for the series of articles in this field.

Deep learning enables stochastic optical reconstruction microscopy-like superresolution image reconstruction from conventional microscopy

Despite its remarkable potential for transforming low-resolution images, deep learning faces significant challenges in achieving high-quality superresolution microscopy imaging from wide-field (conventional) microscopy. Here, we present X-Microscopy, a computational tool comprising two deep learning subnets, UR-Net-8 and X-Net, which enables STORM-like superresolution microscopy image reconstruction from wide-field images with input-size flexibility. X-Microscopy was trained using samples of various subcellular structures, including cytoskeletal filaments, dot-like, beehive-like, and nanocluster-like structures, to generate prediction models capable of producing images of comparable quality to STORM-like images. In addition to enabling multicolour superresolution image reconstructions, X-Microscopy also facilitates superresolution image reconstruction from different conventional microscopic systems. The capabilities of X-Microscopy offer promising prospects for making superresolution microscopy accessible to a broader range of users, going beyond the confines of well-equipped laboratories.

Combination of deep learning and 2D CARS figures for identification of amyloid-β plaques

In vivo imaging and accurate identification of amyloid-β (Aβ) plaque are crucial in Alzheimer's disease (AD) research. In this work, we propose to combine the coherent anti-Stokes Raman scattering (CARS) microscopy, a powerful detection technology for providing Raman spectra and label-free imaging, with deep learning to distinguish Aβ from non-Aβ regions in AD mice brains in vivo. The 1D CARS spectra is firstly converted to 2D CARS figures by using two different methods: spectral recurrence plot (SRP) and spectral Gramian angular field (SGAF). This can provide more learnable information to the network, improving the classification precision. We then devise a cross-stage attention network (CSAN) that automatically learns the features of Aβ plaques and non-Aβ regions by taking advantage of the computational advances in deep learning. Our algorithm yields higher accuracy, precision, sensitivity and specificity than the results of conventional multivariate statistical analysis method and 1D CARS spectra combined with deep learning, demonstrating its competence in identifying Aβ plaques. Last but not least, the CSAN framework requires no prior information on the imaging modality and may be applicable to other spectroscopy analytical fields.

A novel multi-frame wavelet generative adversarial network for scattering reconstruction of structured illumination microscopy

Objective. Structured illumination microscopy (SIM) is widely used in various fields of life science research. In clinical practice, it has low phototoxicity, fast imaging speed and no special fluorescent markers. However, SIM is still affected by the scattering medium of biological tissues, resulting in insufficient resolution of the obtained images, which limits the development of life sciences. A novel multi-frame wavelet generation adversarial network (MWGAN) is proposed to improve the scattering reconstruction capability of SIM.Approach. MWGAN is based on two components derived from the original image. A generative adversarial network constructed by wavelet transform is trained to reconstruct some complex details in the cell structure. Multi-frame adversarial network is used to obtain the inter-frame information of the image and use the complementary information of the before and after frames to improve the quality of the model reconstruction.Results. To demonstrate the robustness of MWGAN, multiple low-quality SIM image datasets are tested. Compared with the state-of-the-art methods, the proposed method achieves superior performance in both of the subjective and objective evaluation.Conclusion. MWGAN is effective for improving the clarity of SIM images. Meanwhile, the SIM images reconstructed by multiple frames improve the reconstruction quality of complex regions and allow clearer and dynamic observation of cellular functions.

Three-dimensional structured illumination microscopy with enhanced axial resolution

The axial resolution of three-dimensional structured illumination microscopy (3D SIM) is limited to ∼300 nm. Here we present two distinct, complementary methods to improve axial resolution in 3D SIM with minimal or no modification to the optical system. We show that placing a mirror directly opposite the sample enables four-beam interference with higher spatial frequency content than 3D SIM illumination, offering near-isotropic imaging with ∼120-nm lateral and 160-nm axial resolution. We also developed a deep learning method achieving ∼120-nm isotropic resolution. This method can be combined with denoising to facilitate volumetric imaging spanning dozens of timepoints. We demonstrate the potential of these advances by imaging a variety of cellular samples, delineating the nanoscale distribution of vimentin and microtubule filaments, observing the relative positions of caveolar coat proteins and lysosomal markers and visualizing cytoskeletal dynamics within T cells in the early stages of immune synapse formation.

Approximation of smooth functionals using deep ReLU networks

In recent years, deep neural networks have been employed to approximate nonlinear continuous functionals F defined on Lp([-1,1]s) for 1≤p≤∞. However, the existing theoretical analysis in the literature either is unsatisfactory due to the poor approximation results, or does not apply to the rectified linear unit (ReLU) activation function. This paper aims to investigate the approximation power of functional deep ReLU networks in two settings: F is continuous with restrictions on the modulus of continuity, and F has higher order Fréchet derivatives. A novel functional network structure is proposed to extract features of higher order smoothness harbored by the target functional F. Quantitative rates of approximation in terms of the depth, width and total number of weights of neural networks are derived for both settings. We give logarithmic rates when measuring the approximation error on the unit ball of a Hölder space. In addition, we establish nearly polynomial rates (i.e., rates of the form exp-a(logM)b with a>0,0 Collapse

Key Words

• Approximation theory
• Deep learning theory
• Fréchet derivative
• Polynomial rates
• ReLU
• Smooth functionals

Collapse

MESH Headings

• Neural Networks, Computer
• Algorithms

Collapse

Grants Collapse

Affiliation(s)

Linhao Song School of Mathematical Science, Beihang University, Beijing, China; School of Data Science, City University of Hong Kong, Kowloon, Hong Kong.
Ying Liu Laboratory for AI-Powered Financial Technologies, Hong Kong Science Park, Shatin, New Territories, Hong Kong.
Jun Fan Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong.
Ding-Xuan Zhou School of Mathematics and Statistics, University of Sydney, Sydney, NSW 2006, Australia.

Collapse

Enhancing Total Optical Throughput of Microscopy with Deep Learning for Intravital Observation

The significance of performing large-depth dynamic microscopic imaging in vivo for life science research cannot be overstated. However, the optical throughput of the microscope limits the available information per unit of time, i.e., it is difficult to obtain both high spatial and temporal resolution at once. Here, a method is proposed to construct a kind of intravital microscopy with high optical throughput, by making near-infrared-II (NIR-II, 900-1880 nm) wide-field fluorescence microscopy learn from two-photon fluorescence microscopy based on a scale-recurrent network. Using this upgraded NIR-II fluorescence microscope, vessels in the opaque brain of a rodent are reconstructed three-dimensionally. Five-fold axial and thirteen-fold lateral resolution improvements are achieved without sacrificing temporal resolution and light utilization. Also, tiny cerebral vessel dilatations in early acute respiratory failure mice are observed, with this high optical throughput NIR-II microscope at an imaging speed of 30 fps.

Deep-MSIM: Fast Image Reconstruction with Deep Learning in Multifocal Structured Illumination Microscopy

Fast and precise reconstruction algorithm is desired for for multifocal structured illumination microscopy (MSIM) to obtain the super-resolution image. This work proposes a deep convolutional neural network (CNN) to learn a direct mapping from raw MSIM images to super-resolution image, which takes advantage of the computational advances of deep learning to accelerate the reconstruction. The method is validated on diverse biological structures and in vivo imaging of zebrafish at a depth of 100 µm. The results show that high-quality, super-resolution images can be reconstructed in one-third of the runtime consumed by conventional MSIM method, without compromising spatial resolution. Last but not least, a fourfold reduction in the number of raw images required for reconstruction is achieved by using the same network architecture, yet with different training data.

Improved sparse domain super-resolution reconstruction algorithm based on CMUT

A novel breast ultrasound tomography system based on a circular array of capacitive micromechanical ultrasound transducers (CMUT) has broad application prospects. However, the images produced by this system are not suitable as input for the training phase of the super-resolution (SR) reconstruction algorithm. To solve the problem, this paper proposes an improved medical image super-resolution (MeSR) method based on the sparse domain. First, we use the simultaneous algebraic reconstruction technique (SART) with high imaging accuracy to reconstruct the image into a training image in a sparse domain model. Secondly, we denoise and enhance the contrast of the SART images to obtain improved detail images before training the dictionary. Then, we use the original detail image as the guide image to further process the improved detail image. Therefore, a high-precision dictionary was obtained during the testing phase and applied to filtered back projection SR reconstruction. We compared the proposed algorithm with previously reported algorithms in the Shepp Logan model and the model based on the CMUT background. The results showed significant improvements in peak signal-to-noise ratio, entropy, and average gradient compared to previously reported algorithms. The experimental results demonstrated that the proposed MeSR method can use noisy reconstructed images as input for the training phase of the SR algorithm and produce excellent visual effects.

Scientific discovery in the age of artificial intelligence

Artificial intelligence (AI) is being increasingly integrated into scientific discovery to augment and accelerate research, helping scientists to generate hypotheses, design experiments, collect and interpret large datasets, and gain insights that might not have been possible using traditional scientific methods alone. Here we examine breakthroughs over the past decade that include self-supervised learning, which allows models to be trained on vast amounts of unlabelled data, and geometric deep learning, which leverages knowledge about the structure of scientific data to enhance model accuracy and efficiency. Generative AI methods can create designs, such as small-molecule drugs and proteins, by analysing diverse data modalities, including images and sequences. We discuss how these methods can help scientists throughout the scientific process and the central issues that remain despite such advances. Both developers and users of AI toolsneed a better understanding of when such approaches need improvement, and challenges posed by poor data quality and stewardship remain. These issues cut across scientific disciplines and require developing foundational algorithmic approaches that can contribute to scientific understanding or acquire it autonomously, making them critical areas of focus for AI innovation.

Advanced imaging techniques for tracking drug dynamics at the subcellular level

Optical microscopes are an important imaging tool that have effectively advanced the development of modern biomedicine. In recent years, super-resolution microscopy (SRM) has become one of the most popular techniques in the life sciences, especially in the field of living cell imaging. SRM has been used to solve many problems in basic biological research and has great potential in clinical application. In particular, the use of SRM to study drug delivery and kinetics at the subcellular level enables researchers to better study drugs' mechanisms of action and to assess the efficacy of their targets in vivo. The purpose of this paper is to review the recent advances in SRM and to highlight some of its applications in assessing subcellular drug dynamics.

Resolution enhancement with a task-assisted GAN to guide optical nanoscopy image analysis and acquisition

Super-resolution fluorescence microscopy methods enable the characterization of nanostructures in living and fixed biological tissues. However, they require the adjustment of multiple imaging parameters while attempting to satisfy conflicting objectives, such as maximizing spatial and temporal resolution while minimizing light exposure. To overcome the limitations imposed by these trade-offs, post-acquisition algorithmic approaches have been proposed for resolution enhancement and image-quality improvement. Here we introduce the task-assisted generative adversarial network (TA-GAN), which incorporates an auxiliary task (for example, segmentation, localization) closely related to the observed biological nanostructure characterization. We evaluate how the TA-GAN improves generative accuracy over unassisted methods, using images acquired with different modalities such as confocal, bright-field, stimulated emission depletion and structured illumination microscopy. The TA-GAN is incorporated directly into the acquisition pipeline of the microscope to predict the nanometric content of the field of view without requiring the acquisition of a super-resolved image. This information is used to automatically select the imaging modality and regions of interest, optimizing the acquisition sequence by reducing light exposure. Data-driven microscopy methods like the TA-GAN will enable the observation of dynamic molecular processes with spatial and temporal resolutions that surpass the limits currently imposed by the trade-offs constraining super-resolution microscopy.