A machine learning approach for online automated optimization of super-resolution optical microscopy

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

Artificial Intelligence in Point-of-Care Biosensing: Challenges and Opportunities

The integration of artificial intelligence (AI) into point-of-care (POC) biosensing has the potential to revolutionize diagnostic methodologies by offering rapid, accurate, and accessible health assessment directly at the patient level. This review paper explores the transformative impact of AI technologies on POC biosensing, emphasizing recent computational advancements, ongoing challenges, and future prospects in the field. We provide an overview of core biosensing technologies and their use at the POC, highlighting ongoing issues and challenges that may be solved with AI. We follow with an overview of AI methodologies that can be applied to biosensing, including machine learning algorithms, neural networks, and data processing frameworks that facilitate real-time analytical decision-making. We explore the applications of AI at each stage of the biosensor development process, highlighting the diverse opportunities beyond simple data analysis procedures. We include a thorough analysis of outstanding challenges in the field of AI-assisted biosensing, focusing on the technical and ethical challenges regarding the widespread adoption of these technologies, such as data security, algorithmic bias, and regulatory compliance. Through this review, we aim to emphasize the role of AI in advancing POC biosensing and inform researchers, clinicians, and policymakers about the potential of these technologies in reshaping global healthcare landscapes.

Characterizing the blood-brain barrier and gut barrier with super-resolution imaging: opportunities and challenges

Brain and gut barriers have been receiving increasing attention in health and diseases including in psychiatry. Recent studies have highlighted changes in the blood-brain barrier and gut barrier structural properties, notably a loss of tight junctions, leading to hyperpermeability, passage of inflammatory mediators, stress vulnerability, and the development of depressive behaviors. To decipher the cellular processes actively contributing to brain and gut barrier function in health and disease, scientists can take advantage of neurophotonic tools and recent advances in super-resolution microscopy techniques to complement traditional imaging approaches like confocal and electron microscopy. Here, we summarize the challenges, pros, and cons of these innovative approaches, hoping that a growing number of scientists will integrate them in their study design exploring barrier-related properties and mechanisms.

Resolution enhancement with a task-assisted GAN to guide optical nanoscopy image analysis and acquisition

Super-resolution fluorescence microscopy methods enable the characterization of nanostructures in living and fixed biological tissues. However, they require the adjustment of multiple imaging parameters while attempting to satisfy conflicting objectives, such as maximizing spatial and temporal resolution while minimizing light exposure. To overcome the limitations imposed by these trade-offs, post-acquisition algorithmic approaches have been proposed for resolution enhancement and image-quality improvement. Here we introduce the task-assisted generative adversarial network (TA-GAN), which incorporates an auxiliary task (for example, segmentation, localization) closely related to the observed biological nanostructure characterization. We evaluate how the TA-GAN improves generative accuracy over unassisted methods, using images acquired with different modalities such as confocal, bright-field, stimulated emission depletion and structured illumination microscopy. The TA-GAN is incorporated directly into the acquisition pipeline of the microscope to predict the nanometric content of the field of view without requiring the acquisition of a super-resolved image. This information is used to automatically select the imaging modality and regions of interest, optimizing the acquisition sequence by reducing light exposure. Data-driven microscopy methods like the TA-GAN will enable the observation of dynamic molecular processes with spatial and temporal resolutions that surpass the limits currently imposed by the trade-offs constraining super-resolution microscopy.

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

In the last few years, electron microscopy has experienced a new methodological paradigm aimed to fix the bottlenecks and overcome the challenges of its analytical workflow. Machine learning and artificial intelligence are answering this call providing powerful resources towards automation, exploration, and development. In this review, we evaluate the state-of-the-art of machine learning applied to electron microscopy (and obliquely, to materials and nano-sciences). We start from the traditional imaging techniques to reach the newest higher-dimensionality ones, also covering the recent advances in spectroscopy and tomography. Additionally, the present review provides a practical guide for microscopists, and in general for material scientists, but not necessarily advanced machine learning practitioners, to straightforwardly apply the offered set of tools to their own research. To conclude, we explore the state-of-the-art of other disciplines with a broader experience in applying artificial intelligence methods to their research (e.g., high-energy physics, astronomy, Earth sciences, and even robotics, videogames, or marketing and finances), in order to narrow down the incoming future of electron microscopy, its challenges and outlook.

Event-driven acquisition for content-enriched microscopy

A common goal of fluorescence microscopy is to collect data on specific biological events. Yet, the event-specific content that can be collected from a sample is limited, especially for rare or stochastic processes. This is due in part to photobleaching and phototoxicity, which constrain imaging speed and duration. We developed an event-driven acquisition framework, in which neural-network-based recognition of specific biological events triggers real-time control in an instant structured illumination microscope. Our setup adapts acquisitions on-the-fly by switching between a slow imaging rate while detecting the onset of events, and a fast imaging rate during their progression. Thus, we capture mitochondrial and bacterial divisions at imaging rates that match their dynamic timescales, while extending overall imaging durations. Because event-driven acquisition allows the microscope to respond specifically to complex biological events, it acquires data enriched in relevant content.

Machine learning approaches for biomolecular, biophysical, and biomaterials research

A fluent conversation with a virtual assistant, person-tailored news feeds, and deep-fake images created within seconds-all those things that have been unthinkable for a long time are now a part of our everyday lives. What these examples have in common is that they are realized by different means of machine learning (ML), a technology that has fundamentally changed many aspects of the modern world. The possibility to process enormous amount of data in multi-hierarchical, digital constructs has paved the way not only for creating intelligent systems but also for obtaining surprising new insight into many scientific problems. However, in the different areas of biosciences, which typically rely heavily on the collection of time-consuming experimental data, applying ML methods is a bit more challenging: Here, difficulties can arise from small datasets and the inherent, broad variability, and complexity associated with studying biological objects and phenomena. In this Review, we give an overview of commonly used ML algorithms (which are often referred to as "machines") and learning strategies as well as their applications in different bio-disciplines such as molecular biology, drug development, biophysics, and biomaterials science. We highlight how selected research questions from those fields were successfully translated into machine readable formats, discuss typical problems that can arise in this context, and provide an overview of how to resolve those encountered difficulties.

Can super-resolution microscopy become a standard characterization technique for materials chemistry?

The characterization of newly synthesized materials is a cornerstone of all chemistry and nanotechnology laboratories. For this purpose, a wide array of analytical techniques have been standardized and are used routinely by laboratories across the globe. With these methods we can understand the structure, dynamics and function of novel molecular architectures and their relations with the desired performance, guiding the development of the next generation of materials. Moreover, one of the challenges in materials chemistry is the lack of reproducibility due to improper publishing of the sample preparation protocol. In this context, the recent adoption of the reporting standard MIRIBEL (Minimum Information Reporting in Bio-Nano Experimental Literature) for material characterization and details of experimental protocols aims to provide complete, reproducible and reliable sample preparation for the scientific community. Thus, MIRIBEL should be immediately adopted in publications by scientific journals to overcome this challenge. Besides current standard spectroscopy and microscopy techniques, there is a constant development of novel technologies that aim to help chemists unveil the structure of complex materials. Among them super-resolution microscopy (SRM), an optical technique that bypasses the diffraction limit of light, has facilitated the study of synthetic materials with multicolor ability and minimal invasiveness at nanometric resolution. Although still in its infancy, the potential of SRM to unveil the structure, dynamics and function of complex synthetic architectures has been highlighted in pioneering reports during the last few years. Currently, SRM is a sophisticated technique with many challenges in sample preparation, data analysis, environmental control and automation, and moreover the instrumentation is still expensive. Therefore, SRM is currently limited to expert users and is not implemented in characterization routines. This perspective discusses the potential of SRM to transition from a niche technique to a standard routine method for material characterization. We propose a roadmap for the necessary developments required for this purpose based on a collaborative effort from scientists and engineers across disciplines.

Pursuit of precision medicine: Systems biology approaches in Alzheimer's disease mouse models

Alzheimer's disease (AD) is a complex disease that is mediated by numerous factors and manifests in various forms. A systems biology approach to studying AD involves analyses of various body systems, biological scales, environmental elements, and clinical outcomes to understand the genotype to phenotype relationship that potentially drives AD development. Currently, there are many research investigations probing how modifiable and nonmodifiable factors impact AD symptom presentation. This review specifically focuses on how imaging modalities can be integrated into systems biology approaches using model mouse populations to link brain level functional and structural changes to disease onset and progression. Combining imaging and omics data promotes the classification of AD into subtypes and paves the way for precision medicine solutions to prevent and treat AD.

Deep-ROCS: from speckle patterns to superior-resolved images by deep learning in rotating coherent scattering microscopy

Rotating coherent scattering (ROCS) microscopy is a label-free imaging technique that overcomes the optical diffraction limit by adding up the scattered laser light from a sample obliquely illuminated from different angles. Although ROCS imaging achieves 150 nm spatial and 10 ms temporal resolution, simply summing different speckle patterns may cause loss of sample information. In this paper we present Deep-ROCS, a neural network-based technique that generates a superior-resolved image by efficient numerical combination of a set of differently illuminated images. We show that Deep-ROCS can reconstruct super-resolved images more accurately than conventional ROCS microscopy, retrieving high-frequency information from a small number (6) of speckle images. We demonstrate the performance of Deep-ROCS experimentally on 200 nm beads and by computer simulations, where we show its potential for even more complex structures such as a filament network.

A survey on applications of deep learning in microscopy image analysis

Advanced microscopy enables us to acquire quantities of time-lapse images to visualize the dynamic characteristics of tissues, cells or molecules. Microscopy images typically vary in signal-to-noise ratios and include a wealth of information which require multiple parameters and time-consuming iterative algorithms for processing. Precise analysis and statistical quantification are often needed for the understanding of the biological mechanisms underlying these dynamic image sequences, which has become a big challenge in the field. As deep learning technologies develop quickly, they have been applied in bioimage processing more and more frequently. Novel deep learning models based on convolution neural networks have been developed and illustrated to achieve inspiring outcomes. This review article introduces the applications of deep learning algorithms in microscopy image analysis, which include image classification, region segmentation, object tracking and super-resolution reconstruction. We also discuss the drawbacks of existing deep learning-based methods, especially on the challenges of training datasets acquisition and evaluation, and propose the potential solutions. Furthermore, the latest development of augmented intelligent microscopy that based on deep learning technology may lead to revolution in biomedical research.

STED nanoscopy - A novel way to image the pore space of geological materials

STED nanoscopy (Stimulated Emission Depletion). which can resolve details far below the diffraction barrier has been applied hitherto preferentially to life sciences. The method is however also ideal for the investigation of geological matrices containing transparent minerals, an application tested here, to our knowledge, for the first time. The measurements on altered granitic rock and sedimentary clay rock, both containing very fine-grained phases, were conducted successfully. The STED fluorophore was dissolved in C-14-labelled methylmethacrylate (C-14-MMA) monomer which was polymerised within the rock matrix, thereby labelling the pore space in the geomaterials. Double labelling provided by the C-14-labelled MMA enables autoradiography and scanning electron microscopy (SEM), providing necessary complementary information for characterisation and quantification of porosity distributions and mineral and structure identification. Promising perspectives for further investigations of geological matrices by using different fluorophores and the optimisation of measuring procedures or even higher resolution are discussed. The combination of these different methods enlarges the observation scale of porosity from nanometre to centimetre scale.

Challenges facing quantitative large-scale optical super-resolution, and some simple solutions

Optical super-resolution microscopy (SRM) has enabled biologists to visualize cellular structures with near-molecular resolution, giving unprecedented access to details about the amounts, sizes, and spatial distributions of macromolecules in the cell. Precisely quantifying these molecular details requires large datasets of high-quality, reproducible SRM images. In this review, we discuss the unique set of challenges facing quantitative SRM, giving particular attention to the shortcomings of conventional specimen preparation techniques and the necessity for optimal labeling of molecular targets. We further discuss the obstacles to scaling SRM methods, such as lengthy image acquisition and complex SRM data analysis. For each of these challenges, we review the recent advances in the field that circumvent these pitfalls and provide practical advice to biologists for optimizing SRM experiments.

Machine-Learning-Driven Synthesis of Carbon Dots with Enhanced Quantum Yields

Knowing the correlation of reaction parameters in the preparation process of carbon dots (CDs) is essential for optimizing the synthesis strategy, exploring exotic properties, and exploiting potential applications. However, the integrated screening experimental data on the synthesis of CDs are huge and noisy. Machine learning (ML) has recently been successfully used for the screening of high-performance materials. Here, we demonstrate how ML-based techniques can offer insight into the successful prediction, optimization, and acceleration of CDs' synthesis process. A regression ML model on hydrothermal-synthesized CDs is established capable of revealing the relationship between various synthesis parameters and experimental outcomes as well as enhancing the process-related properties such as the fluorescent quantum yield (QY). CDs exhibiting a strong green emission with QY up to 39.3% are obtained through the combined ML guidance and experimental verification. The mass of precursors and the volume of alkaline catalysts are identified as the most important features in the synthesis of high-QY CDs by the trained ML model. The CDs are applied as an ultrasensitive fluorescence probe for monitoring the Fe³⁺ ion because of their superior optical behaviors. The probe exhibits the linear response to the Fe³⁺ ion with a wide concentration range (0-150 μM), and its detection limit is 0.039 μM. Our findings demonstrate the great capability of ML to guide the synthesis of high-quality CDs, accelerating the development of intelligent material.

Activity-Dependent Remodeling of Synaptic Protein Organization Revealed by High Throughput Analysis of STED Nanoscopy Images

The organization of proteins in the apposed nanodomains of pre- and postsynaptic compartments is thought to play a pivotal role in synaptic strength and plasticity. As such, the alignment between pre- and postsynaptic proteins may regulate, for example, the rate of presynaptic release or the strength of postsynaptic signaling. However, the analysis of these structures has mainly been restricted to subsets of synapses, providing a limited view of the diversity of synaptic protein cluster remodeling during synaptic plasticity. To characterize changes in the organization of synaptic nanodomains during synaptic plasticity over a large population of synapses, we combined STimulated Emission Depletion (STED) nanoscopy with a Python-based statistical object distance analysis (pySODA), in dissociated cultured hippocampal circuits exposed to treatments driving different forms of synaptic plasticity. The nanoscale organization, characterized in terms of coupling properties, of presynaptic (Bassoon, RIM1/2) and postsynaptic (PSD95, Homer1c) scaffold proteins was differently altered in response to plasticity-inducing stimuli. For the Bassoon - PSD95 pair, treatments driving synaptic potentiation caused an increase in their coupling probability, whereas a stimulus driving synaptic depression had an opposite effect. To enrich the characterization of the synaptic cluster remodeling at the population level, we applied unsupervised machine learning approaches to include selected morphological features into a multidimensional analysis. This combined analysis revealed a large diversity of synaptic protein cluster subtypes exhibiting differential activity-dependent remodeling, yet with common features depending on the expected direction of plasticity. The expanded palette of synaptic features revealed by our unbiased approach should provide a basis to further explore the widely diverse molecular mechanisms of synaptic plasticity.

Single-shot 3D wide-field fluorescence imaging with a Computational Miniature Mesoscope

Fluorescence microscopes are indispensable to biology and neuroscience. The need for recording in freely behaving animals has further driven the development in miniaturized microscopes (miniscopes). However, conventional microscopes/miniscopes are inherently constrained by their limited space-bandwidth product, shallow depth of field (DOF), and inability to resolve three-dimensional (3D) distributed emitters. Here, we present a Computational Miniature Mesoscope (CM2) that overcomes these bottlenecks and enables single-shot 3D imaging across an 8 mm by 7 mm field of view and 2.5-mm DOF, achieving 7-μm lateral resolution and better than 200-μm axial resolution. The CM2 features a compact lightweight design that integrates a microlens array for imaging and a light-emitting diode array for excitation. Its expanded imaging capability is enabled by computational imaging that augments the optics by algorithms. We experimentally validate the mesoscopic imaging capability on 3D fluorescent samples. We further quantify the effects of scattering and background fluorescence on phantom experiments.

Neuronal activity remodels the F-actin based submembrane lattice in dendrites but not axons of hippocampal neurons

The nanoscale organization of the F-actin cytoskeleton in neurons comprises membrane-associated periodical rings, bundles, and longitudinal fibers. The F-actin rings have been observed predominantly in axons but only sporadically in dendrites, where fluorescence nanoscopy reveals various patterns of F-actin arranged in mixed patches. These complex dendritic F-actin patterns pose a challenge for investigating quantitatively their regulatory mechanisms. We developed here a weakly supervised deep learning segmentation approach of fluorescence nanoscopy images of F-actin in cultured hippocampal neurons. This approach enabled the quantitative assessment of F-actin remodeling, revealing the disappearance of the rings during neuronal activity in dendrites, but not in axons. The dendritic F-actin cytoskeleton of activated neurons remodeled into longitudinal fibers. We show that this activity-dependent remodeling involves \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Ca}^{2+}$$\end{document}Ca2+ and NMDA receptor-dependent mechanisms. This highly dynamic restructuring of dendritic F-actin based submembrane lattice into longitudinal fibers may serve to support activity-dependent membrane remodeling, protein trafficking and neuronal plasticity.

Individual Cloud-Based Fingerprint Operation Platform for Latent Fingerprint Identification Using Perovskite Nanocrystals as Eikonogen

Fingerprint formed through lifted papillary ridges is considered the best reference for personal identification. However, the currently available latent fingerprint (LFP) images often suffer from poor resolution, have a low degree of information, and require multifarious steps for identification. Herein, an individual Cloud-based fingerprint operation platform has been designed and fabricated to achieve high-definition LFPs analysis by using CsPbBr₃ perovskite nanocrystals (NCs) as eikonogen. Moreover, since CsPbBr₃ NCs have a special response to some fingerprint-associated amino acids, the proposed platform can be further used to detect metabolites on LFPs. Consequently, in virtue of Cloud computing and artificial intelligence (AI), this study has demonstrated a champion platform to realize the whole LFP identification analysis. In a double-blind simulative crime game, the enhanced LFP images can be easily obtained and used to lock the suspect accurately within one second on a smartphone, which can help investigators track the criminal clue and handle cases efficiently.

Enhancing neuronal chloride extrusion rescues α2/α3 GABAA-mediated analgesia in neuropathic pain

Spinal disinhibition has been hypothesized to underlie pain hypersensitivity in neuropathic pain. Apparently contradictory mechanisms have been reported, raising questions on the best target to produce analgesia. Here, we show that nerve injury is associated with a reduction in the number of inhibitory synapses in the spinal dorsal horn. Paradoxically, this is accompanied by a BDNF-TrkB-mediated upregulation of synaptic GABAARs and by an α1-to-α2GABAAR subunit switch, providing a mechanistic rationale for the analgesic action of the α2,3GABAAR benzodiazepine-site ligand L838,417 after nerve injury. Yet, we demonstrate that impaired Cl- extrusion underlies the failure of L838,417 to induce analgesia at high doses due to a resulting collapse in Cl- gradient, dramatically limiting the benzodiazepine therapeutic window. In turn, enhancing KCC2 activity not only potentiated L838,417-induced analgesia, it rescued its analgesic potential at high doses, revealing a novel strategy for analgesia in pathological pain, by combined targeting of the appropriate GABAAR-subtypes and restoring Cl- homeostasis.

Non-standard trajectories found by machine learning for evaporative cooling of 87Rb atoms

We present a machine-learning experiment involving evaporative cooling of gaseous ⁸⁷Rb atoms. The evaporation trajectory was optimized to maximize the number of atoms cooled down to a Bose-Einstein condensate using Bayesian optimization. After 300 trials within 3 hours, Bayesian optimization discovered trajectories that achieved atom numbers comparable with those of manual tuning by a human expert. Analysis of the machine-learned trajectories revealed minimum requirements for successful evaporative cooling. We found that the manually obtained curve and the machine-learned trajectories were quite similar in terms of evaporation efficiency, although the manual and machine-learned evaporation ramps were significantly different.

Chasing Uptake: Super-Resolution Microscopy in Endocytosis and Phagocytosis

Since their invention about two decades ago, super-resolution microscopes have become a method of choice in cell biology. Owing to a spatial resolution below 50 nm, smaller than the size of most organelles, and an order of magnitude better than the diffraction limit of conventional light microscopes, super-resolution microscopy is a powerful technique for resolving intracellular trafficking. In this review we discuss discoveries in endocytosis and phagocytosis that have been made possible by super-resolution microscopy - from uptake at the plasma membrane, endocytic coat formation, and cytoskeletal rearrangements to endosomal maturation. The detailed visualization of the diverse molecular assemblies that mediate endocytic uptake will provide a better understanding of how cells ingest extracellular material.