Fast detection of micro-objects using scanning electrochemical microscopy based on visual recognition and machine learning

Scanning electrochemical microscopy (SECM) is a scanning probe microscope with an ultramicroelectrode (UME) as a probe. The technique is advantageous in the characterization of the electrochemical properties of surfaces. However, the limitations, such as slow imaging and many functions depending on the user, only allow us to use some of the possibilities. Therefore, we applied visual recognition and machine learning to detect micro-objects from the image and determine their electrochemical activity. The reconstruction of the image from several approach curves allows it to scan faster and detect active areas of the sample. Therefore, the scanning time and presence of the user is diminished. An automated scanning electrochemical microscope with visual recognition has been developed using commercially available modules, relatively low-cost components, design, software solutions proven in other fields, and an original control and data fusion algorithm.

Unsupervised machine learning combined with 4D scanning transmission electron microscopy for bimodal nanostructural analysis

Unsupervised machine learning techniques have been combined with scanning transmission electron microscopy (STEM) to enable comprehensive crystal structure analysis with nanometer spatial resolution. In this study, we investigated large-scale data obtained by four-dimensional (4D) STEM using dimensionality reduction techniques such as non-negative matrix factorization (NMF) and hierarchical clustering with various optimization methods. We developed software scripts incorporating knowledge of electron diffraction and STEM imaging for data preprocessing, NMF, and hierarchical clustering. Hierarchical clustering was performed using cross-correlation instead of conventional Euclidean distances, resulting in rotation-corrected diffractions and shift-corrected maps of major components. An experimental analysis was conducted on a high-pressure-annealed metallic glass, Zr-Cu-Al, revealing an amorphous matrix and crystalline precipitates with an average diameter of approximately 7 nm, which were challenging to detect using conventional STEM techniques. Combining 4D-STEM and optimized unsupervised machine learning enables comprehensive bimodal (i.e., spatial and reciprocal) analyses of material nanostructures.

Evaluating Stage Motion for Automated Electron Microscopy

Precise control is an essential and elusive quality of emerging self-driving transmission electron microscopes (TEMs). It is widely understood these instruments must be capable of performing rapid, high-volume, and arbitrary movements for practical self-driving operation. However, stage movements are difficult to automate at scale, owing to mechanical instability, hysteresis, and thermal drift. Such difficulties pose major barriers to artificial intelligence-directed microscope designs that require repeatable, precise movements. To guide design of emerging instruments, it is necessary to understand the behavior of existing mechanisms to identify rate limiting steps for full autonomy. Here, we describe a general framework to evaluate stage motion in any TEM. We define metrics to evaluate stage degrees of freedom, propose solutions to improve performance, and comment on fundamental limits to automated experimentation using present hardware.

Fast reconstruction of scanning transmission electron microscopy images using Markov random field model

In this study, we proposed a fast method of reconstruction for scanning transmission electron microscopy images. The proposed method is based on the Markov random field model and Bayesian inference, and we found that the method can reconstruct such images of sizes 512 × 512 and 264 × 240 in less than 200 ms and 100 ms, respectively. Furthermore, we showed that the method of reconstruction from multiple images without averaging them has better reconstruction performance than that from the averaged image.

Quantifying noise limitations of neural network segmentations in high-resolution transmission electron microscopy

Motivated by the need for low electron dose transmission electron microscopy imaging, we report the optimal frame dose (i.e.e-/Å2) range for object detection and segmentation tasks with neural networks. The MSD-net architecture shows promising abilities over the industry standard U-net architecture in generalising to frame doses below the range included in the training set, for both simulated and experimental images. It also presents a heightened ability to learn from lower dose images. The MSD-net displays mild visibility of a Au nanoparticle at 20-30 e-/Å2, and converges at 200 e-/Å2 where a full segmentation of the nanoparticle is achieved. Between 30 and 200 e-/Å2 object detection applications are still possible. This work also highlights the importance of modelling the modulation transfer function when training with simulated images for applications on images acquired with scintillator based detectors such as the Gatan Oneview camera. A parametric form of the modulation transfer function is applied with varying ranges of parameters, and the effects on low electron dose segmentation is presented.

Deep learning analysis on transmission electron microscope imaging of atomic defects in two-dimensional materials

Defects are prevalent in two-dimensional (2D) materials due to thermal equilibrium and processing kinetics. The presence of various defect types can affect material properties significantly. With the development of the advanced transmission electron microscopy (TEM), the property-related structures could be investigated in multiple dimensions. It produces TEM datasets containing a large amount of information. Traditional data analysis is influenced by the subjectivity of researchers, and manual analysis is inefficient and imprecise. Recent developments in deep learning provide robust methods for the quantitative identification of defects in 2D materials efficiently and precisely. Taking advantage of big data, it breaks the limitations of TEM as a local characterization tool, making TEM an intelligent macroscopic analysis method. In this review, the recent developments in the TEM data analysis of defects in 2D materials using deep learning technology are summarized. Initially, an in-depth examination of the distinctions between TEM and natural images is presented. Subsequently, a comprehensive exploration of TEM data analysis ensues, encompassing denoising, point defects, line defects, planar defects, quantitative analysis, and applications. Furthermore, an exhaustive assessment of the significant obstacles encountered in the accurate identification of distinct structures is also provided.

Segmentation in large-scale cellular electron microscopy with deep learning: A literature survey

Electron microscopy (EM) enables high-resolution imaging of tissues and cells based on 2D and 3D imaging techniques. Due to the laborious and time-consuming nature of manual segmentation of large-scale EM datasets, automated segmentation approaches are crucial. This review focuses on the progress of deep learning-based segmentation techniques in large-scale cellular EM throughout the last six years, during which significant progress has been made in both semantic and instance segmentation. A detailed account is given for the key datasets that contributed to the proliferation of deep learning in 2D and 3D EM segmentation. The review covers supervised, unsupervised, and self-supervised learning methods and examines how these algorithms were adapted to the task of segmenting cellular and sub-cellular structures in EM images. The special challenges posed by such images, like heterogeneity and spatial complexity, and the network architectures that overcame some of them are described. Moreover, an overview of the evaluation measures used to benchmark EM datasets in various segmentation tasks is provided. Finally, an outlook of current trends and future prospects of EM segmentation is given, especially with large-scale models and unlabeled images to learn generic features across EM datasets.

Pair-Variational Autoencoders for Linking and Cross-Reconstruction of Characterization Data from Complementary Structural Characterization Techniques

In materials research, structural characterization often requires multiple complementary techniques to obtain a holistic morphological view of a synthesized material. Depending on the availability and accessibility of the different characterization techniques (e.g., scattering, microscopy, spectroscopy), each research facility or academic research lab may have access to high-throughput capability in one technique but face limitations (sample preparation, resolution, access time) with other technique(s). Furthermore, one type of structural characterization data may be easier to interpret than another (e.g., microscopy images are easier to interpret than small-angle scattering profiles). Thus, it is useful to have machine learning models that can be trained on paired structural characterization data from multiple techniques (easy and difficult to interpret, fast and slow in data collection or sample preparation) so that the model can generate one set of characterization data from the other. In this paper we demonstrate one such machine learning workflow, Pair-Variational Autoencoders (PairVAE), that works with data from small-angle X-ray scattering (SAXS) that present information about bulk morphology and images from scanning electron microscopy (SEM) that present two-dimensional local structural information on the sample. Using paired SAXS and SEM data of newly observed block copolymer assembled morphologies [open access data from Doerk G. S.; et al. Sci. Adv.2023, 9 ( (2), ), eadd3687], we train our PairVAE. After successful training, we demonstrate that the PairVAE can generate SEM images of the block copolymer morphology when it takes as input that sample's corresponding SAXS 2D pattern and vice versa. This method can be extended to other soft material morphologies as well and serves as a valuable tool for easy interpretation of 2D SAXS patterns as well as an engine for generating ensembles of similar microscopy images to create a database for other downstream calculations of structure-property relationships.

An Unsupervised Classification Algorithm for Heterogeneous Cryo-EM Projection Images Based on Autoencoders

Heterogeneous three-dimensional (3D) reconstruction in single-particle cryo-electron microscopy (cryo-EM) is an important but very challenging technique for recovering the conformational heterogeneity of flexible biological macromolecules such as proteins in different functional states. Heterogeneous projection image classification is a feasible solution to solve the structural heterogeneity problem in single-particle cryo-EM. The majority of heterogeneous projection image classification methods are developed using supervised learning technology or require a large amount of a priori knowledge, such as the orientations or common lines of the projection images, which leads to certain limitations in their practical applications. In this paper, an unsupervised heterogeneous cryo-EM projection image classification algorithm based on autoencoders is proposed, which only needs to know the number of heterogeneous 3D structures in the dataset and does not require any labeling information of the projection images or other a priori knowledge. A simple autoencoder with multi-layer perceptrons trained in iterative mode and a complex autoencoder with residual networks trained in one-pass learning mode are implemented to convert heterogeneous projection images into latent variables. The extracted high-dimensional features are reduced to two dimensions using the uniform manifold approximation and projection dimensionality reduction algorithm, and then clustered using the spectral clustering algorithm. The proposed algorithm is applied to two heterogeneous cryo-EM datasets for heterogeneous 3D reconstruction. Experimental results show that the proposed algorithm can effectively extract category features of heterogeneous projection images and achieve high classification and reconstruction accuracy, indicating that the proposed algorithm is effective for heterogeneous 3D reconstruction in single-particle cryo-EM.

Automated analysis of transmission electron micrographs of metallic nanoparticles by machine learning

Metallic nanoparticles were analysed with respect to size and shape by a machine learning approach. This involved a separation of particles from the background (segmentation), a separation of overlapping particles, and the identification of individual particles. An algorithm to separate overlapping particles, based on ultimate erosion of convex shapes (UECS), was implemented. Finally, particle properties like size, circularity, equivalent diameter, and Feret diameter were computed for each particle of the whole particle population. Thus, particle size distributions can be easily created based on the various parameters. However, strongly overlapping particles are difficult and sometimes impossible to separate because of an a priori unknown shape of a particle that is partially lying in the shadow of another particle. The program is able to extract information from a sequence of images of the same sample, thereby increasing the number of analysed nanoparticles to several thousands. The machine learning approach is well-suited to identify particles at only limited particle-to-background contrast as is demonstrated for ultrasmall gold nanoparticles (2 nm).

Attention-guided multi-scale deep object detection framework for lymphocyte analysis in IHC histological images

Tumor-infiltrating lymphocytes are specialized lymphocytes that can detect and kill cancerous cells. Their detection poses many challenges due to significant morphological variations, overlapping occurrence, artifact regions and high-class resemblance between clustered areas and artifacts. In this regard, a Lymphocyte Analysis Framework based on Deep Convolutional neural network (DC-Lym-AF) is proposed to analyze lymphocytes in immunohistochemistry images. The proposed framework comprises (i) pre-processing, (ii) screening phase, (iii) localization phase and (iv) post-processing. In the screening phase, a custom convolutional neural network architecture (lymphocyte dilated network) is developed to screen lymphocytic regions by performing a patch-level classification. This proposed architecture uses dilated convolutions and shortcut connections to capture multi-level variations and ensure reference-based learning. In contrast, the localization phase utilizes an attention-guided multi-scale lymphocyte detector to detect lymphocytes. The proposed detector extracts refined and multi-scale features by exploiting dilated convolutions, attention mechanism and feature pyramid network (FPN) using its custom attention-aware backbone. The proposed DC-Lym-AF shows exemplary performance on the NuClick dataset compared with the existing detection models, with an F-score and precision of 0.84 and 0.83, respectively. We verified the generalizability of our proposed framework by participating in a publically open LYON'19 challenge. Results in terms of detection rate (0.76) and F-score (0.73) suggest that the proposed DC-Lym-AF can effectively detect lymphocytes in immunohistochemistry-stained images collected from different laboratories. In addition, its promising generalization on several datasets implies that it can be turned into a medical diagnostic tool to investigate various histopathological problems. Graphical Abstract.

Deep learning for automated size and shape analysis of nanoparticles in scanning electron microscopy

The automated analysis of nanoparticles, imaged by scanning electron microscopy, was implemented by a deep-learning (artificial intelligence) procedure based on convolutional neural networks (CNNs). It is possible to extract quantitative information on particle size distributions and particle shapes from pseudo-three-dimensional secondary electron micrographs (SE) as well as from two-dimensional scanning transmission electron micrographs (STEM). After separation of particles from the background (segmentation), the particles were cut out from the image to be classified by their shape (e.g. sphere or cube). The segmentation ability of STEM images was considerably enhanced by introducing distance- and intensity-based pixel weight loss maps. This forced the neural network to put emphasis on areas which separate adjacent particles. Partially covered particles were recognized by training and excluded from the analysis. The separation of overlapping particles, quality control procedures to exclude agglomerates, and the computation of quantitative particle size distribution data (equivalent particle diameter, Feret diameter, circularity) were included into the routine.

Deep learning-based synapse counting and synaptic ultrastructure analysis of electron microscopy images

BACKGROUND

Synapses are the connections between neurons in the central nervous system (CNS) or between neurons and other excitable cells in the peripheral nervous system (PNS), where electrical or chemical signals rapidly travel through one cell to another with high spatial precision. Synaptic analysis, based on synapse numbers and fine morphology, is the basis for understanding neurological functions and diseases. Manual analysis of synaptic structures in electron microscopy (EM) images is often limited by low efficiency and subjective bias.

NEW METHOD

We developed a multifunctional synaptic analysis system based on several advanced deep learning (DL) models. The system achieved synapse counting in low-magnification EM images and synaptic ultrastructure analysis in high-magnification EM images.

RESULTS

The synapse counting system based on ResNet18 and a Faster R-CNN model had a mean average precision (mAP) of 92.55%. For synaptic ultrastructure analysis, the Faster R-CNN model based on ResNet50 achieved a mAP of 91.60%, the DeepLab v3 + model based on ResNet50 enabled high performance in presynaptic and postsynaptic membrane segmentation with a global accuracy of 0.9811, and the Faster R-CNN model based on ResNet18 achieved a mAP of 91.41% for synaptic vesicle detection.

CONCLUSIONS

The proposed multifunctional synaptic analysis system may help to overcome the experimental bias inherent in manual analysis, thereby facilitating EM image-based synaptic function studies.

Structural analysis of melanosomes in living mammalian cells using scanning electron-assisted dielectric microscopy with deep neural network

Melanins are the main pigments found in mammals. Their synthesis and transfer to keratinocytes have been widely investigated for many years. However, analysis has been mainly carried out using fixed rather than live cells. In this study, we have analysed the melanosomes in living mammalian cells using newly developed scanning electron-assisted dielectric microscopy (SE-ADM). The melanosomes in human melanoma MNT-1 cells were observed as clear black particles in SE-ADM. The main structure of melanosomes was toroidal while that of normal melanocytes was ellipsoidal. In tyrosinase knockout MNT-1 cells, not only the black particles in the SE-ADM images but also the Raman shift of melanin peaks completely disappeared suggesting that the black particles were really melanosomes. We developed a deep neural network (DNN) system to automatically detect melanosomes in cells and analysed their diameter and roundness. In terms of melanosome morphology, the diameter of melanosomes in melanoma cells did not change while that in normal melanocytes increased during culture. The established DNN analysis system with SE-ADM can be used for other particles, e.g. exosomes, lysosomes, and other biological particles.