Bayesian Particle Instance Segmentation for Electron Microscopy Image Quantification

Machine Learning-Enabled Image Classification for Automated Electron Microscopy

Traditionally, materials discovery has been driven more by evidence and intuition than by systematic design. However, the advent of "big data" and an exponential increase in computational power have reshaped the landscape. Today, we use simulations, artificial intelligence (AI), and machine learning (ML) to predict materials characteristics, which dramatically accelerates the discovery of novel materials. For instance, combinatorial megalibraries, where millions of distinct nanoparticles are created on a single chip, have spurred the need for automated characterization tools. This paper presents an ML model specifically developed to perform real-time binary classification of grayscale high-angle annular dark-field images of nanoparticles sourced from these megalibraries. Given the high costs associated with downstream processing errors, a primary requirement for our model was to minimize false positives while maintaining efficacy on unseen images. We elaborate on the computational challenges and our solutions, including managing memory constraints, optimizing training time, and utilizing Neural Architecture Search tools. The final model outperformed our expectations, achieving over 95% precision and a weighted F-score of more than 90% on our test data set. This paper discusses the development, challenges, and successful outcomes of this significant advancement in the application of AI and ML to materials discovery.

UTILE-Gen: Automated Image Analysis in Nanoscience Using Synthetic Dataset Generator and Deep Learning

This work presents the development and implementation of a deep learning-based workflow for autonomous image analysis in nanoscience. A versatile, agnostic, and configurable tool was developed to generate instance-segmented imaging datasets of nanoparticles. The synthetic generator tool employs domain randomization to expand the image/mask pairs dataset for training supervised deep learning models. The approach eliminates tedious manual annotation and allows training of high-performance models for microscopy image analysis based on convolutional neural networks. We demonstrate how the expanded training set can significantly improve the performance of the classification and instance segmentation models for a variety of nanoparticle shapes, ranging from spherical-, cubic-, to rod-shaped nanoparticles. Finally, the trained models were deployed in a cloud-based analytics platform for the autonomous particle analysis of microscopy images.

Image histogram decomposition method for particle sizing - A numerical simulation study

Scanning probe microscopy is a useful tool in nanoscience. The effective application of nanotechnologies in various fields requires a knowledge of the characteristic attributes of nanoparticles such as shape, dimensions and statistical distribution, and a wide spectrum of experimental and theoretical methods based on various principles have been developed to determine these characteristics. Image histograms offer a global overview of the characteristics of an image. Their shape can encode specific statistical properties of displayed objects such as the distribution function in the case of similar and scalable objects. The model of height histogram presented here proposes a method which solves the long-term problem of processing images of extremely dense particle distributions. The method is based on the principle of the superposition of histograms of individual particles whose topographic surface is described by a parametric model. The resulting height histogram is defined by a convolution of the model of the particle histogram with the distribution function of particle size, with this construction forming the basis of the regression model. The parameters of the distribution function can be obtained via the optimization of the model. The method has been tested on artificially generated configurations of particles of various shapes and size distributions. Each of these configurations creates a topographic surface which is transformed into an image, and the heights obtained from the image allow a histogram to be calculated. Firstly, various configurations of particles are simulated without the presence of any disruptive influences. Next, several experimental effects are evaluated separately (for example, the background, particle shape irregularity and particle overlap). The decomposition of the histogram by the regression model on artificially generated images shows the robustness of the method with respect to particle density, partial horizontal overlap, randomly generated backgrounds and random fluctuations in particle shape. However, the method is sensitive to uniform changes in particle shape, a factor which limits its use to particles with known parametric models of their shape which allow the means of their parameters to be estimated.

Understanding the Influence of Receptive Field and Network Complexity in Neural Network-Guided TEM Image Analysis

Trained neural networks are promising tools to analyze the ever-increasing amount of scientific image data, but it is unclear how to best customize these networks for the unique features in transmission electron micrographs. Here, we systematically examine how neural network architecture choices affect how neural networks segment, or pixel-wise separate, crystalline nanoparticles from amorphous background in transmission electron microscopy (TEM) images. We focus on decoupling the influence of receptive field, or the area of the input image that contributes to the output decision, from network complexity, which dictates the number of trainable parameters. For low-resolution TEM images which rely on amplitude contrast to distinguish nanoparticles from background, we find that the receptive field does not significantly influence segmentation performance. On the other hand, for high-resolution TEM images which rely on both amplitude and phase-contrast changes to identify nanoparticles, receptive field is an important parameter for increased performance, especially in images with minimal amplitude contrast. Rather than depending on atom or nanoparticle size, the ideal receptive field seems to be inversely correlated to the degree of nanoparticle contrast in the image. Our results provide insight and guidance as to how to adapt neural networks for applications with TEM datasets.

A deep learning-based framework for automatic analysis of the nanoparticle morphology in SEM/TEM images

Scanning electron microscopy (SEM) and transmission electron microscopy (TEM) are important tools for characterizing nanomaterial morphology. Automatic analysis of the nanomaterial morphology in SEM/TEM images plays a crucial role in accelerating research on nanomaterials science. However, achieving a high-throughput automated online statistical analysis of the nanomaterial morphology in various complex SEM/TEM images is still a challenging task. In this paper, we propose a universal framework based on deep learning to perform a fast and accurate online statistical analysis of the nanoparticle morphology in complex SEM/TEM images. The proposed framework consists of three stages that are nanoparticle segmentation using a powerful light-weight deep learning network (NSNet), nanoparticle shape extraction, and statistical analysis. The experimental results show that NSNet in the proposed framework has achieved an accuracy of 86.2% and can process 11 SEM/TEM images per second on an embedded processor. Compared with other semantic segmentation models, NSNet is an optimal choice to ensure that the proposed framework still achieves accurate and fast segmentation even in SEM/TEM images with high background interference, extremely small nanoparticles and dense nanoparticles. Meanwhile, the equivalent diameter and Blaschke shape coefficient of the nanoparticle obtained by the proposed framework are 17.14 ± 5.9 and 0.18 ± 0.04, which are well consistent with those of manual statistical analysis. In short, the proposed framework has a promising future in driving the development of automatic and intelligent analysis technology for nanomaterial morphology.

Uncertainty quantification: Can we trust artificial intelligence in drug discovery?

The problem of human trust is one of the most fundamental problems in applied artificial intelligence in drug discovery. In silico models have been widely used to accelerate the process of drug discovery in recent years. However, most of these models can only give reliable predictions within a limited chemical space that the training set covers (applicability domain). Predictions of samples falling outside the applicability domain are unreliable and sometimes dangerous for the drug-design decision-making process. Uncertainty quantification accordingly has drawn great attention to enable autonomous drug designing. By quantifying the confidence level of model predictions, the reliability of the predictions can be quantitatively represented to assist researchers in their molecular reasoning and experimental design. Here we summarize the state-of-the-art approaches to uncertainty quantification and underline how they can be used for drug design and discovery projects. Furthermore, we also outline four representative application scenarios of uncertainty quantification in drug discovery.

Automatic quantification and classification of microplastics in scanning electron micrographs via deep learning

Microplastics quantification and classification are demanding jobs to monitor microplastic pollution and evaluate the potential health risks. In this paper, microplastics from daily supplies in diverse chemical compositions and shapes are imaged by scanning electron microscopy. It offers a greater depth and finer details of microplastics at a wider range of magnification than visible light microscopy or a digital camera, and permits further chemical composition analysis. However, it is labour-intensive to manually extract microplastics from micrographs, especially for small particles and thin fibres. A deep learning approach facilitates microplastics quantification and classification with a manually annotated dataset including 237 micrographs of microplastic particles (fragments or beads) in the range of 50 μm-1 mm and fibres with diameters around 10 μm. For microplastics quantification, two deep learning models (U-Net and MultiResUNet) were implemented for semantic segmentation. Both significantly outmatched conventional computer vision techniques and achieved a high average Jaccard index over 0.75. Especially, U-Net was combined with object-aware pixel embedding to perform instance segmentation on densely packed and tangled fibres for further quantification. For shape classification, a fine-tuned VGG16 neural network classifies microplastics based on their shapes with high accuracy of 98.33%. With trained models, it takes only seconds to segment and classify a new micrograph in high accuracy, which is remarkably cheaper and faster than manual labour. The growing datasets may benefit the identification and quantification of microplastics in environmental samples in future work.

Calculating small-angle scattering intensity functions from electron-microscopy images

We outline procedures to calculate small-angle scattering (SAS) intensity functions from 2-dimensional electron-microscopy (EM) images. Two types of scattering systems were considered: (a) the sample is a set of particles confined to a plane; or (b) the sample is modelled as parallel, infinitely long cylinders that extend into the image plane. In each case, an EM image is segmented into particle instances and the background, whereby coordinates and morphological parameters are computed and used to calculate the constituents of the SAS-intensity function. We compare our results with experimental SAS data, discuss limitations, both general and case specific, and outline some applications of this method which could potentially complement experimental SAS.

We outline procedures to calculate small-angle scattering (SAS) intensity functions from 2-dimensional electron-microscopy (EM) images for two types of scattering systems.

Deep learning for the automation of particle analysis in catalyst layers for polymer electrolyte fuel cells

The rapidly growing use of imaging infrastructure in the energy materials domain drives significant data accumulation in terms of their amount and complexity. The applications of routine techniques for image processing in materials research are often ad hoc, indiscriminate, and empirical, which renders the crucial task of obtaining reliable metrics for quantifications obscure. Moreover, these techniques are expensive, slow, and often involve several preprocessing steps. This paper presents a novel deep learning-based approach for the high-throughput analysis of the particle size distributions from transmission electron microscopy (TEM) images of carbon-supported catalysts for polymer electrolyte fuel cells. A dataset of 40 high-resolution TEM images at different magnification levels, from 10 to 100 nm scales, was annotated manually. This dataset was used to train the U-Net model, with the StarDist formulation for the loss function, for the nanoparticle segmentation task. StarDist reached a precision of 86%, recall of 85%, and an F1-score of 85% by training on datasets as small as thirty images. The segmentation maps outperform models reported in the literature for a similar problem, and the results on particle size analyses agree well with manual particle size measurements, albeit at a significantly lower cost.