Nanoparticle Recognition on Scanning Probe Microscopy Images Using Computer Vision and Deep Learning

A Data Augmentation Methodology to Reduce the Class Imbalance in Histopathology Images

Deep learning techniques have recently yielded remarkable results across various fields. However, the quality of these results depends heavily on the quality and quantity of data used during the training phase. One common issue in multi-class and multi-label classification is class imbalance, where one or several classes make up a substantial portion of the total instances. This imbalance causes the neural network to prioritize features of the majority classes during training, as their detection leads to higher scores. In the context of object detection, two types of imbalance can be identified: (1) an imbalance between the space occupied by the foreground and background and (2) an imbalance in the number of instances for each class. This paper aims to address the second type of imbalance without exacerbating the first. To achieve this, we propose a modification of the copy-paste data augmentation technique, combined with weight-balancing methods in the loss function. This strategy was specifically tailored to improve the performance in datasets with a high instance density, where instance overlap could be detrimental. To validate our methodology, we applied it to a highly unbalanced dataset focused on nuclei detection. The results show that this hybrid approach improves the classification of minority classes without significantly compromising the performance of majority classes.

DLgram cloud service for deep-learning analysis of microscopy images

To analyze images in various fields of science and technology, it is often necessary to count observed objects and determine their parameters. This can be quite labor-intensive and time-consuming. This article presents DLgram, a universal, user-friendly cloud service that is developed for this purpose. It is based on deep learning technologies and does not require programming skills. The user labels several objects in the image and uploads it to the cloud where the neural network is trained to recognize the objects being studied. The user receives recognition results, which if necessary, can be corrected, errors removed, or missing objects added. In addition, it is possible to carry out mathematical processing of the data obtained to get information about the sizes, areas, and coordinates of the observed objects. The article describes the service features and discusses examples of its application. The DLgram service allows to reduce significantly the time spent on quantitative image analysis, reduce subjective factor influence, and increase the accuracy of analysis. RESEARCH HIGHLIGHTS: DLgram automatically recognizes and counts the number of objects in images and their parameters. DLgram is a universal service, which was created on the basis of the latest deep learning developments and does not require programming skills.

Label-free quantification of gold nanoparticles at the single-cell level using a multi-column convolutional neural network (MC-CNN)

Gold nanoparticles (AuNPs) are extensively used in cellular imaging, single-particle tracking, disease diagnosis, studying membrane protein interaction, and drug delivery. Understanding the dynamics of AuNP uptake in live cells is crucial for optimizing their efficacy and safety. Traditional manual methods for quantifying AuNP uptake are time-consuming and subjective, limiting their scalability and accuracy. The available fluorescence-based techniques are limited to photobleaching and photoblinking. Optical microscopy techniques are limited by diffraction limits. Electron microscopy-based imaging techniques are destructive and unsuitable for live cell imaging. Furthermore, the resulting images may contain hundreds of particles with varied intensities, blurring, and substantial occlusion, making it difficult to manually quantify AuNP uptake. To overcome this issue and measure AuNP uptake by live cells, we annotated a dataset of dark-field images of 50 nanometer-radius AuNPs at different incubation durations. Then, to count the number of particles present in a cell, we created a customized multi-column convolutional neural network (MC-CNN). The customized MC-CNN outperformed typical particle counting architectures when compared to spectroscopy-based counting. This will allow researchers to gain a better understanding of AuNP behavior and interactions with cells, paving the way for advancements in nanomedicine, drug delivery, and biomedical research. The code for this paper is available at the following link: https://github.com/Namerlight/LabelFree_AuNP_Quantification.

Thermally Induced Surface Structure and Morphology Evolution in Bimetallic Pt-Au/HOPG Nanoparticles as Probed Using XPS and STM

Bimetallic nanoparticles expand the possibilities of catalyst design, providing an extra degree of freedom for tailoring the catalyst structure in comparison to purely monometallic systems. The distribution mode of two metal species defines the structure of surface catalytic sites, and current research efforts are focused on the development of methods for their controlled tuning. In light of this, a comprehensive investigation of the factors which influence the changes in the morphology of bimetallic nanoparticles, including the elemental redistribution, are mandatory for each particular bimetallic system. Here we present the combined XPS/STM study of the surface structure and morphology of bimetallic Pt-Au/HOPG nanoparticles prepared by thermal vacuum deposition and show that thermal annealing up to 350 °C induces the alloying process between the two bulk-immiscible metal components. Increasing the treatment temperature enhances the extent of Pt-Au alloying. However, the sintering of nanoparticles starts to occur above 500 °C. The approach implemented in this work includes the theoretical simulation of XPS signal intensities for a more meticulous analysis of the compositional distribution and can be helpful from a methodological perspective for other XPS/STM studies of bimetallic nanoparticles on planar supports.

Electrochemical Imaging of Interfaces in Energy Storage via Scanning Probe Methods: Techniques, Applications, and Prospects

Developing a deeper understanding of dynamic chemical, electronic, and morphological changes at interfaces is key to solving practical issues in electrochemical energy storage systems (EESSs). To unravel this complexity, an assortment of tools with distinct capabilities and spatiotemporal resolutions have been used to creatively visualize interfacial processes as they occur. This review highlights how electrochemical scanning probe techniques (ESPTs) such as electrochemical atomic force microscopy, scanning electrochemical microscopy, scanning ion conductance microscopy, and scanning electrochemical cell microscopy are uniquely positioned to address these challenges in EESSs. We describe the operating principles of ESPTs, focusing on the inspection of interfacial structure and chemical processes involved in Li-ion batteries and beyond. We discuss current examples, performance limitations, and complementary ESPTs. Finally, we discuss prospects for imaging improvements and deep learning for automation. We foresee that ESPTs will play an enabling role in advancing EESSs as we transition to renewable energies.

Chirality Analysis of Complex Microparticles using Deep Learning on Realistic Sets of Microscopy Images

Nanoscale chirality is an actively growing research field spurred by the giant chiroptical activity, enantioselective biological activity, and asymmetric catalytic activity of chiral nanostructures. Compared to chiral molecules, the handedness of chiral nano- and microstructures can be directly established via electron microscopy, which can be utilized for the automatic analysis of chiral nanostructures and prediction of their properties. However, chirality in complex materials may have multiple geometric forms and scales. Computational identification of chirality from electron microscopy images rather than optical measurements is convenient but is fundamentally challenging, too, because (1) image features differentiating left- and right-handed particles can be ambiguous and (2) three-dimensional structure essential for chirality is 'flattened' into two-dimensional projections. Here, we show that deep learning algorithms can identify twisted bowtie-shaped microparticles with nearly 100% accuracy and classify them as left- and right-handed with as high as 99% accuracy. Importantly, such accuracy was achieved with as few as 30 original electron microscopy images of bowties. Furthermore, after training on bowtie particles with complex nanostructured features, the model can recognize other chiral shapes with different geometries without retraining for their specific chiral geometry with 93% accuracy, indicating the true learning abilities of the employed neural networks. These findings indicate that our algorithm trained on a practically feasible set of experimental data enables automated analysis of microscopy data for the accelerated discovery of chiral particles and their complex systems for multiple applications.

Automated Recognition of Nanoparticles in Electron Microscopy Images of Nanoscale Palladium Catalysts

Automated computational analysis of nanoparticles is the key approach urgently required to achieve further progress in catalysis, the development of new nanoscale materials, and applications. Analysis of nanoscale objects on the surface relies heavily on scanning electron microscopy (SEM) as the experimental analytic method, allowing direct observation of nanoscale structures and morphology. One of the important examples of such objects is palladium on carbon catalysts, allowing access to various chemical reactions in laboratories and industry. SEM images of Pd/C catalysts show a large number of nanoparticles that are usually analyzed manually. Manual analysis of a statistically significant number of nanoparticles is a tedious and highly time-consuming task that is impossible to perform in a reasonable amount of time for practically needed large amounts of samples. This work provides a comprehensive comparison of various computer vision methods for the detection of metal nanoparticles. In addition, multiple new types of data representations were developed, and their applicability in practice was assessed.

Digital Innovation Enabled Nanomaterial Manufacturing; Machine Learning Strategies and Green Perspectives

Machine learning has been an emerging scientific field serving the modern multidisciplinary needs in the Materials Science and Manufacturing sector. The taxonomy and mapping of nanomaterial properties based on data analytics is going to ensure safe and green manufacturing with consciousness raised on effective resource management. The utilization of predictive modelling tools empowered with artificial intelligence (AI) has proposed novel paths in materials discovery and optimization, while it can further stimulate the cutting-edge and data-driven design of a tailored behavioral profile of nanomaterials to serve the special needs of application environments. The previous knowledge of the physics and mathematical representation of material behaviors, as well as the utilization of already generated testing data, received specific attention by scientists. However, the exploration of available information is not always manageable, and machine intelligence can efficiently (computational resources, time) meet this challenge via high-throughput multidimensional search exploration capabilities. Moreover, the modelling of bio-chemical interactions with the environment and living organisms has been demonstrated to connect chemical structure with acute or tolerable effects upon exposure. Thus, in this review, a summary of recent computational developments is provided with the aim to cover excelling research and present challenges towards unbiased, decentralized, and data-driven decision-making, in relation to increased impact in the field of advanced nanomaterials manufacturing and nanoinformatics, and to indicate the steps required to realize rapid, safe, and circular-by-design nanomaterials.

Nanoparticle Detection on SEM Images Using a Neural Network and Semi-Synthetic Training Data

Processing images represents a necessary step in the process of analysing the information gathered about nanoparticles after characteristic material samples have been scanned with electron microscopy, which often requires the use of image processing techniques or general purpose image manipulation software to carry out tasks such as nanoparticle detection and measurement. In recent years, the use of networks has been successfully implemented to detect and classify electron microscopy images as well as the objects within them. In this work, we present four detection models using two versions of the YOLO neural network architectures trained to detect cubical and quasi-spherical particles in SEM images; the training datasets are a mixture of real images and synthetic ones generated by a semi-arbitrary method. The resulting models were capable of detecting nanoparticles in images different than the ones used for training and identifying them in some cases as the close proximity between nanoparticles proved a challenge for the neural networks in most situations.

Particle Recognition on Transmission Electron Microscopy Images Using Computer Vision and Deep Learning for Catalytic Applications

Recognition and measuring particles on microscopy images is an important part of many scientific studies, including catalytic investigations. In this paper, we present the results of the application of deep learning to the automated recognition of nanoparticles deposited on porous supports (heterogeneous catalysts) on images obtained by transmission electron microscopy (TEM). The Cascade Mask-RCNN neural network was used. During the training, two types of objects were labeled on raw TEM images of ‘real’ catalysts: visible particles and overlapping particle projections. The trained neural network recognized nanoparticles in the test dataset with 0.71 precision and 0.72 recall for both classes of objects and 0.84 precision and 0.79 recall for visible particles. The developed model is integrated into the open-access web service ‘ParticlesNN’, which can be used by any researcher in the world. Instead of hours, TEM data processing per one image analysis is reduced to a maximum of a couple of minutes and the divergence of mean particle size determination is approximately 2% compared to manual analysis. The proposed tool encourages accelerating catalytic research and improving the objectivity and accuracy of analysis.

AI-based atomic force microscopy image analysis allows to predict electrochemical impedance spectra of defects in tethered bilayer membranes

Atomic force microscopy (AFM) image analysis of supported bilayers, such as tethered bilayer membranes (tBLMs) can reveal the nature of the membrane damage by pore-forming proteins and predict the electrochemical impedance spectroscopy (EIS) response of such objects. However, automated analysis involving pore detection in such images is often non-trivial and can require AI-based object detection techniques. The specific object-detection algorithm we used to determine the defect coordinates in real AFM images was a convolutional neural network (CNN). Defect coordinates allow to predict the EIS response of tBLMs populated by the pore-forming toxins using finite element analysis (FEA) modeling. We tested if the accuracy of the CNN algorithm affected the EIS spectral features sensitive to defect densities and other physical parameters of tBLMs. We found that the EIS spectra can be predicted sufficiently well, however, systematic errors of characteristic spectral points were observed and need to be taken into account. Importantly, the comparison of predicted EIS curves with experimental ones allowed to estimate important physical parameters of tBLMs such as the specific resistance of submembrane reservoir. This reservoir separates phospholipid bilayer from the solid support. We found that the specific resistance of the reservoir amounts to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$10^{4.25 \pm 0.10}$$\end{document}104.25±0.10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega \cdot cm$$\end{document}Ω·cm which is approximately two orders of a magnitude higher compared to the specific resistance of the buffer bathing tBLMs studied in this work. We hypothesize that such effect may be related in part due to decreased concentration of ionic carriers in the submembrane due to decreased relative dielectric permittivity in this region.

Gold-viral particle identification by deep learning in wide-field photon scattering parametric images

The ability to identify virus particles is important for research and clinical applications. Because of the optical diffraction limit, conventional optical microscopes are generally not suitable for virus particle detection, and higher resolution instruments such as transmission electron microscopy (TEM) and scanning electron microscopy (SEM) are required. In this paper, we propose a new method for identifying virus particles based on polarization parametric indirect microscopic imaging (PIMI) and deep learning techniques. By introducing an abrupt change of refractivity at the virus particle using antibody-conjugated gold nanoparticles (AuNPs), the strength of the photon scattering signal can be magnified. After acquiring the PIMI images, a deep learning method was applied to identify discriminating features and classify the virus particles, using electron microscopy (EM) images as the ground truth. Experimental results confirm that gold-virus particles can be identified in PIMI images with a high level of confidence.

SRPES and STM data for the model bimetallic Pd-In/HOPG catalysts: Effects of mild post-synthesis oxidative treatments

Post-synthesis treatment of bimetallic catalysts in different gas phases resulting in the adsorption-induced segregation is among promising approaches to enhance their activity not compromising selectivity towards a number of low-temperature reactions. Our recently published paper (M.A. Panafidin, A.V. Bukhtiyarov, I.P. Prosvirin, I.A. Chetyrin, A.Yu. Klyushin, A. Knop-Gericke, N.S. Smirnova, P.V. Markov, I.S. Mashkovsky, Y.V. Zubavichus, A.Yu. Stakheev, V.I. Bukhtiyarov, A mild post-synthesis oxidative treatment of Pd-In/HOPG bimetallic catalysts as a tool of their surface structure fine tuning. Appl. Surf. Sci.) reports on Pd-In intermetallic formation regularities and their evolution after storage in air as well as during treatment in oxygen at submillibar pressures. The current paper gives an extended representation of experimental ex situ/in situ synchrotron-based photoelectron spectroscopy (SRPES) and scanning tunnelling microscopy (STM) data used to derive scientific conclusions in the paper quoted above.

Near-Ambient Pressure XPS and MS Study of CO Oxidation over Model Pd-Au/HOPG Catalysts: The Effect of the Metal Ratio

In this study, the dependence of the catalytic activity of highly oriented pyrolytic graphite (HOPG)-supported bimetallic Pd-Au catalysts towards the CO oxidation based on the Pd/Au atomic ratio was investigated. The activities of two model catalysts differing from each other in the initial Pd/Au atomic ratios appeared as distinctly different in terms of their ignition temperatures. More specifically, the PdAu-2 sample with a lower Pd/Au surface ratio (~0.75) was already active at temperatures less than 150 °C, while the PdAu-1 sample with a higher Pd/Au surface ratio (~1.0) became active only at temperatures above 200 °C. NAP XPS revealed that the exposure of the catalysts to a reaction mixture at RT induces the palladium surface segregation accompanied by an enrichment of the near-surface regions of the two-component Pd-Au alloy nanoparticles with Pd due to adsorption of CO on palladium atoms. The segregation extent depends on the initial Pd/Au surface ratio. The difference in activity between these two catalysts is determined by the presence or higher concentration of specific active Pd sites on the surface of bimetallic particles, i.e., by the ensemble effect. Upon cooling the sample down to room temperature, the reverse redistribution of the atomic composition within near-surface regions occurs, which switches the catalyst back into inactive state. This observation strongly suggests that the optimum active sites emerge under reaction conditions exclusively, involving both high temperature and a reactive atmosphere.

Synthetic Image Rendering Solves Annotation Problem in Deep Learning Nanoparticle Segmentation

Nanoparticles occur in various environments as a consequence of man-made processes, which raises concerns about their impact on the environment and human health. To allow for proper risk assessment, a precise and statistically relevant analysis of particle characteristics (such as size, shape, and composition) is required that would greatly benefit from automated image analysis procedures. While deep learning shows impressive results in object detection tasks, its applicability is limited by the amount of representative, experimentally collected and manually annotated training data. Here, an elegant, flexible, and versatile method to bypass this costly and tedious data acquisition process is presented. It shows that using a rendering software allows to generate realistic, synthetic training data to train a state-of-the art deep neural network. Using this approach, a segmentation accuracy can be derived that is comparable to man-made annotations for toxicologically relevant metal-oxide nanoparticle ensembles which were chosen as examples. The presented study paves the way toward the use of deep learning for automated, high-throughput particle detection in a variety of imaging techniques such as in microscopies and spectroscopies, for a wide range of applications, including the detection of micro- and nanoplastic particles in water and tissue samples.

Deep neural network analysis of nanoparticle ordering to identify defects in layered carbon materials

Smoothness/defectiveness of the carbon material surface is a key issue for many applications, spanning from electronics to reinforced materials, adsorbents and catalysis. Several surface defects cannot be observed with conventional analytic techniques, thus requiring the development of a new imaging approach. Here, we evaluate a convenient method for mapping such "hidden" defects on the surface of carbon materials using 1-5 nm metal nanoparticles as markers. A direct relationship between the presence of defects and the ordering of nanoparticles was studied experimentally and modeled using quantum chemistry calculations and Monte Carlo simulations. An automated pipeline for analyzing microscopic images is described: the degree of smoothness of experimental images was determined by a classification neural network, and then the images were searched for specific types of defects using a segmentation neural network. An informative set of features was generated from both networks: high-dimensional embeddings of image patches and statics of defect distribution.

Deep Learning Based Instance Segmentation of Titanium Dioxide Particles in the Form of Agglomerates in Scanning Electron Microscopy

The size characterization of particles present in the form of agglomerates in images measured by scanning electron microscopy (SEM) requires a powerful image segmentation tool in order to properly define the boundaries of each particle. In this work, we propose to use an algorithm from the deep statistical learning community, the Mask-RCNN, coupled with transfer learning to overcome the problem of generalization of the commonly used image processing methods such as watershed or active contour. Indeed, the adjustment of the parameters of these algorithms is almost systematically necessary and slows down the automation of the processing chain. The Mask-RCNN is adapted here to the case study and we present results obtained on titanium dioxide samples (non-spherical particles) with a level of performance evaluated by different metrics such as the DICE coefficient, which reaches an average value of 0.95 on the test images.