MULTEM: A new multislice program to perform accurate and fast electron diffraction and imaging simulations using Graphics Processing Units with CUDA

Pillar data-acquisition strategies for cryo-electron tomography of beam-sensitive biological samples

For cryo-electron tomography (cryo-ET) of beam-sensitive biological specimens, a planar sample geometry is typically used. As the sample is tilted, the effective thickness of the sample along the direction of the electron beam increases and the signal-to-noise ratio concomitantly decreases, limiting the transfer of information at high tilt angles. In addition, the tilt range where data can be collected is limited by a combination of various sample-environment constraints, including the limited space in the objective lens pole piece and the possible use of fixed conductive braids to cool the specimen. Consequently, most tilt series are limited to a maximum of ±70°, leading to the presence of a missing wedge in Fourier space. The acquisition of cryo-ET data without a missing wedge, for example using a cylindrical sample geometry, is hence attractive for volumetric analysis of low-symmetry structures such as organelles or vesicles, lysis events, pore formation or filaments for which the missing information cannot be compensated by averaging techniques. Irrespective of the geometry, electron-beam damage to the specimen is an issue and the first images acquired will transfer more high-resolution information than those acquired last. There is also an inherent trade-off between higher sampling in Fourier space and avoiding beam damage to the sample. Finally, the necessity of using a sufficient electron fluence to align the tilt images means that this fluence needs to be fractionated across a small number of images; therefore, the order of data acquisition is also a factor to consider. Here, an n-helix tilt scheme is described and simulated which uses overlapping and interleaved tilt series to maximize the use of a pillar geometry, allowing the entire pillar volume to be reconstructed as a single unit. Three related tilt schemes are also evaluated that extend the continuous and classic dose-symmetric tilt schemes for cryo-ET to pillar samples to enable the collection of isotropic information across all spatial frequencies. A fourfold dose-symmetric scheme is proposed which provides a practical compromise between uniform information transfer and complexity of data acquisition.

Atomic-level imaging of beam-sensitive COFs and MOFs by low-dose electron microscopy

Electron microscopy, an important technique that allows for the precise determination of structural information with high spatiotemporal resolution, has become indispensable in unravelling the complex relationships between material structure and properties ranging from mesoscale morphology to atomic arrangement. However, beam-sensitive materials, particularly those comprising organic components such as metal-organic frameworks (MOFs) and covalent organic frameworks (COFs), would suffer catastrophic damage from the high energy electrons, hindering the determination of atomic structures. A low-dose approach has arisen as a possible solution to this problem based on the integration of advancements in several aspects: electron optical system, detector, image processing, and specimen preservation. This article summarizes the transmission electron microscopy characterization of MOFs and COFs, including local structures, host-guest interactions, and interfaces at the atomic level. Revolutions in advanced direct electron detectors, algorithms in image acquisition and processing, and emerging methodology for high quality low-dose imaging are also reviewed. Finally, perspectives on the future development of electron microscopy methodology with the support of computer science are presented.

Element specific atom counting for heterogeneous nanostructures: Combining multiple ADF STEM images for simultaneous thickness and composition determination

In this paper, a methodology is presented to count the number of atoms in heterogeneous nanoparticles based on the combination of multiple annular dark field scanning transmission electron microscopy (ADF STEM) images. The different non-overlapping annular detector collection regions are selected based on the principles of optimal statistical experiment design for the atom-counting problem. To count the number of atoms, the total intensities of scattered electrons for each atomic column, the so-called scattering cross-sections, are simultaneously compared with simulated library values for the different detector regions by minimising the squared differences. The performance of the method is evaluated for simulated Ni@Pt and Au@Ag core-shell nanoparticles. Our approach turns out to be a dose efficient alternative for the investigation of beam-sensitive heterogeneous materials as compared to the combination of ADF STEM and energy dispersive X-ray spectroscopy.

Computational models of amorphous ice for accurate simulation of cryo-EM images of biological samples

Simulations of cryo-electron microscopy (cryo-EM) images of biological samples can be used to produce test datasets to support the development of instrumentation, methods, and software, as well as to assess data acquisition and analysis strategies. To be useful, these simulations need to be based on physically realistic models which include large volumes of amorphous ice. The gold standard model for EM image simulation is a physical atom-based ice model produced using molecular dynamics simulations. Although practical for small sample volumes; for simulation of cryo-EM data from large sample volumes, this can be too computationally expensive. We have evaluated a Gaussian Random Field (GRF) ice model which is shown to be more computationally efficient for large sample volumes. The simulated EM images are compared with the gold standard atom-based ice model approach and shown to be directly comparable. Comparison with experimentally acquired data shows the Gaussian random field ice model produces realistic simulations. The software required has been implemented in the Parakeet software package and the underlying atomic models are available online for use by the wider community.

Atom counting from a combination of two ADF STEM images

To understand the structure-property relationship of nanostructures, reliably quantifying parameters, such as the number of atoms along the projection direction, is important. Advanced statistical methodologies have made it possible to count the number of atoms for monotype crystalline nanoparticles from a single ADF STEM image. Recent developments enable one to simultaneously acquire multiple ADF STEM images. Here, we present an extended statistics-based method for atom counting from a combination of multiple statistically independent ADF STEM images reconstructed from non-overlapping annular detector collection regions which improves the accuracy and allows one to retrieve precise atom-counts, especially for images acquired with low electron doses and multiple element structures.

InFluence: An Open-Source Python Package to Model Images Captured with Direct Electron Detectors

The high detection efficiencies of direct electron detectors facilitate the routine collection of low fluence electron micrographs and diffraction patterns. Low dose and low fluence electron microscopy experiments are the only practical way to acquire useful data from beam sensitive pharmaceutical and biological materials. Appropriate modeling of low fluence images acquired using direct electron detectors is, therefore, paramount for quantitative analysis of the experimental images. We have developed a new open-source Python package to accurately model any single layer direct electron detector for low and high fluence imaging conditions, including a means to validate against experimental data through computation of modulation transfer function and detective quantum efficiency.

Real-time simulations of ADF STEM probe position-integrated scattering cross-sections for single element fcc crystals in zone axis orientation using a densely connected neural network

Quantification of annular dark field (ADF) scanning transmission electron microscopy (STEM) images in terms of composition or thickness often relies on probe-position integrated scattering cross sections (PPISCS). In order to compare experimental PPISCS with theoretically predicted ones, expensive simulations are needed for a given specimen, zone axis orientation, and a variety of microscope settings. The computation time of such simulations can be in the order of hours using a single GPU card. ADF STEM simulations can be efficiently parallelized using multiple GPUs, as the calculation of each pixel is independent of other pixels. However, most research groups do not have the necessary hardware, and, in the best-case scenario, the simulation time will only be reduced proportionally to the number of GPUs used. In this manuscript, we use a learning approach and present a densely connected neural network that is able to perform real-time ADF STEM PPISCS predictions as a function of atomic column thickness for most common face-centered cubic (fcc) crystals (i.e., Al, Cu, Pd, Ag, Pt, Au and Pb) along [100] and [111] zone axis orientations, root-mean-square displacements, and microscope parameters. The proposed architecture is parameter efficient and yields accurate predictions for the PPISCS values for a wide range of input parameters that are commonly used for aberration-corrected transmission electron microscopes.

A decade of atom-counting in STEM: From the first results toward reliable 3D atomic models from a single projection

Quantitative structure determination is needed in order to study and understand nanomaterials at the atomic scale. Materials characterisation resulting in precise structural information is a crucial point to understand the structure-property relation of materials. Counting the number of atoms and retrieving the 3D atomic structure of nanoparticles plays an important role here. In this paper, an overview will be given of the atom-counting methodology and its applications over the past decade. The procedure to count the number of atoms will be discussed in detail and it will be shown how the performance of the method can be further improved. Furthermore, advances toward mixed element nanostructures, 3D atomic modelling based on the atom-counting results, and quantifying the nanoparticle dynamics will be highlighted.

Fast generation of calculated ADF-EDX scattering cross-sections under channelling conditions

Advanced materials often consist of multiple elements which are arranged in a complicated structure. Quantitative scanning transmission electron microscopy is useful to determine the composition and thickness of nanostructures at the atomic scale. However, significant difficulties remain to quantify mixed columns by comparing the resulting atomic resolution images and spectroscopy data with multislice simulations where dynamic scattering needs to be taken into account. The combination of the computationally intensive nature of these simulations and the enormous amount of possible mixed column configurations for a given composition indeed severely hamper the quantification process. To overcome these challenges, we here report the development of an incoherent non-linear method for the fast prediction of ADF-EDX scattering cross-sections of mixed columns under channelling conditions. We first explain the origin of the ADF and EDX incoherence from scattering physics suggesting a linear dependence between those two signals in the case of a high-angle ADF detector. Taking EDX as a perfect incoherent reference mode, we quantitatively examine the ADF longitudinal incoherence under different microscope conditions using multislice simulations. Based on incoherent imaging, the atomic lensing model previously developed for ADF is now expanded to EDX, which yields ADF-EDX scattering cross-section predictions in good agreement with multislice simulations for mixed columns in a core-shell nanoparticle and a high entropy alloy. The fast and accurate prediction of ADF-EDX scattering cross-sections opens up new opportunities to explore the wide range of ordering possibilities of heterogeneous materials with multiple elements.

Towards real-time STEM simulations through targeted subsampling strategies

Scanning transmission electron microscopy images can be complex to interpret on the atomic scale as the contrast is sensitive to multiple factors such as sample thickness, composition, defects and aberrations. Simulations are commonly used to validate or interpret real experimental images, but they come at a cost of either long computation times or specialist hardware such as graphics processing units. Recent works in compressive sensing for experimental STEM images have shown that it is possible to significantly reduce the amount of acquired signal and still recover the full image without significant loss of image quality, and therefore it is proposed here that similar methods can be applied to STEM simulations. In this paper, we demonstrate a method that can significantly increase the efficiency of STEM simulations through a targeted sampling strategy, along with a new approach to independently subsample each frozen phonon layer. We show the effectiveness of this method by simulating a SrTiO₃ grain boundary and monolayer 2H-MoS₂ containing a sulphur vacancy using the abTEM software. We also show how this method is not limited to only traditional multislice methods, but also increases the speed of the PRISM simulation method. Furthermore, we discuss the possibility for STEM simulations to seed the acquisition of real data, to potentially lead the way to self-driving (correcting) STEM.

Multislice forward modeling of coherent surface scattering imaging on surface and interfacial structures

To study nanostructures on substrates, surface-sensitive reflection-geometry scattering techniques such as grazing incident small angle X-ray scattering are commonly used to yield an averaged statistical structural information of the surface sample. Grazing incidence geometry can probe the absolute three-dimensional structural morphology of the sample if a highly coherent beam is used. Coherent surface scattering imaging (CSSI) is a powerful yet non-invasive technique similar to coherent X-ray diffractive imaging (CDI) but performed at small angles and grazing-incidence reflection geometry. A challenge with CSSI is that conventional CDI reconstruction techniques cannot be directly applied to CSSI because the Fourier-transform-based forward models cannot reproduce the dynamical scattering phenomenon near the critical angle of total external reflection of the substrate-supported samples. To overcome this challenge, we have developed a multislice forward model which can successfully simulate the dynamical or multi-beam scattering generated from surface structures and the underlying substrate. The forward model is also demonstrated to be able to reconstruct an elongated 3D pattern from a single shot scattering image in the CSSI geometry through fast-performing CUDA-assisted PyTorch optimization with automatic differentiation.

Probing the Boundary between Classical and Quantum Mechanics by Analyzing the Energy Dependence of Single-Electron Scattering Events at the Nanoscale

The relation between the energy-dependent particle and wave descriptions of electron-matter interactions on the nanoscale was analyzed by measuring the delocalization of an evanescent field from energy-filtered amplitude images of sample/vacuum interfaces with a special aberration-corrected electron microscope. The spatial field extension coincided with the energy-dependent self-coherence length of propagating wave packets that obeyed the time-dependent Schrödinger equation, and underwent a Goos-Hänchen shift. The findings support the view that wave packets are created by self-interferences during coherent-inelastic Coulomb interactions with a decoherence phase close to Δφ = 0.5 rad. Due to a strictly reciprocal dependence on energy, the wave packets shrink below atomic dimensions for electron energy losses beyond 1000 eV, and thus appear particle-like. Consequently, our observations inevitably include pulse-like wave propagations that stimulate structural dynamics in nanomaterials at any electron energy loss, which can be exploited to unravel time-dependent structure-function relationships on the nanoscale.

Element Specific Atom Counting at the Atomic Scale by Combining High Angle Annular Dark Field Scanning Transmission Electron Microscopy and Energy Dispersive X-ray Spectroscopy

A new methodology is presented to count the number of atoms in multimetallic nanocrystals by combining energy dispersive X-ray spectroscopy (EDX) and high angle annular dark field scanning transmission electron microscopy (HAADF STEM). For this purpose, the existence of a linear relationship between the incoherent HAADF STEM and EDX images is exploited. Next to the number of atoms for each element in the atomic columns, the method also allows quantification of the error in the obtained number of atoms, which is of importance given the noisy nature of the acquired EDX signals. Using experimental images of an Au@Ag core-shell nanorod, it is demonstrated that 3D structural information can be extracted at the atomic scale. Furthermore, simulated data of an Au@Pt core-shell nanorod show the prospect to characterize heterogeneous nanostructures with adjacent atomic numbers.

Accurate quantification of lattice temperature dynamics from ultrafast electron diffraction of single-crystal films using dynamical scattering simulations

In ultrafast electron diffraction (UED) experiments, accurate retrieval of time-resolved structural parameters, such as atomic coordinates and thermal displacement parameters, requires an accurate scattering model. Unfortunately, kinematical models are often inaccurate even for relativistic electron probes, especially for dense, oriented single crystals where strong channeling and multiple scattering effects are present. This article introduces and demonstrates dynamical scattering models tailored for quantitative analysis of UED experiments performed on single-crystal films. As a case study, we examine ultrafast laser heating of single-crystal gold films. Comparison of kinematical and dynamical models reveals the strong effects of dynamical scattering within nm-scale films and their dependence on sample topography and probe kinetic energy. Applying to UED experiments on an 11 nm thick film using 750 keV electron probe pulses, the dynamical models provide a tenfold improvement over a comparable kinematical model in matching the measured UED patterns. Also, the retrieved lattice temperature rise is in very good agreement with predictions based on previously measured optical constants of gold, whereas fitting the Debye-Waller factor retrieves values that are more than three times lower. Altogether, these results show the importance of a dynamical scattering theory for quantitative analysis of UED and demonstrate models that can be practically applied to single-crystal materials and heterostructures.

SIM-STEM Lab: Incorporating Compressed Sensing Theory for Fast STEM Simulation

Recently it has been shown that precise dose control and an increase in the overall acquisition speed of atomic resolution scanning transmission electron microscope (STEM) images can be achieved by acquiring only a small fraction of the pixels in the image experimentally and then reconstructing the full image using an inpainting algorithm. In this paper, we apply the same inpainting approach (a form of compressed sensing) to simulated, sub-sampled atomic resolution STEM images. We find that it is possible to significantly sub-sample the area that is simulated, the number of g-vectors contributing the image, and the number of frozen phonon configurations contributing to the final image while still producing an acceptable fit to a fully sampled simulation. Here we discuss the parameters that we use and how the resulting simulations can be quantifiably compared to the full simulations. As with any Compressed Sensing methodology, care must be taken to ensure that isolated events are not excluded from the process, but the observed increase in simulation speed provides significant opportunities for real time simulations, image classification and analytics to be performed as a supplement to experiments on a microscope to be developed in the future.

Atomic-scale detection of individual lead clusters confined in Linde Type A zeolites

Structural analysis of metal clusters confined in nanoporous materials is typically performed by X-ray-driven techniques. Although X-ray analysis has proved its strength in the characterization of metal clusters, it provides averaged structural information. Therefore, we here present an alternative workflow for bringing the characterization of confined metal clusters towards the local scale. This workflow is based on the combination of aberration-corrected transmission electron microscopy (TEM), TEM image simulations, and powder X-ray diffraction (XRD) with advanced statistical techniques. In this manner, we were able to characterize the clustering of Pb atoms in Linde Type A (LTA) zeolites with Pb loadings as low as 5 wt%. Moreover, individual Pb clusters could be directly detected. The proposed methodology thus enables a local-scale characterization of confined metal clusters in zeolites. This is important for further elucidation of the connection between the structure and the physicochemical properties of such systems.

ToTEM: A software for fast TEM image simulation

ToTEM, a multislice based image simulation software is developed for transmission electron microscope (TEM). This software implements the following major features: i) capability of assigning three-dimensional potentials of atom into multiple slices and precise introduction of phase shift caused by the sub-pixel atomic position, ii) employing CUDA coding and graphical processing units (GPU) with multi-threading parallel algorithm based on the powerful batch (inverse) fast Fourier transform (FFT), which is especially beneficial for image simulation of scanning transmission electron microscopy (STEM) or (integrated) differential phase contrast (i)DPC, iii) design for efficiently generating large batch of dataset of high resolution transmission electron microscopy (HRTEM) images. Image simulation acceleration for STEM has been verified by simulating a large-scale SrTiO₃ . Additionally, iDPC image of MFI-type zeolites with xylene molecules encapsulated in straight channels demonstrates the advantage of iDPC in detecting light molecules. This article is protected by copyright. All rights reserved.

Photoluminescence Induced by Substitutional Nitrogen in Single-Layer Tungsten Disulfide

The electronic and optical properties of two-dimensional materials can be strongly influenced by defects, some of which can find significant implementations, such as controllable doping, prolonged valley lifetime, and single-photon emissions. In this work, we demonstrate that defects created by remote N₂ plasma exposure in single-layer WS₂ can induce a distinct low-energy photoluminescence (PL) peak at 1.59 eV, which is in sharp contrast to that caused by remote Ar plasma. This PL peak has a critical requirement on the N₂ plasma exposure dose, which is strongest for WS₂ with about 2.0% sulfur deficiencies (including substitutions and vacancies) and vanishes at 5.6% or higher sulfur deficiencies. Both experiments and first-principles calculations suggest that this 1.59 eV PL peak is caused by defects related to the sulfur substitutions by nitrogen, even though low-temperature PL measurements also reveal that not all the sulfur vacancies are remedied by the substitutional nitrogen. The distinct low-energy PL peak suggests that the substitutional nitrogen defect in single-layer WS₂ can potentially serve as an isolated artificial atom for creating single-photon emitters, and its intensity can also be used to monitor the doping concentrations of substitutional nitrogen.

Direct visualisation of the surface atomic active sites of carbon-supported Co3O4 nanocrystals via high-resolution phase restoration

The atomic arrangement of the terminating facets on spinel Co₃O₄ nanocrystals is strongly linked to their catalytic performance. However, the spinel crystal structure offers multiple possible surface terminations depending on the synthesis. Thus, understanding the terminating surface atomic structure is essential in developing high‐performance Co₃O₄ nanocrystals. In this work, we present direct atomic‐scale observation of the surface terminations of Co₃O₄ nanoparticles supported on hollow carbon spheres (HCSs) using exit wavefunction reconstruction from aberration‐corrected transmission electron microscopy focal‐series. The restored high‐resolution phases show distinct resolved oxygen and cobalt atomic columns. The data show that the structure of {100}, {110}, and {111} facets of spinel Co₃O₄ exhibit characteristic active sites for carbon monoxide (CO) adsorption, in agreement with density functional theory calculations. Of these facets, the {100} and {110} surface terminations are better suited for CO adsorption than the {111}. However, the presence of oxygen on the {111} surface termination indicates this facet also plays an essential role in CO adsorption. Our results demonstrate direct evidence of the surface termination atomic structure beyond the assumed stoichiometry of the surface.

ReciPro: free and open-source multipurpose crystallographic software integrating a crystal model database and viewer, diffraction and microscopy simulators, and diffraction data analysis tools

ReciPro is a comprehensive multipurpose crystallographic program equipped with an intuitive graphical user interface (GUI), and it is completely free and open source. This software has a built-in crystal database consisting of over 20 000 crystal models, and the visualization system can seamlessly display a specified crystal model as an attractive three-dimensional graphic. The comprehensive features are not confined to these crystal model databases and viewers. It can smoothly and quantitatively simulate not only single-crystal and/or polycrystalline (powder) diffraction patterns of X-ray, electron and neutron diffraction of a selected crystal model, based on the kinematic scattering theory, but also various electron diffraction patterns and high-resolution transmission electron microscopy (TEM) images, based on the dynamical scattering theory. The features of stereographic projection of crystal planes/axes to explore crystal orientation relationships and the semi-automatic diffraction spot indexing function for experimental diffraction patterns assist diffraction experiments and analyses. These features are linked through a user-friendly GUI, and the results can be synchronously displayed almost in real time. ReciPro will assist a wide range of crystallographers (including beginners) using X-ray, electron and neutron diffraction crystallography and TEM.

Aberration-corrected transmission electron microscopy of a non-graphitizing carbon

Non-graphitizing carbons (NGCs) are an important class of solid carbons which cannot be converted into graphite by high-temperature heat treatment. They include commercially valuable materials such as activated carbon and glassy carbon. These carbons have been intensively studied for decades, but there is still no agreement about their detailed atomic structure, or the reasons for their resistance to graphitization. The first models for graphitizing and NGCs were proposed by Rosalind Franklin in the early 1950s, and while these are broadly correct, they are incomplete. Many alternative models of NGCs have been put forward since Franklin's time, but none has received universal acceptance. Diffraction and spectroscopic techniques can provide important insights into the nature of these carbons, but only direct microscopic imaging can reveal their true atomic structure. Here, we apply aberration-corrected transmission electron microscopy to an activated carbon prepared from waste biomass and present evidence for the presence of pentagonal and heptagonal carbon rings. This provides support for a model of the structure of NGC made up of curved fragments in which non-hexagonal rings are dispersed randomly throughout hexagonal networks.

Parakeet: a digital twin software pipeline to assess the impact of experimental parameters on tomographic reconstructions for cryo-electron tomography

In cryo-electron tomography (cryo-ET) of biological samples, the quality of tomographic reconstructions can vary depending on the transmission electron microscope (TEM) instrument and data acquisition parameters. In this paper, we present Parakeet, a 'digital twin' software pipeline for the assessment of the impact of various TEM experiment parameters on the quality of three-dimensional tomographic reconstructions. The Parakeet digital twin is a digital model that can be used to optimize the performance and utilization of a physical instrument to enable in silico optimization of sample geometries, data acquisition schemes and instrument parameters. The digital twin performs virtual sample generation, TEM image simulation, and tilt series reconstruction and analysis within a convenient software framework. As well as being able to produce physically realistic simulated cryo-ET datasets to aid the development of tomographic reconstruction and subtomogram averaging programs, Parakeet aims to enable convenient assessment of the effects of different microscope parameters and data acquisition parameters on reconstruction quality. To illustrate the use of the software, we present the example of a quantitative analysis of missing wedge artefacts on simulated planar and cylindrical biological samples and discuss how data collection parameters can be modified for cylindrical samples where a full 180° tilt range might be measured.

Prismatic 2.0 - Simulation software for scanning and high resolution transmission electron microscopy (STEM and HRTEM)

Scanning transmission electron microscopy (STEM), where a converged electron probe is scanned over a sample's surface and an imaging, diffraction, or spectroscopic signal is measured as a function of probe position, is an extremely powerful tool for materials characterization. The widespread adoption of hardware aberration correction, direct electron detectors, and computational imaging methods have made STEM one of the most important tools for atomic-resolution materials science. Many of these imaging methods rely on accurate imaging and diffraction simulations in order to interpret experimental results. However, STEM simulations have traditionally required large calculation times, as modeling the electron scattering requires a separate simulation for each of the typically millions of probe positions. We have created the Prismatic simulation code for fast simulation of STEM experiments with support for multi-CPU and multi-GPU (graphics processing unit) systems, using both the conventional multislice and our recently-introduced PRISM method. In this paper, we introduce Prismatic version 2.0, which adds many new algorithmic improvements, an updated graphical user interface (GUI), post-processing of simulation data, and additional operating modes such as plane-wave TEM. We review various aspects of the simulation methods and codes in detail and provide various simulation examples. Prismatic 2.0 is freely available both as an open-source package that can be run using a C++ or Python command line interface, or GUI, as well within a Docker container environment.

An improved method assigning three-dimensional atomic potentials to multiple slices in exit-wave simulations of Transmission Electron Microscopy

An atomic potential should be assigned to several slices in the multislice method owing to its three-dimensional (3D) distribution. Based on a formula including several Gaussian functions to fit electron atomic scattering factors, a simple analytical expression is proposed to calculate the potential of atoms projected onto multiple slices. The potential in 3D distribution is properly projected onto slices that do not contain the atomic centroid. Thus, the fluctuations in atomic altitude influence the assigning of atomic potentials is considered correctly. The projected potential is calculated in a reciprocal space and involves an accurate 3D atomic position. Tests conducted with a plausible Ag chain verify the good performance of this new approach. The simulated exit wave leaving a complex crystal of Ba₆Nd₂Ti₄O₁₇ demonstrates that the proposed simulation approach is better than the traditional multislice method.

A Fast Frozen Phonon Algorithm Using Mixed Static Potentials

Image simulation in electron microscopy is vital for advanced image analysis but can be prohibitively long. This is especially true when including thermal diffuse scattering through the frozen phonon method, requiring repeat simulations for a number of phonon configurations. Here a method of reducing frozen phonon simulation time is demonstrated by emulating random phonon displacements through randomly mixing a set of precalculated static potentials. This avoids excessive recalculation of atom potentials and can lead to significant time improvement. The validity and limitations of this method are also demonstrated with respect to convergent beam electron diffraction and scanning transmission electron microscopy.

The abTEM code: transmission electron microscopy from first principles

Simulation of transmission electron microscopy (TEM) images or diffraction patterns is often required to interpret experimental data. Since nuclear cores dominate electron scattering, the scattering potential is typically described using the independent atom model, which completely neglects valence bonding and its effect on the transmitting electrons. As instrumentation has advanced, new measurements have revealed subtle details of the scattering potential that were previously not accessible to experiment. We have created an open-source simulation code designed to meet these demands by integrating the ability to calculate the potential via density functional theory (DFT) with a flexible modular software design. abTEM can simulate most standard imaging modes and incorporates the latest algorithmic developments. The development of new techniques requires a program that is accessible to domain experts without extensive programming experience. abTEM is written purely in Python and designed for easy modification and extension. The effective use of modern open-source libraries makes the performance of abTEM highly competitive with existing optimized codes on both CPUs and GPUs and allows us to leverage an extensive ecosystem of libraries, such as the Atomic Simulation Environment and the DFT code GPAW. abTEM is designed to work in an interactive Python notebook, creating a seamless and reproducible workflow from defining an atomic structure, calculating molecular dynamics (MD) and electrostatic potentials, to the analysis of results, all in a single, easy-to-read document.  This article provides ongoing documentation of abTEM development. In this first version, we show use cases for hexagonal boron nitride, where valence bonding can be detected, a 4D-STEM simulation of molybdenum disulfide including ptychographic phase reconstruction, a comparison of MD and frozen phonon modeling for convergent-beam electron diffraction of a 2.6-million-atom silicon system, and a performance comparison of our fast implementation of the PRISM algorithm for a decahedral 20000-atom gold nanoparticle.

The abTEM code: transmission electron microscopy from first principles

Simulation of transmission electron microscopy (TEM) images or diffraction patterns is often required to interpret experimental data. Since nuclear cores dominate electron scattering, the scattering potential is typically described using the independent atom model, which completely neglects valence bonding and its effect on the transmitting electrons. As instrumentation has advanced, new measurements have revealed subtle details of the scattering potential that were previously not accessible to experiment. We have created an open-source simulation code designed to meet these demands by integrating the ability to calculate the potential via density functional theory (DFT) with a flexible modular software design. abTEM can simulate most standard imaging modes and incorporates the latest algorithmic developments. The development of new techniques requires a program that is accessible to domain experts without extensive programming experience. abTEM is written purely in Python and designed for easy modification and extension. The effective use of modern open-source libraries makes the performance of abTEM highly competitive with existing optimized codes on both CPUs and GPUs and allows us to leverage an extensive ecosystem of libraries, such as the Atomic Simulation Environment and the DFT code GPAW. abTEM is designed to work in an interactive Python notebook, creating a seamless and reproducible workflow from defining an atomic structure, calculating molecular dynamics (MD) and electrostatic potentials, to the analysis of results, all in a single, easy-to-read document.  This article provides ongoing documentation of abTEM development. In this first version, we show use cases for hexagonal boron nitride, where valence bonding can be detected, a 4D-STEM simulation of molybdenum disulfide including ptychographic phase reconstruction, a comparison of MD and frozen phonon modeling for convergent-beam electron diffraction of a 2.6-million-atom silicon system, and a performance comparison of our fast implementation of the PRISM algorithm for a decahedral 20000-atom gold nanoparticle.

Deep learning in electron microscopy

Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy.

ab initio description of bonding for transmission electron microscopy

The simulation of transmission electron microscopy (TEM) images or diffraction patterns is often required to interpret their contrast and extract specimen features. This is especially true for high-resolution phase-contrast imaging of materials, but electron scattering simulations based on atomistic models are widely used in materials science and structural biology. Since electron scattering is dominated by the nuclear cores, the scattering potential is typically described by the widely applied independent atom model. This approximation is fast and fairly accurate, especially for scanning TEM (STEM) annular dark-field contrast, but it completely neglects valence bonding and its effect on the transmitting electrons. However, an emerging trend in electron microscopy is to use new instrumentation and methods to extract the maximum amount of information from each electron. This is evident in the increasing popularity of techniques such as 4D-STEM combined with ptychography in materials science, and cryogenic microcrystal electron diffraction in structural biology, where subtle differences in the scattering potential may be both measurable and contain additional insights. Thus, there is increasing interest in electron scattering simulations based on electrostatic potentials obtained from first principles, mainly via density functional theory, which was previously mainly required for holography. In this Review, we discuss the motivation and basis for these developments, survey the pioneering work that has been published thus far, and give our outlook for the future. We argue that a physically better justified ab initio description of the scattering potential is both useful and viable for an increasing number of systems, and we expect such simulations to steadily gain in popularity and importance.

Three-dimensional atomic structure of supported Au nanoparticles at high temperature

Au nanoparticles (NPs) deposited on CeO2 are extensively used as thermal catalysts since the morphology of the NPs is expected to be stable at elevated temperatures. Although it is well known that the activity of Au NPs depends on their size and surface structure, their three-dimensional (3D) structure at the atomic scale has not been completely characterized as a function of temperature. In this paper, we overcome the limitations of conventional electron tomography by combining atom counting applied to aberration-corrected scanning transmission electron microscopy images and molecular dynamics relaxation. In this manner, we are able to perform an atomic resolution 3D investigation of supported Au NPs. Our results enable us to characterize the 3D equilibrium structure of single NPs as a function of temperature. Moreover, the dynamic 3D structural evolution of the NPs at high temperatures, including surface layer jumping and crystalline transformations, has been studied.

Angle-resolved STEM using an iris aperture: Scattering contributions and sources of error for the quantitative analysis in Si

The angle-resolved electron scattering is investigated in scanning-transmission electron microscopy (STEM) using a motorised iris aperture placed above a conventional annular detector. The electron intensity scattered into various angle ranges is compared with simulations that were carried out in the frozen-lattice approximation. As figure of merit for the agreement of experiment and simulation we evaluate the specimen thickness which is compared with the thickness obtained from position-averaged convergent beam electron diffraction (PACBED). We find deviations whose strengths depend on the angular range of the detected electrons. As possible sources of error we investigate, for example, the influences of amorphous surface layers, inelastic scattering (plasmon excitation), phonon-correlation within the frozen-lattice approach, and distortions in the diffraction plane of the microscope. The evaluation is performed for four experimental thicknesses and two angle-resolved STEM series under different camera lengths. The results clearly show that especially for scattering angles below 50 mrad, it is mandatory that the simulations take scattering effects into account which are usually neglected for simulating high-angle scattering. Most influences predominantly affect the low-angle range, but also high scattering angles can be affected (e.g. by amorphous surface covering).

Contrast transfer and noise considerations in focused-probe electron ptychography

Electron ptychography is a 4-D STEM phase-contrast imaging technique with applications to light-element and beam-sensitive materials. Although the electron dose (electrons incident per unit area on the sample) is the primary figure of merit for imaging beam-sensitive materials, it is also necessary to consider the contrast transfer properties of the imaging technique. Here, we explore the contrast transfer properties of electron ptychography. The contrast transfer of focused-probe, non-iterative electron ptychography using the single-side-band (SSB) method is demonstrated experimentally. The band-pass nature of the phase-contrast transfer function (PCTF) for SSB ptychography places strict limitations on the probe convergence semi-angles required to resolve specific sample features with high contrast. The PCTF of the extended ptychographic iterative engine (ePIE) is broader than that for SSB ptychography, although when both high and low spatial frequencies are transferred, band-pass filtering is required to remove image artefacts. Normalisation of the transfer function with respect to the noise level shows that the transfer window is increased while avoiding noise amplification. Avoiding algorithms containing deconvolution steps may also increase the dose-efficiency of ptychographic phase reconstructions.

Strain effects in core-shell PtCo nanoparticles: a comparison of experimental observations and computational modelling

Strain in Pt nanoalloys induced by the secondary metal has long been suggested as a major contributor to the modification of catalytic properties. Here, we investigate strain in PtCo nanoparticles using a combination of computational modelling and microscopy experiments. We have used a combination of molecular dynamics (MD) and large-scale density functional theory (DFT) for our models, alongside experimental work using annular dark field scanning transmission electron microscopy (ADF-STEM). We have performed extensive validation of the interatomic potential against DFT using a Pt₅₆₈Co₁₈ nanoparticle. Modelling gives access to 3 dimensional structures that can be compared to the 2D ADF-STEM images, which we use to build an understanding of nanoparticle structure and composition. Strain has been measured for PtCo and pure Pt nanoparticles, with MD annealed models compared to ADF-STEM images. Our analysis was performed on a layer by layer basis, where distinct trends between the Pt and PtCo alloy nanoparticles are observed. To our knowledge, we show for the first time a way in which detailed atomistic simulations can be used to augment and help interpret the results of ADF-STEM strain mapping experiments, which will enhance their use in characterisation towards the development of improved catalysts.

Scanning transmission electron microscopy image simulations of complex dislocation structures generated by discrete dislocation dynamics

Scanning Transmission Electron Microscopy Diffraction Contrast Imaging (STEM-DCI) has been gaining popularity for the identification and analysis of dislocations in crystalline materials due to its ability to supress undesirable image features that are often present in conventional TEM images. However, there does not yet exist a robust body of work demonstrating expected contrast in these imaging conditions. A novel approach for the simulation of STEM-DCI images was developed using a modified form of the scattering matrix formalism. This algorithm was used to simulate a variety of dislocation configurations generated using three-dimensional discrete dislocation dynamics.

Atom column detection from simultaneously acquired ABF and ADF STEM images

In electron microscopy, the maximum a posteriori (MAP) probability rule has been introduced as a tool to determine the most probable atomic structure from high-resolution annular dark-field (ADF) scanning transmission electron microscopy (STEM) images exhibiting low contrast-to-noise ratio (CNR). Besides ADF imaging, STEM can also be applied in the annular bright-field (ABF) regime. The ABF STEM mode allows to directly visualize light-element atomic columns in the presence of heavy columns. Typically, light-element nanomaterials are sensitive to the electron beam, limiting the incoming electron dose in order to avoid beam damage and leading to images exhibiting low CNR. Therefore, it is of interest to apply the MAP probability rule not only to ADF STEM images, but to ABF STEM images as well. In this work, the methodology of the MAP rule, which combines statistical parameter estimation theory and model-order selection, is extended to be applied to simultaneously acquired ABF and ADF STEM images. For this, an extension of the commonly used parametric models in STEM is proposed. Hereby, the effect of specimen tilt has been taken into account, since small tilts from the crystal zone axis affect, especially, ABF STEM intensities. Using simulations as well as experimental data, it is shown that the proposed methodology can be successfully used to detect light elements in the presence of heavy elements.

Coherent Lattice Wobbling and Out-of-Phase Intensity Oscillations of Friedel Pairs Observed by Ultrafast Electron Diffraction

The inspection of Friedel's law in ultrafast electron diffraction (UED) is important to gain a comprehensive understanding of material atomic structure and its dynamic response. Here, monoclinic gallium telluride (GaTe), as a low-symmetry, layered crystal in contrast to many other 2D materials, is investigated by mega-electronvolt UED. Strong out-of-phase oscillations of Bragg peak intensities are observed for Friedel pairs, which does not obey Friedel's law. As evidenced by the preserved mirror symmetry and supported by both kinematic and dynamic scattering simulations, the intensity oscillations are provoked by the lowest-order longitudinal acoustic breathing phonon. Our results provide a generalized understanding of Friedel's law in UED and demonstrate that by designed misalignment of surface normal and primitive lattice vectors, coherent lattice wobbling and effective shear strain can be generated in crystal films by laser pulse excitation, which is otherwise hard to achieve and can be further utilized to dynamically tune and switch material properties.

Quantifying oxygen distortions in lithium-rich transition-metal-oxide cathodes using ABF STEM

Lithium-rich cathodes can store excess charge beyond the transition metal redox capacity by participation of oxygen in reversible anionic redox reactions. Although these processes are crucial for achieving high energy densities, their structural origins are not yet fully understood. Here, we explore the use of annular bright-field (ABF) imaging in scanning transmission electron microscopy (STEM) to measure oxygen distortions in charged Li_1.2Ni_0.2Mn_0.6O₂. We show that ABF STEM data can provide positional accuracies below 20 pm but this is restricted to cases where no specimen mistilt is present, and only for a range of thicknesses above 3.5 nm. The reliability of these measurements is compromised even when the experimental and post-processing designs are optimised for accuracy and precision, indicating that extreme care must be taken when attempting to quantify distortions in these materials.

The maximum a posteriori probability rule for atom column detection from HAADF STEM images

Recently, the maximum a posteriori (MAP) probability rule has been proposed as an objective and quantitative method to detect atom columns and even single atoms from high-resolution high-angle annular dark-field (HAADF) scanning transmission electron microscopy (STEM) images. The method combines statistical parameter estimation and model-order selection using a Bayesian framework and has been shown to be especially useful for the analysis of the structure of beam-sensitive nanomaterials. In order to avoid beam damage, images of such materials are usually acquired using a limited incoming electron dose resulting in a low contrast-to-noise ratio (CNR) which makes visual inspection unreliable. This creates a need for an objective and quantitative approach. The present paper describes the methodology of the MAP probability rule, gives its step-by-step derivation and discusses its algorithmic implementation for atom column detection. In addition, simulation results are presented showing that the performance of the MAP probability rule to detect the correct number of atomic columns from HAADF STEM images is superior to that of other model-order selection criteria, including the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Moreover, the MAP probability rule is used as a tool to evaluate the relation between STEM image quality measures and atom detectability resulting in the introduction of the so-called integrated CNR (ICNR) as a new image quality measure that better correlates with atom detectability than conventional measures such as signal-to-noise ratio (SNR) and CNR.

Advances in image processing for single-particle analysis by electron cryomicroscopy and challenges ahead

Electron cryomicroscopy (cryoEM) is essential for the study and functional understanding of non-crystalline macromolecules such as proteins. These molecules cannot be imaged using X-ray crystallography or other popular methods. CryoEM has been successfully used to visualize macromolecular complexes such as ribosomes, viruses, and ion channels. Determination of structural models of these at various conformational states leads to insight on how these molecules function. Recent advances in imaging technology have given cryoEM a scientific rebirth. As a result of these technological advances image processing and analysis have yielded molecular structures at atomic resolution. Nevertheless there continue to be challenges in image processing, and in this article we will touch on the most essential in order to derive an accurate three-dimensional model from noisy projection images. Traditional approaches, such as k-means clustering for class averaging, will be provided as background. We will then highlight new approaches for each image processing subproblem, including a 3D reconstruction method for asymmetric molecules using just two projection images and deep learning algorithms for automated particle picking.

Single Atom Detection from Low Contrast-to-Noise Ratio Electron Microscopy Images

Single atom detection is of key importance to solving a wide range of scientific and technological problems. The strong interaction of electrons with matter makes transmission electron microscopy one of the most promising techniques. In particular, aberration correction using scanning transmission electron microscopy has made a significant step forward toward detecting single atoms. However, to overcome radiation damage, related to the use of high-energy electrons, the incoming electron dose should be kept low enough. This results in images exhibiting a low signal-to-noise ratio and extremely weak contrast, especially for light-element nanomaterials. To overcome this problem, a combination of physics-based model fitting and the use of a model-order selection method is proposed, enabling one to detect single atoms with high reliability.

STEMcl-A multi-GPU multislice algorithm for simulation of large structure and imaging parameter series

Electron microscopy images are interference patterns and can generally not be interpreted in a straight forward manner. Typically, time consuming numerical simulations have to be employed to separate specimen features from imaging artifacts. Directly comparing numerical predictions to experimental results, realistic simulation box sizes and varying imaging parameters are needed. In this work, we introduce an accelerated multislice algorithm, named STEMcl, that is capable of simulating series of large super cells typical for defective and amorphous systems, in addition to parameter series using the massive parallelization accessible in today's commercial PC-hardware, e.g. graphics processing units (GPUs). A new numerical approach is used to overcome the memory constraint limiting the maximum computable system size. This approach creates the possibility to study systematically the contrast formation arising by structural differences. STEM simulations of structure series of a crystalline Si and an amorphous CuZr system are presented and the contrast formation of vacancies/voids are studied. The detectability of vacancies/voids in STEM experiments is discussed in terms of density changes.