The potential for Bayesian compressive sensing to significantly reduce electron dose in high-resolution STEM images

Atomic-level imaging of beam-sensitive COFs and MOFs by low-dose electron microscopy

Electron microscopy, an important technique that allows for the precise determination of structural information with high spatiotemporal resolution, has become indispensable in unravelling the complex relationships between material structure and properties ranging from mesoscale morphology to atomic arrangement. However, beam-sensitive materials, particularly those comprising organic components such as metal-organic frameworks (MOFs) and covalent organic frameworks (COFs), would suffer catastrophic damage from the high energy electrons, hindering the determination of atomic structures. A low-dose approach has arisen as a possible solution to this problem based on the integration of advancements in several aspects: electron optical system, detector, image processing, and specimen preservation. This article summarizes the transmission electron microscopy characterization of MOFs and COFs, including local structures, host-guest interactions, and interfaces at the atomic level. Revolutions in advanced direct electron detectors, algorithms in image acquisition and processing, and emerging methodology for high quality low-dose imaging are also reviewed. Finally, perspectives on the future development of electron microscopy methodology with the support of computer science are presented.

High-speed 4-dimensional scanning transmission electron microscopy using compressive sensing techniques

Here we show that compressive sensing allows 4-dimensional (4-D) STEM data to be obtained and accurately reconstructed with both high-speed and reduced electron fluence. The methodology needed to achieve these results compared to conventional 4-D approaches requires only that a random subset of probe locations is acquired from the typical regular scanning grid, which immediately generates both higher speed and the lower fluence experimentally. We also consider downsampling of the detector, showing that oversampling is inherent within convergent beam electron diffraction (CBED) patterns and that detector downsampling does not reduce precision but allows faster experimental data acquisition. Analysis of an experimental atomic resolution yttrium silicide dataset shows that it is possible to recover over 25 dB peak signal-to-noise ratio in the recovered phase using 0.3% of the total data. Lay abstract: Four-dimensional scanning transmission electron microscopy (4-D STEM) is a powerful technique for characterizing complex nanoscale structures. In this method, a convergent beam electron diffraction pattern (CBED) is acquired at each probe location during the scan of the sample. This means that a 2-dimensional signal is acquired at each 2-D probe location, equating to a 4-D dataset. Despite the recent development of fast direct electron detectors, some capable of 100kHz frame rates, the limiting factor for 4-D STEM is acquisition times in the majority of cases, where cameras will typically operate on the order of 2kHz. This means that a raster scan containing 256^2 probe locations can take on the order of 30s, approximately 100-1000 times longer than a conventional STEM imaging technique using monolithic radial detectors. As a result, 4-D STEM acquisitions can be subject to adverse effects such as drift, beam damage, and sample contamination. Recent advances in computational imaging techniques for STEM have allowed for faster acquisition speeds by way of acquiring only a random subset of probe locations from the field of view. By doing this, the acquisition time is significantly reduced, in some cases by a factor of 10-100 times. The acquired data is then processed to fill-in or inpaint the missing data, taking advantage of the inherently low-complex signals which can be linearly combined to recover the information. In this work, similar methods are demonstrated for the acquisition of 4-D STEM data, where only a random subset of CBED patterns are acquired over the raster scan. We simulate the compressive sensing acquisition method for 4-D STEM and present our findings for a variety of analysis techniques such as ptychography and differential phase contrast. Our results show that acquisition times can be significantly reduced on the order of 100-300 times, therefore improving existing frame rates, as well as further reducing the electron fluence beyond just using a faster camera.

Framework of compressive sensing and data compression for 4D-STEM

Four-dimensional Scanning Transmission Electron Microscopy (4D-STEM) is a powerful technique for high-resolution and high-precision materials characterization at multiple length scales, including the characterization of beam-sensitive materials. However, the field of view of 4D-STEM is relatively small, which in absence of live processing is limited by the data size required for storage. Furthermore, the rectilinear scan approach currently employed in 4D-STEM places a resolution- and signal-dependent dose limit for the study of beam sensitive materials. Improving 4D-STEM data and dose efficiency, by keeping the data size manageable while limiting the amount of electron dose, is thus critical for broader applications. Here we introduce a general method for reconstructing 4D-STEM data with subsampling in both real and reciprocal spaces at high fidelity. The approach is first tested on the subsampled datasets created from a full 4D-STEM dataset, and then demonstrated experimentally using random scan in real-space. The same reconstruction algorithm can also be used for compression of 4D-STEM datasets, leading to a large reduction (100 times or more) in data size, while retaining the fine features of 4D-STEM imaging, for crystalline samples.

Machine Learning Refinement of In Situ Images Acquired by Low Electron Dose LC-TEM

We have studied a machine learning (ML) technique for refining images acquired during in situ observation using liquid-cell transmission electron microscopy. Our model is constructed using a U-Net architecture and a ResNet encoder. For training our ML model, we prepared an original image dataset that contained pairs of images of samples acquired with and without a solution present. The former images were used as noisy images, and the latter images were used as corresponding ground truth images. The number of pairs of image sets was 1,204, and the image sets included images acquired at several different magnifications and electron doses. The trained model converted a noisy image into a clear image. The time necessary for the conversion was on the order of 10 ms, and we applied the model to in situ observations using the software Gatan DigitalMicrograph (DM). Even if a nanoparticle was not visible in a view window in the DM software because of the low electron dose, it was visible in a successive refined image generated by our ML model.

The Potential of Subsampling and Inpainting for Fast Low-Dose Cryo FIB-SEM Imaging

Traditional image acquisition for cryo focused ion-beam scanning electron microscopy (FIB-SEM) tomography often sees thousands of images being captured over a period of many hours, with immense data sets being produced. When imaging beam sensitive materials, these images are often compromised by additional constraints related to beam damage and the devitrification of the material during imaging, which renders data acquisition both costly and unreliable. Subsampling and inpainting are proposed as solutions for both of these aspects, allowing fast and low-dose imaging to take place in the Focused ion-beam scanning electron microscopy FIB-SEM without an appreciable loss in image quality. In this work, experimental data are presented which validate subsampling and inpainting as a useful tool for convenient and reliable data acquisition in a FIB-SEM, with new methods of handling three-dimensional data being employed in the context of dictionary learning and inpainting algorithms using a newly developed microscope control software and data recovery algorithm.

Enhancing performance of electron holography with mathematical and machine learning-based denoising techniques

Electron holography is a useful tool for analyzing functional properties, such as electromagnetic fields and strains of materials and devices. The performance of electron holography is limited by the 'shot noise' inherent in electron micrographs (holograms), which are composed of a finite number of electrons. A promising approach for addressing this issue is to use mathematical and machine learning-based image-processing techniques for hologram denoising. With the advancement of information science, denoising methods have become capable of extracting signals that are completely buried in noise, and they are being applied to electron microscopy, including electron holography. However, these advanced denoising methods are complex and have many parameters to be tuned; therefore, it is necessary to understand their principles in depth and use them carefully. Herein, we present an overview of the principles and usage of sparse coding, the wavelet hidden Markov model and tensor decomposition, which have been applied to electron holography. We also present evaluation results for the denoising performance of these methods obtained through their application to simulated and experimentally recorded holograms. Our analysis, review and comparison of the methods clarify the impact of denoising on electron holography research.

Interlacing in Atomic Resolution Scanning Transmission Electron Microscopy

Fast frame rates are desirable in scanning transmission electron microscopy for a number of reasons: controlling electron beam dose, capturing in situ events, or reducing the appearance of scan distortions. While several strategies exist for increasing frame rates, many impact image quality or require investment in advanced scan hardware. Here, we present an interlaced imaging approach to achieve minimal loss of image quality with faster frame rates that can be implemented on many existing scan controllers. We further demonstrate that our interlacing approach provides the best possible strain precision for a given electron dose compared with other contemporary approaches.

Pulsed-beam transmission electron microscopy and radiation damage

We review the use of pulsed electron-beams in transmission electron microscopes (TEMs) for the purpose of mitigating specimen damage. We begin by placing the importance of TEMs with respect to materials characterization into proper context, and we provide a brief overview of established methods for reducing or eliminating the deleterious effects of beam-induced damage. We then introduce the concept of pulsed-beam TEM, and we briefly describe the basic methods and instrument configurations used to create so-called temporally structured electron beams. Following a brief overview of the use of high-dose-rate pulsed-electron beams in cancer radiation therapy, we review historical speculations and more recent compelling but mostly anecdotal findings of a pulsed-beam TEM damage effect. This is followed by an in-depth technical review of recent works seeking to establish cause-and-effect relationships, to conclusively uncover the presence of an effect, and to explore the practicality of the approach. These studies, in particular, provide the most compelling evidence to date that using a pulsed electron beam in the TEM is indeed a viable way to mitigate damage. Throughout, we point out current gaps in understanding, and we conclude with a brief perspective of current needs and future directions.

In Situ and Emerging Transmission Electron Microscopy for Catalysis Research

Catalysts are the primary facilitator in many dynamic processes. Therefore, a thorough understanding of these processes has vast implications for a myriad of energy systems. The scanning/transmission electron microscope (S/TEM) is a powerful tool not only for atomic-scale characterization but also in situ catalytic experimentation. Techniques such as liquid and gas phase electron microscopy allow the observation of catalysts in an environment conducive to catalytic reactions. Correlated algorithms can greatly improve microscopy data processing and expand multidimensional data handling. Furthermore, new techniques including 4D-STEM, atomic electron tomography, cryogenic electron microscopy, and monochromated electron energy loss spectroscopy (EELS) push the boundaries of our comprehension of catalyst behavior. In this review, we discuss the existing and emergent techniques for observing catalysts using S/TEM. Challenges and opportunities highlighted aim to inspire and accelerate the use of electron microscopy to further investigate the complex interplay of catalytic systems.

Towards real-time STEM simulations through targeted subsampling strategies

Scanning transmission electron microscopy images can be complex to interpret on the atomic scale as the contrast is sensitive to multiple factors such as sample thickness, composition, defects and aberrations. Simulations are commonly used to validate or interpret real experimental images, but they come at a cost of either long computation times or specialist hardware such as graphics processing units. Recent works in compressive sensing for experimental STEM images have shown that it is possible to significantly reduce the amount of acquired signal and still recover the full image without significant loss of image quality, and therefore it is proposed here that similar methods can be applied to STEM simulations. In this paper, we demonstrate a method that can significantly increase the efficiency of STEM simulations through a targeted sampling strategy, along with a new approach to independently subsample each frozen phonon layer. We show the effectiveness of this method by simulating a SrTiO₃ grain boundary and monolayer 2H-MoS₂ containing a sulphur vacancy using the abTEM software. We also show how this method is not limited to only traditional multislice methods, but also increases the speed of the PRISM simulation method. Furthermore, we discuss the possibility for STEM simulations to seed the acquisition of real data, to potentially lead the way to self-driving (correcting) STEM.

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

In the last few years, electron microscopy has experienced a new methodological paradigm aimed to fix the bottlenecks and overcome the challenges of its analytical workflow. Machine learning and artificial intelligence are answering this call providing powerful resources towards automation, exploration, and development. In this review, we evaluate the state-of-the-art of machine learning applied to electron microscopy (and obliquely, to materials and nano-sciences). We start from the traditional imaging techniques to reach the newest higher-dimensionality ones, also covering the recent advances in spectroscopy and tomography. Additionally, the present review provides a practical guide for microscopists, and in general for material scientists, but not necessarily advanced machine learning practitioners, to straightforwardly apply the offered set of tools to their own research. To conclude, we explore the state-of-the-art of other disciplines with a broader experience in applying artificial intelligence methods to their research (e.g., high-energy physics, astronomy, Earth sciences, and even robotics, videogames, or marketing and finances), in order to narrow down the incoming future of electron microscopy, its challenges and outlook.

SIM-STEM Lab: Incorporating Compressed Sensing Theory for Fast STEM Simulation

Recently it has been shown that precise dose control and an increase in the overall acquisition speed of atomic resolution scanning transmission electron microscope (STEM) images can be achieved by acquiring only a small fraction of the pixels in the image experimentally and then reconstructing the full image using an inpainting algorithm. In this paper, we apply the same inpainting approach (a form of compressed sensing) to simulated, sub-sampled atomic resolution STEM images. We find that it is possible to significantly sub-sample the area that is simulated, the number of g-vectors contributing the image, and the number of frozen phonon configurations contributing to the final image while still producing an acceptable fit to a fully sampled simulation. Here we discuss the parameters that we use and how the resulting simulations can be quantifiably compared to the full simulations. As with any Compressed Sensing methodology, care must be taken to ensure that isolated events are not excluded from the process, but the observed increase in simulation speed provides significant opportunities for real time simulations, image classification and analytics to be performed as a supplement to experiments on a microscope to be developed in the future.

Sub-Sampled Imaging for STEM: Maximising Image Speed, Resolution and Precision Through Reconstruction Parameter Refinement

Sub-sampling during image acquisition in scanning transmission electron microscopy (STEM) has been shown to provide a means to increase the overall speed of acquisition while at the same time providing an efficient means to control the dose, dose rate and dose overlap delivered to the sample. In this paper, we discuss specifically the parameters used to reconstruct sub-sampled images and highlight their effect on inpainting using the beta-process factor analysis (BPFA) methodology. The selection of the main control parameters can have a significant effect on the resolution, precision and sensitivity of the final inpainted images, and here we demonstrate a method by which these parameters can be optimised for any image in STEM. As part of this work, we also provide a link to open source code and a tutorial on its use, whereby these parameters can be tested for any datasets. When coupled with the hardware necessary to rapidly sub-sample images in STEM, this approach can have significant implications for imaging beam sensitive materials and dynamic processes.

Applications of deep learning in electron microscopy

We review the growing use of machine learning in electron microscopy (EM) driven in part by the availability of fast detectors operating at kiloHertz frame rates leading to large data sets that cannot be processed using manually implemented algorithms. We summarize the various network architectures and error metrics that have been applied to a range of EM-related problems including denoising and inpainting. We then provide a review of the application of these in both physical and life sciences, highlighting how conventional networks and training data have been specifically modified for EM.

Towards automating structural discovery in scanning transmission electron microscopy *

Scanning transmission electron microscopy is now the primary tool for exploring functional materials on the atomic level. Often, features of interest are highly localized in specific regions in the material, such as ferroelectric domain walls, extended defects, or second phase inclusions. Selecting regions to image for structural and chemical discovery via atomically resolved imaging has traditionally proceeded via human operators making semi-informed judgements on sampling locations and parameters. Recent efforts at automation for structural and physical discovery have pointed towards the use of ‘active learning’ methods that utilize Bayesian optimization with surrogate models to quickly find relevant regions of interest. Yet despite the potential importance of this direction, there is a general lack of certainty in selecting relevant control algorithms and how to balance a priori knowledge of the material system with knowledge derived during experimentation. Here we address this gap by developing the automated experiment workflows with several combinations to both illustrate the effects of these choices and demonstrate the tradeoffs associated with each in terms of accuracy, robustness, and susceptibility to hyperparameters for structural discovery. We discuss possible methods to build descriptors using the raw image data and deep learning based semantic segmentation, as well as the implementation of variational autoencoder based representation. Furthermore, each workflow is applied to a range of feature sizes including NiO pillars within a La:SrMnO3 matrix, ferroelectric domains in BiFeO3, and topological defects in graphene. The code developed in this manuscript is open sourced and will be released at github.com/nccreang/AE_Workflows.

Compressed sensing in fluorescence microscopy

Compressed sensing (CS) is a signal processing approach that solves ill-posed inverse problems, from under-sampled data with respect to the Nyquist criterium. CS exploits sparsity constraints based on the knowledge of prior information, relative to the structure of the object in the spatial or other domains. It is commonly used in image and video compression as well as in scientific and medical applications, including computed tomography and magnetic resonance imaging. In the field of fluorescence microscopy, it has been demonstrated to be valuable for fast and high-resolution imaging, from single-molecule localization, super-resolution to light-sheet microscopy. Furthermore, CS has found remarkable applications in the field of mesoscopic imaging, facilitating the study of small animals' organs and entire organisms. This review article illustrates the working principles of CS, its implementations in optical imaging and discusses several relevant uses of CS in the field of fluorescence imaging from super-resolution microscopy to mesoscopy.

High temporal-resolution scanning transmission electron microscopy using sparse-serpentine scan pathways

Scanning transmission electron microscopy (STEM) provides structural analysis with sub-angstrom resolution. But the pixel-by-pixel scanning process is a limiting factor in acquiring high-speed data. Different strategies have been implemented to increase scanning speeds while at the same time minimizing beam damage via optimizing the scanning strategy. Here, we achieve the highest possible scanning speed by eliminating the image acquisition dead time induced by the beam flyback time combined with reducing the amount of scanning pixels via sparse imaging. A calibration procedure was developed to compensate for the hysteresis of the magnetic scan coils. A combination of sparse and serpentine scanning routines was tested for a crystalline thin film, gold nanoparticles, and in an in-situ liquid phase STEM experiment. Frame rates of 92, 23 and 5.8 s^-1 were achieved for images of a width of 128, 256, and 512 pixels, respectively. The methods described here can be applied to single-particle tracking and analysis of radiation sensitive materials.

Making the Most of your Electrons: Challenges and Opportunities in Characterizing Hybrid Interfaces with STEM

Inspired by the unique architectures composed of hard and soft materials in natural and biological systems, synthetic hybrid structures and associated soft-hard interfaces have recently evoked significant interest. Soft matter is typically dominated by fluctuations even at room temperature, while hard matter (which often serves as the substrate or anchor for the soft component) is governed by rigid mechanical behavior. This dichotomy offers considerable opportunities to leverage the disparate properties offered by these components across a wide spectrum spanning from basic science to engineering insights with significant technological overtones. Such hybrid structures, which include polymer nanocomposites, DNA functionalized nanoparticle superlattices and metal organic frameworks to name a few, have delivered promising insights into the areas of catalysis, environmental remediation, optoelectronics, medicine, and beyond. The interfacial structure between these hard and soft phases exists across a variety of length scales and often strongly influence the functionality of hybrid systems. While scanning/transmission electron microscopy (S/TEM) has proven to be a valuable tool for acquiring intricate molecular and nanoscale details of these interfaces, the unusual nature of hybrid composites presents a suite of challenges that make assessing or establishing the classical structure-property relationships especially difficult. These include challenges associated with preparing electron-transparent samples and obtaining sufficient contrast to resolve the interface between dissimilar materials given the dose sensitivity of soft materials. We discuss each of these challenges and supplement a review of recent developments in the field with additional experimental investigations and simulations to present solutions for attaining a nano or atomic-level understanding of these interfaces. These solutions present a host of opportunities for investigating and understanding the role interfaces play in this unique class of functional materials.

RapidEELS: machine learning for denoising and classification in rapid acquisition electron energy loss spectroscopy

Recent advances in detectors for imaging and spectroscopy have afforded in situ, rapid acquisition of hyperspectral data. While electron energy loss spectroscopy (EELS) data acquisition speeds with electron counting are regularly reaching 400 frames per second with near-zero read noise, signal to noise ratio (SNR) remains a challenge owing to fundamental counting statistics. In order to advance understanding of transient materials phenomena during rapid acquisition EELS, trustworthy analysis of noisy spectra must be demonstrated. In this study, we applied machine learning techniques to denoise high frame rate spectra, benchmarking with slower frame rate "ground truths". The results provide a foundation for reliable use of low SNR data acquired in rapid, in-situ spectroscopy experiments. Such a tool-set is a first step toward both automation in microscopy as well as use of these methods to interrogate otherwise poorly understood transformations.

Automated and Autonomous Experiments in Electron and Scanning Probe Microscopy

Machine learning and artificial intelligence (ML/AI) are rapidly becoming an indispensable part of physics research, with domain applications ranging from theory and materials prediction to high-throughput data analysis. In parallel, the recent successes in applying ML/AI methods for autonomous systems from robotics to self-driving cars to organic and inorganic synthesis are generating enthusiasm for the potential of these techniques to enable automated and autonomous experiments (AE) in imaging. Here, we aim to analyze the major pathways toward AE in imaging methods with sequential image formation mechanisms, focusing on scanning probe microscopy (SPM) and (scanning) transmission electron microscopy ((S)TEM). We argue that automated experiments should necessarily be discussed in a broader context of the general domain knowledge that both informs the experiment and is increased as the result of the experiment. As such, this analysis should explore the human and ML/AI roles prior to and during the experiment and consider the latencies, biases, and prior knowledge of the decision-making process. Similarly, such discussion should include the limitations of the existing imaging systems, including intrinsic latencies, non-idealities, and drifts comprising both correctable and stochastic components. We further pose that the role of the AE in microscopy is not the exclusion of human operators (as is the case for autonomous driving), but rather automation of routine operations such as microscope tuning, etc., prior to the experiment, and conversion of low latency decision making processes on the time scale spanning from image acquisition to human-level high-order experiment planning. Overall, we argue that ML/AI can dramatically alter the (S)TEM and SPM fields; however, this process is likely to be highly nontrivial and initiated by combined human-ML workflows and will bring challenges both from the microscope and ML/AI sides. At the same time, these methods will enable opportunities and paradigms for scientific discovery and nanostructure fabrication.

Computational Evaluation of Sparse Coding on off-axis Electron Holograms: Comparison Between Charge-Coupled Device and Direct-Detection Device Cameras

The effectiveness of sparse coding for image inpainting and denoising of off-axis electron holograms was examined computationally based on hologram simulations according to considerations of two types of electron detectors, namely, charge-coupled device (CCD) and direct-detection device (DDD) cameras. In this simulation, we used a simple-phase object with a phase step such as a semiconductor p-n junction and assumed that the holograms recorded by the CCD camera include shot noise, dark-current, and read-out noise, while those recorded by the DDD camera include only shot noise. Simulated holograms with various electron doses were sparsely coded. Even though interference fringes cannot be recognized in the simulated CCD and DDD holograms when subjected to electron doses (per pixel) equal to 1 and 0.01, respectively, both the corresponding sparse-coded holograms exhibit meaningful interference fringes. We demonstrate that a combination of the DDD camera and sparse coding reduces the requisite dose used to obtain holograms to values less than one-thousandth compared with the CCD camera without image postprocessing. This combination is expected to generate lower-dose and/or higher-speed electron holography.

A Close Look at Molecular Self-Assembly with the Transmission Electron Microscope

Molecular self-assembly is pervasive in the formation of living and synthetic materials. Knowledge gained from research into the principles of molecular self-assembly drives innovation in the biological, chemical, and materials sciences. Self-assembly processes span a wide range of temporal and spatial domains and are often unintuitive and complex. Studying such complex processes requires an arsenal of analytical and computational tools. Within this arsenal, the transmission electron microscope stands out for its unique ability to visualize and quantify self-assembly structures and processes. This review describes the contribution that the transmission electron microscope has made to the field of molecular self-assembly. An emphasis is placed on which TEM methods are applicable to different structures and processes and how TEM can be used in combination with other experimental or computational methods. Finally, we provide an outlook on the current challenges to, and opportunities for, increasing the impact that the transmission electron microscope can have on molecular self-assembly.

Super-compression of large electron microscopy time series by deep compressive sensing learning

The development of ultrafast detectors for electron microscopy (EM) opens a new door to exploring dynamics of nanomaterials; however, it raises grand challenges for big data processing and storage. Here, we combine deep learning and temporal compressive sensing (TCS) to propose a novel EM big data compression strategy. Specifically, TCS is employed to compress sequential EM images into a single compressed measurement; an end-to-end deep learning network is leveraged to reconstruct the original images. Owing to the significantly improved compression efficiency and built-in denoising capability of the deep learning framework over conventional JPEG compression, compressed videos with a compression ratio of up to 30 can be reconstructed with high fidelity. Using this approach, considerable encoding power, memory, and transmission bandwidth can be saved, allowing it to be deployed to existing detectors. We anticipate the proposed technique will have far-reaching applications in edge computing for EM and other imaging techniques.

•

A novel framework, i.e., TCS-DL, has been proposed for big data compressing for EM

•

The proposed TCL-DL outperforms JPEG due to the built-in denoising capability

•

Considerable power, in situ memory, and transmission bandwidth could be saved

•

The proposed TCL-DL is a novel and promising way for EM data compressing

The rapid development of electron microscopy (EM) opens a new door to exploring physical sciences; however, it raises grand challenges and urgent needs for big data processing. Therefore, it is crucial to compress the EM data. But existing compression methods developed for natural images do not perform well in EM images. In this paper, by combining deep learning and temporal compressive sensing, we propose a novel compression strategy specifically for EM data processing. Owing to the improved compression efficiency and built-in denoising capability of our framework over JPEG compression, compressed videos with compression ratio of 30 can be reconstructed with high fidelity. Therefore, considerable (encoding) power, in situ memory, and transmission bandwidth are expected to be saved. In the future, we will strive to increase the compression ratio without reducing the reconstruction quality. And we believe our proposed EM compression method has a wide application for the EM community.

Deep learning in electron microscopy

Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy.

Controlling radiolysis chemistry on the nanoscale in liquid cell scanning transmission electron microscopy

When high-energy electrons from a scanning transmission electron microscope (STEM) are incident on a liquid, the vast majority of the chemical reactions that are observed are induced by the radiolysis breakdown of the liquid molecules. In the study of liquids, the radiolysis products of pure water are well known, and their rate of formation for a given flux of high-energy electrons has been studied intensively over the last few years for uniform TEM illumination. In this paper, we demonstrate that the temporal and spatial distribution of the electron illumination can significantly affect the final density of radiolysis products in water and even change the type of reaction taking place. We simulate the complex array of possible spatial/temporal distributions of electrons that are accessible experimentally by controlling the size, the scan rate and the hopping distance of the electron probe in STEM mode and then compare the results to the uniformly illuminated TEM mode of imaging. By distributing the electron dose both spatially and temporally in the STEM through a randomised "spot-scan" mode of imaging, the diffusion overlap of the radiolysis products can be reduced, and the resulting reactions can be more readily controlled. This control allows the resolution of the images to be separated from the speed of the induced reaction (which is based on beam current alone) and this facet of the experiment will allow a wide range of chemical reactions to be uniquely tailored and observed in all liquid cell STEM experiments.

Development of a Practicable Digital Pulse Read-Out for Dark-Field STEM

When characterizing beam-sensitive materials in the scanning transmission electron microscope (STEM), low-dose techniques are essential for the reliable observation of samples in their true state. A simple route to minimize both the total electron-dose and the dose-rate is to reduce the electron beam-current and/or raster the probe at higher speeds. At the limit of these settings, and with current detectors, the resulting images suffer from unacceptable artifacts, including signal-streaking, detector-afterglow, and poor signal-to-noise ratios (SNRs). In this article, we present an alternative approach to capture dark-field STEM images by pulse-counting individual electrons as they are scattered to the annular dark-field (ADF) detector. Digital images formed in this way are immune from analog artifacts of streaking or afterglow and allow clean, high-SNR images to be obtained even at low beam-currents. We present results from both a ThermoFisher FEI Titan G2 operated at 300 kV and a Nion UltraSTEM200 operated at 200 kV, and compare the images to conventional analog recordings. ADF data are compared with analog counterparts for each instrument, a digital detector-response scan is performed on the Titan, and the overall rastering efficiency is evaluated for various scanning parameters.

Minimising damage in high resolution scanning transmission electron microscope images of nanoscale structures and processes

Beam damage caused during acquisition of the highest resolution images is the current limitation in the vast majority of experiments performed in a scanning transmission electron microscope (STEM). While the principles behind the processes of knock-on and radiolysis damage are well-known (as are other contributing effects, such as heat and electric fields), understanding how and especially when beam damage is distributed across the entire sample volume during an experiment has not been examined in detail. Here we use standard models for damage and diffusion to elucidate how beam damage spreads across the sample as a function of the microscope conditions to determine an "optimum" sampling approach that maximises the high-resolution information in any image acquisition. We find that the standard STEM approach of scanning an image sequentially accelerates damage because of increased overlap of diffusion processes. These regions of accelerated damage can be significantly decelerated by increasing the distance between the acquired pixels in the scan, forming a "spotscan" mode of acquisition. The optimum distance between these pixels can be broadly defined by the fundamental properties of each material, allowing experiments to be designed for specific beam sensitive materials. As an added bonus, if we use inpainting to reconstruct the sparse distribution of pixels in the image we can significantly increase the speed of the STEM process, allowing dynamic phenomena, and the onset of damage, to be studied directly.

Denoising of series electron holograms using tensor decomposition

In this study, a noise-reduction technique for series low-dose electron holograms using tensor decomposition is demonstrated through simulation. We treated an entire dataset of the series holograms with Poisson noise as a third-order tensor, which is a stack of 2D holograms. The third-order tensor, which is decomposed into a core tensor and three factor matrices, is approximated as a lower-rank tensor using only noise-free principal components. This technique is applied to simulated holograms by assuming a p-n junction in a semiconductor sample. The peak signal-to-noise ratios of the holograms and the reconstructed phase maps have been improved significantly using tensor decomposition. Moreover, the proposed method was applied to a more practical situation of time-resolved in situ electron holography by considering a nonuniform fringe contrast and fringe drift relative to the sample. The accuracy and precision of the reconstructed phase maps were quantitatively evaluated to demonstrate its effectiveness for in situ experiments and low-dose experiments on beam-sensitive materials.

Fast reconstruction of atomic-scale STEM-EELS images from sparse sampling

This paper discusses the reconstruction of partially sampled spectrum-images to accelerate the acquisition in scanning transmission electron microscopy (STEM). The problem of image reconstruction has been widely considered in the literature for many imaging modalities, but only a few attempts handled 3D data such as spectral images acquired by STEM electron energy loss spectroscopy (EELS). Besides, among the methods proposed in the microscopy literature, some are fast but inaccurate while others provide accurate reconstruction but at the price of a high computation burden. Thus none of the proposed reconstruction methods fulfills our expectations in terms of accuracy and computation complexity. In this paper, we propose a fast and accurate reconstruction method suited for atomic-scale EELS. This method is compared to popular solutions such as beta process factor analysis (BPFA) which is used for the first time on STEM-EELS images. Experiments based on real as synthetic data will be conducted.

Dynamic imaging of lithium in solid-state batteries by operando electron energy-loss spectroscopy with sparse coding

Lithium-ion transport in cathodes, anodes, solid electrolytes, and through their interfaces plays a crucial role in the electrochemical performance of solid-state lithium-ion batteries. Direct visualization of the lithium-ion dynamics at the nanoscale provides valuable insight for understanding the fundamental ion behaviour in batteries. Here, we report the dynamic changes of lithium-ion movement in a solid-state battery under charge and discharge reactions by time-resolved operando electron energy-loss spectroscopy with scanning transmission electron microscopy. Applying image denoising and super-resolution via sparse coding drastically improves the temporal and spatial resolution of lithium imaging. Dynamic observation reveals that the lithium ions in the lithium cobaltite cathode are complicatedly extracted with diffusion through the lithium cobaltite domain boundaries during charging. Even in the open-circuit state, they move inside the cathode. Operando electron energy-loss spectroscopy with sparse coding is a promising combination to visualize the ion dynamics and clarify the fundamentals of solid-state electrochemistry.

Understanding lithium ion dynamics holds the key to unlocking better battery materials and devices. Here, by combining electron energy-loss spectroscopy and machine learning, the authors reveal how lithium is extracted from LiCoO₂ cathode used in a solid-state battery.

Liquid-Phase Electron Microscopy for Soft Matter Science and Biology

Innovations in liquid-phase electron microscopy (LP-EM) have made it possible to perform experiments at the optimized conditions needed to examine soft matter. The main obstacle is conducting experiments in such a way that electron beam radiation can be used to obtain answers for scientific questions without changing the structure and (bio)chemical processes in the sample due to the influence of the radiation. By overcoming these experimental difficulties at least partially, LP-EM has evolved into a new microscopy method with nanometer spatial resolution and sub-second temporal resolution for analysis of soft matter in materials science and biology. Both experimental design and applications of LP-EM for soft matter materials science and biological research are reviewed, and a perspective of possible future directions is given.

Simulation-Trained Sparse Coding for High-Precision Phase Imaging in Low-Dose Electron Holography

We broaden the applicability of sparse coding, a machine learning method, to low-dose electron holography by using simulated holograms for learning and validation processes. The holograms, with shot noise, are prepared to generate a model, or a dictionary, that includes basic features representing interference fringes. The dictionary is applied to sparse representations of other simulated holograms with various signal-to-noise ratios (SNRs). Results demonstrate that this approach successfully removes noise for holograms with an extremely small SNR of 0.10, and that the denoised holograms provide the accurate phase distribution. Furthermore, this study demonstrates that the dictionary learned from the simulated holograms can be applied to denoising of experimental holograms of a p-n junction specimen recorded with different exposure times. The results indicate that the simulation-trained sparse coding is suitable for use over a wide range of imaging conditions, in particular for observing electron beam-sensitive materials.

Partial Scanning Transmission Electron Microscopy with Deep Learning

Compressed sensing algorithms are used to decrease electron microscope scan time and electron beam exposure with minimal information loss. Following successful applications of deep learning to compressed sensing, we have developed a two-stage multiscale generative adversarial neural network to complete realistic 512 × 512 scanning transmission electron micrographs from spiral, jittered gridlike, and other partial scans. For spiral scans and mean squared error based pre-training, this enables electron beam coverage to be decreased by 17.9× with a 3.8% test set root mean squared intensity error, and by 87.0× with a 6.2% error. Our generator networks are trained on partial scans created from a new dataset of 16227 scanning transmission electron micrographs. High performance is achieved with adaptive learning rate clipping of loss spikes and an auxiliary trainer network. Our source code, new dataset, and pre-trained models are publicly available.

Imaging Beam-Sensitive Materials by Electron Microscopy

Electron microscopy allows the extraction of multidimensional spatiotemporally correlated structural information of diverse materials down to atomic resolution, which is essential for figuring out their structure-property relationships. Unfortunately, the high-energy electrons that carry this important information can cause damage by modulating the structures of the materials. This has become a significant problem concerning the recent boost in materials science applications of a wide range of beam-sensitive materials, including metal-organic frameworks, covalent-organic frameworks, organic-inorganic hybrid materials, 2D materials, and zeolites. To this end, developing electron microscopy techniques that minimize the electron beam damage for the extraction of intrinsic structural information turns out to be a compelling but challenging need. This article provides a comprehensive review on the revolutionary strategies toward the electron microscopic imaging of beam-sensitive materials and associated materials science discoveries, based on the principles of electron-matter interaction and mechanisms of electron beam damage. Finally, perspectives and future trends in this field are put forward.

Liquid-phase electron microscopy imaging of cellular and biomolecular systems

Liquid-phase electron microscopy, a new method for real-time nanoscopic imaging in liquid, makes it possible to study cells or biomolecules with a singular combination of spatial and temporal resolution. We review the state of the art in biological research in this growing and promising field.

Emerging Electron Microscopy Techniques for Probing Functional Interfaces in Energy Materials

Interfaces play a fundamental role in many areas of chemistry. However, their localized nature requires characterization techniques with high spatial resolution in order to fully understand their structure and properties. State-of-the-art atomic resolution or in situ scanning transmission electron microscopy and electron energy-loss spectroscopy are indispensable tools for characterizing the local structure and chemistry of materials with single-atom resolution, but they are not able to measure many properties that dictate function, such as vibrational modes or charge transfer, and are limited to room-temperature samples containing no liquids. Here, we outline emerging electron microscopy techniques that are allowing these limitations to be overcome and highlight several recent studies that were enabled by these techniques. We then provide a vision for how these techniques can be paired with each other and with in situ methods to deliver new insights into the static and dynamic behavior of functional interfaces.

Toward Electrochemical Studies on the Nanometer and Atomic Scales: Progress, Challenges, and Opportunities

Electrochemical reactions and ionic transport underpin the operation of a broad range of devices and applications, from energy storage and conversion to information technologies, as well as biochemical processes, artificial muscles, and soft actuators. Understanding the mechanisms governing function of these applications requires probing local electrochemical phenomena on the relevant time and length scales. Here, we discuss the challenges and opportunities for extending electrochemical characterization probes to the nanometer and ultimately atomic scales, including challenges in down-scaling classical methods, the emergence of novel probes enabled by nanotechnology and based on emergent physics and chemistry of nanoscale systems, and the integration of local data into macroscopic models. Scanning probe microscopy (SPM) methods based on strain detection, potential detection, and hysteretic current measurements are discussed. We further compare SPM to electron beam probes and discuss the applicability of electron beam methods to probe local electrochemical behavior on the mesoscopic and atomic levels. Similar to a SPM tip, the electron beam can be used both for observing behavior and as an active electrode to induce reactions. We briefly discuss new challenges and opportunities for conducting fundamental scientific studies, matter patterning, and atomic manipulation arising in this context.

Electrostatic subframing and compressive-sensing video in transmission electron microscopy

We present kilohertz-scale video capture rates in a transmission electron microscope, using a camera normally limited to hertz-scale acquisition. An electrostatic deflector rasters a discrete array of images over a large camera, decoupling the acquisition time per subframe from the camera readout time. Total-variation regularization allows features in overlapping subframes to be correctly placed in each frame. Moreover, the system can be operated in a compressive-sensing video mode, whereby the deflections are performed in a known pseudorandom sequence. Compressive sensing in effect performs data compression before the readout, such that the video resulting from the reconstruction can have substantially more total pixels than that were read from the camera. This allows, for example, 100 frames of video to be encoded and reconstructed using only 15 captured subframes in a single camera exposure. We demonstrate experimental tests including laser-driven melting/dewetting, sintering, and grain coarsening of nanostructured gold, with reconstructed video rates up to 10 kHz. The results exemplify the power of the technique by showing that it can be used to study the fundamentally different temporal behavior for the three different physical processes. Both sintering and coarsening exhibited self-limiting behavior, whereby the process essentially stopped even while the heating laser continued to strike the material. We attribute this to changes in laser absorption and to processes inherent to thin-film coarsening. In contrast, the dewetting proceeded at a relatively uniform rate after an initial incubation time consistent with the establishment of a steady-state temperature profile.

Inpainting vs denoising for dose reduction in scanning-beam microscopies

We consider sampling strategies for reducing the radiation dose during image acquisition in scanning-beam microscopies, such as SEM, STEM, and STXM. Our basic assumption is that we may acquire subsampled image data (with some pixels missing) and then inpaint the missing data using a compressed-sensing approach. Our noise model consists of Poisson noise plus random Gaussian noise. We include the possibility of acquiring fully-sampled image data, in which case the inpainting approach reduces to a denoising procedure. We use numerical simulations to compare the accuracy of reconstructed images with the "ground truths." The results generally indicate that, for sufficiently high radiation doses, higher sampling rates achieve greater accuracy, commensurate with well-established literature. However, for very low radiation doses, where the Poisson noise and/or random Gaussian noise begins to dominate, then our results indicate that subsampling/inpainting can result in smaller reconstruction errors. We also present an information-theoretic analysis, which allows us to quantify the amount of information gained through the different sampling strategies and enables some broader discussion of the main results.

Tomographic Collection of Block-Based Sparse STEM Images: Practical Implementation and Impact on the Quality of the 3D Reconstructed Volume

The reduction of the electron dose in electron tomography of biological samples is of high significance to diminish radiation damages. Simulations have shown that sparse data collection can perform efficient electron dose reduction. Frameworks based on compressive-sensing or inpainting algorithms have been proposed to accurately reconstruct missing information in sparse data. The present work proposes a practical implementation to perform tomographic collection of block-based sparse images in scanning transmission electron microscopy. The method has been applied on sections of chemically-fixed and resin-embedded Trypanosoma brucei cells. There are 3D reconstructions obtained from various amounts of downsampling, which are compared and eventually the limits of electron dose reduction using this method are explored.

Structure-Property Relationships in Lanthanide-Doped Upconverting Nanocrystals: Recent Advances in Understanding Core-Shell Structures

The production of upconverting nanostructures with tailored optical properties is of major technological interest, and rapid progress toward the realization of such production has been made in recent years. Ultimately, accurate understanding of nanostructure organization will lead to design rules for accurately tailoring optical properties. Here, the context of open questions still of general importance to the upconversion and nanocrystal communities is presented, with a particular emphasis on the structure-property relationships of core-shell upconverting nanocrystals. Although the optical properties of the latter have been thoroughly investigated, little is known regarding their atomic-scale organization. Indeed, solving the atomic-scale structure of such nanomaterials is challenging because of their intrinsic nonperiodic nature. Familiar concepts of crystallography are no longer appropriate; chemical and structural modulation waves must be introduced. To reveal the exact core-shell structures, innovative characterization techniques need to be applied and developed, as discussed herein. The continued development and application of structural characterization techniques will be vital to consolidate the currently incomplete link between atomic-scale structure and upconversion properties. This will ultimately provide a valuable contribution to the emerging detailed guidelines on how to better design upconverting nanostructures to achieve given optical properties in terms of efficiency, absorption, spectral emission, and dynamics.

Compressed Sensing of Scanning Transmission Electron Microscopy (STEM) With Nonrectangular Scans

Scanning transmission electron microscopy (STEM) has become the main stay for materials characterization on atomic level, with applications ranging from visualization of localized and extended defects to mapping order parameter fields. In recent years, attention has focused on the potential of STEM to explore beam induced chemical processes and especially manipulating atomic motion, enabling atom-by-atom fabrication. These applications, as well as traditional imaging of beam sensitive materials, necessitate increasing the dynamic range of STEM in imaging and manipulation modes, and increasing the absolute scanning speed which can be achieved by combining sparse sensing methods with nonrectangular scanning trajectories. Here we have developed a general method for real-time reconstruction of sparsely sampled images from high-speed, noninvasive and diverse scanning pathways, including spiral scan and Lissajous scan. This approach is demonstrated on both the synthetic data and experimental STEM data on the beam sensitive material graphene. This work opens the door for comprehensive investigation and optimal design of dose efficient scanning strategies and real-time adaptive inference and control of e-beam induced atomic fabrication.