Applying compressive sensing to TEM video: a substantial frame rate increase on any camera

Simultaneous 2D Projection and 3D Topographic Imaging of Gas-Dependent Dynamics of Catalytic Nanoparticles

Catalyst deactivation through pathways such as sintering of nanoparticles and degradation of the support is a critical factor when designing high-performance catalysts. Here, structural changes of supported nanoparticle catalysts are investigated in controlled gas environments (O2, H2O, and H2) at different temperatures by imaging simultaneously the nanoparticle structures in 2D projection and the 3D surface-sensitive topography. Platinum nanoparticles on carbon support as a model system are imaged in an environmental transmission electron microscope (ETEM), with concurrent acquisition of high-angle annular dark field scanning TEM (HAADF-STEM) and secondary electron (SE) images. Particle migration and coalescence occurs and shows gas-dependent kinetics, with nanoparticles moving across and through the support during and after coalescence. The temperature required for motion is lower in O2 than in H2O and H2, explained through the nature of the gas/nanoparticle interactions. In O2 and H2, the carbon support degrades by trench formation along migration pathways, and the particles move continuously, indicating a chemical reaction between gas and support. In H2O gas, motion is more discontinuous and oriented particle attachment occurs, as expected from theoretical predictions. These results suggest that multimodal imaging in ETEM that combines HAADF-STEM and SE data provides comprehensive information regarding catalyst dynamics and degradation mechanisms.

Atomic-level imaging of beam-sensitive COFs and MOFs by low-dose electron microscopy

Electron microscopy, an important technique that allows for the precise determination of structural information with high spatiotemporal resolution, has become indispensable in unravelling the complex relationships between material structure and properties ranging from mesoscale morphology to atomic arrangement. However, beam-sensitive materials, particularly those comprising organic components such as metal-organic frameworks (MOFs) and covalent organic frameworks (COFs), would suffer catastrophic damage from the high energy electrons, hindering the determination of atomic structures. A low-dose approach has arisen as a possible solution to this problem based on the integration of advancements in several aspects: electron optical system, detector, image processing, and specimen preservation. This article summarizes the transmission electron microscopy characterization of MOFs and COFs, including local structures, host-guest interactions, and interfaces at the atomic level. Revolutions in advanced direct electron detectors, algorithms in image acquisition and processing, and emerging methodology for high quality low-dose imaging are also reviewed. Finally, perspectives on the future development of electron microscopy methodology with the support of computer science are presented.

Pulsed-beam transmission electron microscopy and radiation damage

We review the use of pulsed electron-beams in transmission electron microscopes (TEMs) for the purpose of mitigating specimen damage. We begin by placing the importance of TEMs with respect to materials characterization into proper context, and we provide a brief overview of established methods for reducing or eliminating the deleterious effects of beam-induced damage. We then introduce the concept of pulsed-beam TEM, and we briefly describe the basic methods and instrument configurations used to create so-called temporally structured electron beams. Following a brief overview of the use of high-dose-rate pulsed-electron beams in cancer radiation therapy, we review historical speculations and more recent compelling but mostly anecdotal findings of a pulsed-beam TEM damage effect. This is followed by an in-depth technical review of recent works seeking to establish cause-and-effect relationships, to conclusively uncover the presence of an effect, and to explore the practicality of the approach. These studies, in particular, provide the most compelling evidence to date that using a pulsed electron beam in the TEM is indeed a viable way to mitigate damage. Throughout, we point out current gaps in understanding, and we conclude with a brief perspective of current needs and future directions.

Time-resolved transmission electron microscopy for nanoscale chemical dynamics

The ability of transmission electron microscopy (TEM) to image a structure ranging from millimetres to Ångströms has made it an indispensable component of the toolkit of modern chemists. TEM has enabled unprecedented understanding of the atomic structures of materials and how structure relates to properties and functions. Recent developments in TEM have advanced the technique beyond static material characterization to probing structural evolution on the nanoscale in real time. Accompanying advances in data collection have pushed the temporal resolution into the microsecond regime with the use of direct-electron detectors and down to the femtosecond regime with pump-probe microscopy. Consequently, studies have deftly applied TEM for understanding nanoscale dynamics, often in operando. In this Review, time-resolved in situ TEM techniques and their applications for probing chemical and physical processes are discussed, along with emerging directions in the TEM field.

SIM-STEM Lab: Incorporating Compressed Sensing Theory for Fast STEM Simulation

Recently it has been shown that precise dose control and an increase in the overall acquisition speed of atomic resolution scanning transmission electron microscope (STEM) images can be achieved by acquiring only a small fraction of the pixels in the image experimentally and then reconstructing the full image using an inpainting algorithm. In this paper, we apply the same inpainting approach (a form of compressed sensing) to simulated, sub-sampled atomic resolution STEM images. We find that it is possible to significantly sub-sample the area that is simulated, the number of g-vectors contributing the image, and the number of frozen phonon configurations contributing to the final image while still producing an acceptable fit to a fully sampled simulation. Here we discuss the parameters that we use and how the resulting simulations can be quantifiably compared to the full simulations. As with any Compressed Sensing methodology, care must be taken to ensure that isolated events are not excluded from the process, but the observed increase in simulation speed provides significant opportunities for real time simulations, image classification and analytics to be performed as a supplement to experiments on a microscope to be developed in the future.

Sub-Sampled Imaging for STEM: Maximising Image Speed, Resolution and Precision Through Reconstruction Parameter Refinement

Sub-sampling during image acquisition in scanning transmission electron microscopy (STEM) has been shown to provide a means to increase the overall speed of acquisition while at the same time providing an efficient means to control the dose, dose rate and dose overlap delivered to the sample. In this paper, we discuss specifically the parameters used to reconstruct sub-sampled images and highlight their effect on inpainting using the beta-process factor analysis (BPFA) methodology. The selection of the main control parameters can have a significant effect on the resolution, precision and sensitivity of the final inpainted images, and here we demonstrate a method by which these parameters can be optimised for any image in STEM. As part of this work, we also provide a link to open source code and a tutorial on its use, whereby these parameters can be tested for any datasets. When coupled with the hardware necessary to rapidly sub-sample images in STEM, this approach can have significant implications for imaging beam sensitive materials and dynamic processes.

Towards automating structural discovery in scanning transmission electron microscopy *

Scanning transmission electron microscopy is now the primary tool for exploring functional materials on the atomic level. Often, features of interest are highly localized in specific regions in the material, such as ferroelectric domain walls, extended defects, or second phase inclusions. Selecting regions to image for structural and chemical discovery via atomically resolved imaging has traditionally proceeded via human operators making semi-informed judgements on sampling locations and parameters. Recent efforts at automation for structural and physical discovery have pointed towards the use of ‘active learning’ methods that utilize Bayesian optimization with surrogate models to quickly find relevant regions of interest. Yet despite the potential importance of this direction, there is a general lack of certainty in selecting relevant control algorithms and how to balance a priori knowledge of the material system with knowledge derived during experimentation. Here we address this gap by developing the automated experiment workflows with several combinations to both illustrate the effects of these choices and demonstrate the tradeoffs associated with each in terms of accuracy, robustness, and susceptibility to hyperparameters for structural discovery. We discuss possible methods to build descriptors using the raw image data and deep learning based semantic segmentation, as well as the implementation of variational autoencoder based representation. Furthermore, each workflow is applied to a range of feature sizes including NiO pillars within a La:SrMnO3 matrix, ferroelectric domains in BiFeO3, and topological defects in graphene. The code developed in this manuscript is open sourced and will be released at github.com/nccreang/AE_Workflows.

Automated and Autonomous Experiments in Electron and Scanning Probe Microscopy

Machine learning and artificial intelligence (ML/AI) are rapidly becoming an indispensable part of physics research, with domain applications ranging from theory and materials prediction to high-throughput data analysis. In parallel, the recent successes in applying ML/AI methods for autonomous systems from robotics to self-driving cars to organic and inorganic synthesis are generating enthusiasm for the potential of these techniques to enable automated and autonomous experiments (AE) in imaging. Here, we aim to analyze the major pathways toward AE in imaging methods with sequential image formation mechanisms, focusing on scanning probe microscopy (SPM) and (scanning) transmission electron microscopy ((S)TEM). We argue that automated experiments should necessarily be discussed in a broader context of the general domain knowledge that both informs the experiment and is increased as the result of the experiment. As such, this analysis should explore the human and ML/AI roles prior to and during the experiment and consider the latencies, biases, and prior knowledge of the decision-making process. Similarly, such discussion should include the limitations of the existing imaging systems, including intrinsic latencies, non-idealities, and drifts comprising both correctable and stochastic components. We further pose that the role of the AE in microscopy is not the exclusion of human operators (as is the case for autonomous driving), but rather automation of routine operations such as microscope tuning, etc., prior to the experiment, and conversion of low latency decision making processes on the time scale spanning from image acquisition to human-level high-order experiment planning. Overall, we argue that ML/AI can dramatically alter the (S)TEM and SPM fields; however, this process is likely to be highly nontrivial and initiated by combined human-ML workflows and will bring challenges both from the microscope and ML/AI sides. At the same time, these methods will enable opportunities and paradigms for scientific discovery and nanostructure fabrication.

Denoising of series electron holograms using tensor decomposition

In this study, a noise-reduction technique for series low-dose electron holograms using tensor decomposition is demonstrated through simulation. We treated an entire dataset of the series holograms with Poisson noise as a third-order tensor, which is a stack of 2D holograms. The third-order tensor, which is decomposed into a core tensor and three factor matrices, is approximated as a lower-rank tensor using only noise-free principal components. This technique is applied to simulated holograms by assuming a p-n junction in a semiconductor sample. The peak signal-to-noise ratios of the holograms and the reconstructed phase maps have been improved significantly using tensor decomposition. Moreover, the proposed method was applied to a more practical situation of time-resolved in situ electron holography by considering a nonuniform fringe contrast and fringe drift relative to the sample. The accuracy and precision of the reconstructed phase maps were quantitatively evaluated to demonstrate its effectiveness for in situ experiments and low-dose experiments on beam-sensitive materials.

Liquid-Phase Electron Microscopy for Soft Matter Science and Biology

Innovations in liquid-phase electron microscopy (LP-EM) have made it possible to perform experiments at the optimized conditions needed to examine soft matter. The main obstacle is conducting experiments in such a way that electron beam radiation can be used to obtain answers for scientific questions without changing the structure and (bio)chemical processes in the sample due to the influence of the radiation. By overcoming these experimental difficulties at least partially, LP-EM has evolved into a new microscopy method with nanometer spatial resolution and sub-second temporal resolution for analysis of soft matter in materials science and biology. Both experimental design and applications of LP-EM for soft matter materials science and biological research are reviewed, and a perspective of possible future directions is given.

Simulation-Trained Sparse Coding for High-Precision Phase Imaging in Low-Dose Electron Holography

We broaden the applicability of sparse coding, a machine learning method, to low-dose electron holography by using simulated holograms for learning and validation processes. The holograms, with shot noise, are prepared to generate a model, or a dictionary, that includes basic features representing interference fringes. The dictionary is applied to sparse representations of other simulated holograms with various signal-to-noise ratios (SNRs). Results demonstrate that this approach successfully removes noise for holograms with an extremely small SNR of 0.10, and that the denoised holograms provide the accurate phase distribution. Furthermore, this study demonstrates that the dictionary learned from the simulated holograms can be applied to denoising of experimental holograms of a p-n junction specimen recorded with different exposure times. The results indicate that the simulation-trained sparse coding is suitable for use over a wide range of imaging conditions, in particular for observing electron beam-sensitive materials.

Imaging Beam-Sensitive Materials by Electron Microscopy

Electron microscopy allows the extraction of multidimensional spatiotemporally correlated structural information of diverse materials down to atomic resolution, which is essential for figuring out their structure-property relationships. Unfortunately, the high-energy electrons that carry this important information can cause damage by modulating the structures of the materials. This has become a significant problem concerning the recent boost in materials science applications of a wide range of beam-sensitive materials, including metal-organic frameworks, covalent-organic frameworks, organic-inorganic hybrid materials, 2D materials, and zeolites. To this end, developing electron microscopy techniques that minimize the electron beam damage for the extraction of intrinsic structural information turns out to be a compelling but challenging need. This article provides a comprehensive review on the revolutionary strategies toward the electron microscopic imaging of beam-sensitive materials and associated materials science discoveries, based on the principles of electron-matter interaction and mechanisms of electron beam damage. Finally, perspectives and future trends in this field are put forward.

Laser-free GHz stroboscopic transmission electron microscope: Components, system integration, and practical considerations for pump-probe measurements

A 300 keV transmission electron microscope was modified to produce broadband pulsed beams that can be, in principle, between 40 MHz and 12 GHz, corresponding to temporal resolution in the nanosecond to picosecond range without an excitation laser. The key enabling technology is a pair of phase-matched modulating and de-modulating traveling wave metallic comb striplines (pulsers). An initial temporal resolution of 30 ps was achieved with a strobe frequency of 6.0 GHz. The placement of the pulsers, mounted immediately below the gun, allows for preservation of all optical configurations, otherwise available to the unmodified instrument, and therefore makes such a post-modified instrument for dual-use, i.e., both pulsed-beam mode (i.e., stroboscopic time-resolved) and conventional continuous waveform mode. In this article, we describe the elements inserted into the beam path, challenges encountered during integration with an in-service microscope, and early results from an electric-field-driven pump-probe experiment. We conclude with ideas for making this class of instruments broadly applicable for examining cyclical and repeatable phenomena.

Liquid-phase electron microscopy imaging of cellular and biomolecular systems

Liquid-phase electron microscopy, a new method for real-time nanoscopic imaging in liquid, makes it possible to study cells or biomolecules with a singular combination of spatial and temporal resolution. We review the state of the art in biological research in this growing and promising field.

Methane Mapping with Future Satellite Imaging Spectrometers

This study evaluates a new generation of satellite imaging spectrometers to measure point source methane emissions from anthropogenic sources. We used the Airborne Visible and Infrared Imaging Spectrometer Next Generation(AVIRIS-NG) images with known methane plumes to create two simulated satellite products. One simulation had a 30 m spatial resolution with ~200 Signal-to-Noise Ratio (SNR) in the Shortwave Infrared (SWIR) and the other had a 60 m spatial resolution with ~400 SNR in the SWIR; both products had a 7.5 nm spectral spacing. We applied a linear matched filter with a sparsity prior and an albedo correction to detect and quantify the methane emission in the original AVIRIS-NG images and in both satellite simulations. We also calculated an emission flux for all images. We found that all methane plumes were detectable in all satellite simulations. The flux calculations for the simulated satellite images correlated well with the calculated flux for the original AVIRIS-NG images. We also found that coarsening spatial resolution had the largest impact on the sensitivity of the results. These results suggest that methane detection and quantification of point sources will be possible with the next generation of satellite imaging spectrometers.

Tunable electron beam pulser for picoseconds stroboscopic microscopy in transmission electron microscopes

For two decades, time-resolved transmission electron microscopes (TEM) have relied on pulsed-laser photoemission to generate electron bunches to explore sub-microsecond to sub-picosecond dynamics. Despite the vast successes of photoemission time-resolved TEMs, laser-based systems are inherently complex, thus tend not to be turn-key. In this paper, we report on the successful retrofit of a commercial 200 keV TEM, without an external laser, capable of producing continuously tunable pulsed electron beams with repetition rates from 0.1 GHz up to 12 GHz and a tunable bunch length from tens of nanoseconds down to 10 ps. This innovation enables temporal access into previously inaccessible regimes: i.e., high repetition rate stroboscopic experiments. Combination of a pair of RF-driven traveling wave stripline elements, quadrupole magnets, and a variable beam aperture enables operation of the instrument in (1) continuous waveform (CW) mode as though the instrument was never modified (i.e. convention TEM operation mode, where the electrons from the emission cathode randomly arrive at the sample without resolvable time information), (2) stroboscopic (pump-probe) mode, and (3) pulsed beam mode for dose rate sensitive materials. To assess the effect of a pulsed beam on image quality, we examined Au nanoparticles using bright field, high-resolution TEM imaging and selected area diffraction in both continuous and pulsed-beam mode. In comparison of conventional TEMs, the add-on beam pulser enables the observation of ultrafast dynamic behavior in materials that are reversible under synchronized excitation.

Toward Electrochemical Studies on the Nanometer and Atomic Scales: Progress, Challenges, and Opportunities

Electrochemical reactions and ionic transport underpin the operation of a broad range of devices and applications, from energy storage and conversion to information technologies, as well as biochemical processes, artificial muscles, and soft actuators. Understanding the mechanisms governing function of these applications requires probing local electrochemical phenomena on the relevant time and length scales. Here, we discuss the challenges and opportunities for extending electrochemical characterization probes to the nanometer and ultimately atomic scales, including challenges in down-scaling classical methods, the emergence of novel probes enabled by nanotechnology and based on emergent physics and chemistry of nanoscale systems, and the integration of local data into macroscopic models. Scanning probe microscopy (SPM) methods based on strain detection, potential detection, and hysteretic current measurements are discussed. We further compare SPM to electron beam probes and discuss the applicability of electron beam methods to probe local electrochemical behavior on the mesoscopic and atomic levels. Similar to a SPM tip, the electron beam can be used both for observing behavior and as an active electrode to induce reactions. We briefly discuss new challenges and opportunities for conducting fundamental scientific studies, matter patterning, and atomic manipulation arising in this context.

Single-shot real-time sub-nanosecond electron imaging aided by compressed sensing: Analytical modeling and simulation

Bringing ultrafast (nanosecond and below) temporal resolution to transmission electron microscopy (TEM) has historically been challenging. Despite significant recent progress in this direction, it remains difficult to achieve sub-nanosecond temporal resolution with a single electron pulse, in real-time (i.e., duration in which the event occurs) imaging. To address this limitation, here, we propose a methodology that combines laser-assisted TEM with computational imaging methodologies based on compressed sensing (CS). In this technique, a two-dimensional (2D) transient event [i.e. (x,y) frames that vary in time] is recorded through a CS paradigm, which consists of spatial encoding, temporal shearing via streaking, and spatiotemporal integration of an electron pulse. The 2D image generated on a camera is used to reconstruct the datacube of the ultrafast event, with two spatial and one temporal dimensions, via a CS-based image reconstruction algorithm. Using numerical simulation, we find that the reconstructed results are in good agreement with the ground truth, which demonstrates the applicability of CS-based computational imaging methodologies to laser-assisted TEM. Our proposed method, complementing the existing ultrafast stroboscopic and nanosecond single-shot techniques, opens up the possibility for single-shot, real-time, spatiotemporal imaging of irreversible structural phenomena with sub-nanosecond temporal resolution.

Locating and Visualizing Crystals for X-Ray Diffraction Experiments

Macromolecular crystallography has advanced from using macroscopic crystals, which might be >1 mm on a side, to crystals that are essentially invisible to the naked eye, or even under a standard laboratory microscope. As crystallography requires recognizing crystals when they are produced, and then placing them in an X-ray, electron, or neutron beam, this provides challenges, particularly in the case of advanced X-ray sources, where beams have very small cross sections and crystals may be vanishingly small. Methods for visualizing crystals are reviewed here, and examples of different types of cases are presented, including: standard crystals, crystals grown in mesophase, in situ crystallography, and crystals grown for X-ray Free Electron Laser or Micro Electron Diffraction experiments. As most techniques have limitations, it is desirable to have a range of complementary techniques available to identify and locate crystals. Ideally, a given technique should not cause sample damage, but sometimes it is necessary to use techniques where damage can only be minimized. For extreme circumstances, the act of probing location may be coincident with collecting X-ray diffraction data. Future challenges and directions are also discussed.

Detecting structural variances of Co3O4 catalysts by controlling beam-induced sample alterations in the vacuum of a transmission electron microscope

This article summarizes core aspects of beam-sample interactions in research that aims at exploiting the ability to detect single atoms at atomic resolution by mid-voltage transmission electron microscopy. Investigating the atomic structure of catalytic Co₃O₄ nanocrystals underscores how indispensable it is to rigorously control electron dose rates and total doses to understand native material properties on this scale. We apply in-line holography with variable dose rates to achieve this goal. Genuine object structures can be maintained if dose rates below ~100 e/Ųs are used and the contrast required for detection of single atoms is generated by capturing large image series. Threshold doses for the detection of single atoms are estimated. An increase of electron dose rates and total doses to common values for high resolution imaging of solids stimulates object excitations that restructure surfaces, interfaces, and defects and cause grain reorientation or growth. We observe a variety of previously unknown atom configurations in surface proximity of the Co₃O₄ spinel structure. These are hidden behind broadened diffraction patterns in reciprocal space but become visible in real space by solving the phase problem. An exposure of the Co₃O₄ spinel structure to water vapor or other gases induces drastic structure alterations that can be captured in this manner.

Big, Deep, and Smart Data in Scanning Probe Microscopy

Scanning probe microscopy (SPM) techniques have opened the door to nanoscience and nanotechnology by enabling imaging and manipulation of the structure and functionality of matter at nanometer and atomic scales. Here, we analyze the scientific discovery process in SPM by following the information flow from the tip-surface junction, to knowledge adoption by the wider scientific community. We further discuss the challenges and opportunities offered by merging SPM with advanced data mining, visual analytics, and knowledge discovery technologies.