Compressed Sensing and Electron Microscopy

High-speed 4-dimensional scanning transmission electron microscopy using compressive sensing techniques

Here we show that compressive sensing allows 4-dimensional (4-D) STEM data to be obtained and accurately reconstructed with both high-speed and reduced electron fluence. The methodology needed to achieve these results compared to conventional 4-D approaches requires only that a random subset of probe locations is acquired from the typical regular scanning grid, which immediately generates both higher speed and the lower fluence experimentally. We also consider downsampling of the detector, showing that oversampling is inherent within convergent beam electron diffraction (CBED) patterns and that detector downsampling does not reduce precision but allows faster experimental data acquisition. Analysis of an experimental atomic resolution yttrium silicide dataset shows that it is possible to recover over 25 dB peak signal-to-noise ratio in the recovered phase using 0.3% of the total data. Lay abstract: Four-dimensional scanning transmission electron microscopy (4-D STEM) is a powerful technique for characterizing complex nanoscale structures. In this method, a convergent beam electron diffraction pattern (CBED) is acquired at each probe location during the scan of the sample. This means that a 2-dimensional signal is acquired at each 2-D probe location, equating to a 4-D dataset. Despite the recent development of fast direct electron detectors, some capable of 100kHz frame rates, the limiting factor for 4-D STEM is acquisition times in the majority of cases, where cameras will typically operate on the order of 2kHz. This means that a raster scan containing 256^2 probe locations can take on the order of 30s, approximately 100-1000 times longer than a conventional STEM imaging technique using monolithic radial detectors. As a result, 4-D STEM acquisitions can be subject to adverse effects such as drift, beam damage, and sample contamination. Recent advances in computational imaging techniques for STEM have allowed for faster acquisition speeds by way of acquiring only a random subset of probe locations from the field of view. By doing this, the acquisition time is significantly reduced, in some cases by a factor of 10-100 times. The acquired data is then processed to fill-in or inpaint the missing data, taking advantage of the inherently low-complex signals which can be linearly combined to recover the information. In this work, similar methods are demonstrated for the acquisition of 4-D STEM data, where only a random subset of CBED patterns are acquired over the raster scan. We simulate the compressive sensing acquisition method for 4-D STEM and present our findings for a variety of analysis techniques such as ptychography and differential phase contrast. Our results show that acquisition times can be significantly reduced on the order of 100-300 times, therefore improving existing frame rates, as well as further reducing the electron fluence beyond just using a faster camera.

Enhancing performance of electron holography with mathematical and machine learning-based denoising techniques

Electron holography is a useful tool for analyzing functional properties, such as electromagnetic fields and strains of materials and devices. The performance of electron holography is limited by the 'shot noise' inherent in electron micrographs (holograms), which are composed of a finite number of electrons. A promising approach for addressing this issue is to use mathematical and machine learning-based image-processing techniques for hologram denoising. With the advancement of information science, denoising methods have become capable of extracting signals that are completely buried in noise, and they are being applied to electron microscopy, including electron holography. However, these advanced denoising methods are complex and have many parameters to be tuned; therefore, it is necessary to understand their principles in depth and use them carefully. Herein, we present an overview of the principles and usage of sparse coding, the wavelet hidden Markov model and tensor decomposition, which have been applied to electron holography. We also present evaluation results for the denoising performance of these methods obtained through their application to simulated and experimentally recorded holograms. Our analysis, review and comparison of the methods clarify the impact of denoising on electron holography research.

Sub-Sampled Imaging for STEM: Maximising Image Speed, Resolution and Precision Through Reconstruction Parameter Refinement

Sub-sampling during image acquisition in scanning transmission electron microscopy (STEM) has been shown to provide a means to increase the overall speed of acquisition while at the same time providing an efficient means to control the dose, dose rate and dose overlap delivered to the sample. In this paper, we discuss specifically the parameters used to reconstruct sub-sampled images and highlight their effect on inpainting using the beta-process factor analysis (BPFA) methodology. The selection of the main control parameters can have a significant effect on the resolution, precision and sensitivity of the final inpainted images, and here we demonstrate a method by which these parameters can be optimised for any image in STEM. As part of this work, we also provide a link to open source code and a tutorial on its use, whereby these parameters can be tested for any datasets. When coupled with the hardware necessary to rapidly sub-sample images in STEM, this approach can have significant implications for imaging beam sensitive materials and dynamic processes.

Computational Evaluation of Sparse Coding on off-axis Electron Holograms: Comparison Between Charge-Coupled Device and Direct-Detection Device Cameras

The effectiveness of sparse coding for image inpainting and denoising of off-axis electron holograms was examined computationally based on hologram simulations according to considerations of two types of electron detectors, namely, charge-coupled device (CCD) and direct-detection device (DDD) cameras. In this simulation, we used a simple-phase object with a phase step such as a semiconductor p-n junction and assumed that the holograms recorded by the CCD camera include shot noise, dark-current, and read-out noise, while those recorded by the DDD camera include only shot noise. Simulated holograms with various electron doses were sparsely coded. Even though interference fringes cannot be recognized in the simulated CCD and DDD holograms when subjected to electron doses (per pixel) equal to 1 and 0.01, respectively, both the corresponding sparse-coded holograms exhibit meaningful interference fringes. We demonstrate that a combination of the DDD camera and sparse coding reduces the requisite dose used to obtain holograms to values less than one-thousandth compared with the CCD camera without image postprocessing. This combination is expected to generate lower-dose and/or higher-speed electron holography.

Simulation-Trained Sparse Coding for High-Precision Phase Imaging in Low-Dose Electron Holography

We broaden the applicability of sparse coding, a machine learning method, to low-dose electron holography by using simulated holograms for learning and validation processes. The holograms, with shot noise, are prepared to generate a model, or a dictionary, that includes basic features representing interference fringes. The dictionary is applied to sparse representations of other simulated holograms with various signal-to-noise ratios (SNRs). Results demonstrate that this approach successfully removes noise for holograms with an extremely small SNR of 0.10, and that the denoised holograms provide the accurate phase distribution. Furthermore, this study demonstrates that the dictionary learned from the simulated holograms can be applied to denoising of experimental holograms of a p-n junction specimen recorded with different exposure times. The results indicate that the simulation-trained sparse coding is suitable for use over a wide range of imaging conditions, in particular for observing electron beam-sensitive materials.

Laser-free GHz stroboscopic transmission electron microscope: Components, system integration, and practical considerations for pump-probe measurements

A 300 keV transmission electron microscope was modified to produce broadband pulsed beams that can be, in principle, between 40 MHz and 12 GHz, corresponding to temporal resolution in the nanosecond to picosecond range without an excitation laser. The key enabling technology is a pair of phase-matched modulating and de-modulating traveling wave metallic comb striplines (pulsers). An initial temporal resolution of 30 ps was achieved with a strobe frequency of 6.0 GHz. The placement of the pulsers, mounted immediately below the gun, allows for preservation of all optical configurations, otherwise available to the unmodified instrument, and therefore makes such a post-modified instrument for dual-use, i.e., both pulsed-beam mode (i.e., stroboscopic time-resolved) and conventional continuous waveform mode. In this article, we describe the elements inserted into the beam path, challenges encountered during integration with an in-service microscope, and early results from an electric-field-driven pump-probe experiment. We conclude with ideas for making this class of instruments broadly applicable for examining cyclical and repeatable phenomena.

Electrostatic subframing and compressive-sensing video in transmission electron microscopy

We present kilohertz-scale video capture rates in a transmission electron microscope, using a camera normally limited to hertz-scale acquisition. An electrostatic deflector rasters a discrete array of images over a large camera, decoupling the acquisition time per subframe from the camera readout time. Total-variation regularization allows features in overlapping subframes to be correctly placed in each frame. Moreover, the system can be operated in a compressive-sensing video mode, whereby the deflections are performed in a known pseudorandom sequence. Compressive sensing in effect performs data compression before the readout, such that the video resulting from the reconstruction can have substantially more total pixels than that were read from the camera. This allows, for example, 100 frames of video to be encoded and reconstructed using only 15 captured subframes in a single camera exposure. We demonstrate experimental tests including laser-driven melting/dewetting, sintering, and grain coarsening of nanostructured gold, with reconstructed video rates up to 10 kHz. The results exemplify the power of the technique by showing that it can be used to study the fundamentally different temporal behavior for the three different physical processes. Both sintering and coarsening exhibited self-limiting behavior, whereby the process essentially stopped even while the heating laser continued to strike the material. We attribute this to changes in laser absorption and to processes inherent to thin-film coarsening. In contrast, the dewetting proceeded at a relatively uniform rate after an initial incubation time consistent with the establishment of a steady-state temperature profile.

Improved Three-Dimensional (3D) Resolution of Electron Tomograms Using Robust Mathematical Data-Processing Techniques

Electron tomography has become an essential tool for three-dimensional (3D) characterization of nanomaterials. In recent years, advances have been made in specimen preparation and mounting, acquisition geometries, and reconstruction algorithms. All of these components work together to optimize the resolution and clarity of an electron tomogram. However, one important component of the data-processing has received less attention: the 2D tilt series alignment. This is challenging for a number of reasons, namely because the nature of the data sets and the need to be coherently aligned over the full range of angles. An inaccurate alignment may be difficult to identify, yet can significantly limit the final 3D resolution. In this work, we present an improved center-of-mass alignment model that allows us to overcome discrepancies from unwanted objects that enter the imaging area throughout the tilt series. In particular, we develop an approach to overcome changes in the total mass upon rotation of the imaging area. We apply our approach to accurately recover small Pt nanoparticles embedded in a zeolite that may otherwise go undetected both in the 2D microscopy images and the 3D reconstruction. In addition to this, we highlight the particular effectiveness of the compressed sensing methods with this data set.

A Survey of the Use of Iterative Reconstruction Algorithms in Electron Microscopy

One of the key steps in Electron Microscopy is the tomographic reconstruction of a three-dimensional (3D) map of the specimen being studied from a set of two-dimensional (2D) projections acquired at the microscope. This tomographic reconstruction may be performed with different reconstruction algorithms that can be grouped into several large families: direct Fourier inversion methods, back-projection methods, Radon methods, or iterative algorithms. In this review, we focus on the latter family of algorithms, explaining the mathematical rationale behind the different algorithms in this family as they have been introduced in the field of Electron Microscopy. We cover their use in Single Particle Analysis (SPA) as well as in Electron Tomography (ET).

Atomic-Scale Analytical Tomography

The concept of atomic-scale tomography has been proposed in the past decade as a technique that could deliver the position of all atoms with high precision and their elemental (isotopic) identity. The technique was never intended to be limited to merely structural information and there is clearly a rich array of additional analytical information that can be brought to bear on such tomographs. In this paper, some of these types of information are considered and the implications are explored. The fuller realm of this analytical and structural information may be called atomic-scale analytical tomography.

Biological application of Compressed Sensing Tomography in the Scanning Electron Microscope

The three-dimensional tomographic reconstruction of a biological sample, namely collagen fibrils in human dermal tissue, was obtained from a set of projection-images acquired in the Scanning Electron Microscope. A tailored strategy for the transmission imaging mode was implemented in the microscope and proved effective in acquiring the projections needed for the tomographic reconstruction. Suitable projection alignment and Compressed Sensing formulation were used to overcome the limitations arising from the experimental acquisition strategy and to improve the reconstruction of the sample. The undetermined problem of structure reconstruction from a set of projections, limited in number and angular range, was indeed supported by exploiting the sparsity of the object projected in the electron microscopy images. In particular, the proposed system was able to preserve the reconstruction accuracy even in presence of a significant reduction of experimental projections.

Suppressing Electron Exposure Artifacts: An Electron Scanning Paradigm with Bayesian Machine Learning

Electron microscopy of biological, polymeric, and other beam-sensitive structures is often hampered by deleterious electron beam interactions. In fact, imaging of such beam-sensitive materials is limited by the allowable radiation dosage rather that capabilities of the microscope itself, which has been compounded by the availability of high brightness electron sources. Reducing dwell times to overcome dose-related artifacts, such as radiolysis and electrostatic charging, is challenging due to the inherently low contrast in imaging of many such materials. These challenges are particularly exacerbated during dynamic time-resolved, fluidic cell imaging, or three-dimensional tomographic reconstruction-all of which undergo additional dosage. Thus, there is a pressing need for the development of techniques to produce high-quality images at ever lower electron doses. In this contribution, we demonstrate direct dose reduction and suppression of beam-induced artifacts through under-sampling pixels, by as much as 80% reduction in dosage, using a commercial scanning electron microscope with an electrostatic beam blanker and a dictionary learning in-painting algorithm. This allows for multiple sparse recoverable images to be acquired at the cost of one fully sampled image. We believe this approach may open new ways to conduct imaging, which otherwise require compromising beam current and/or exposure conditions.

Compressed Sensing Electron Tomography for Determining Biological Structure

There has been growing interest in applying compressed sensing (CS) theory and practice to reconstruct 3D volumes at the nanoscale from electron tomography datasets of inorganic materials, based on known sparsity in the structure of interest. Here we explore the application of CS for visualizing the 3D structure of biological specimens from tomographic tilt series acquired in the scanning transmission electron microscope (STEM). CS-ET reconstructions match or outperform commonly used alternative methods in full and undersampled tomogram recovery, but with less significant performance gains than observed for the imaging of inorganic materials. We propose that this disparity stems from the increased structural complexity of biological systems, as supported by theoretical CS sampling considerations and numerical results in simulated phantom datasets. A detailed analysis of the efficacy of CS-ET for undersampled recovery is therefore complicated by the structure of the object being imaged. The numerical nonlinear decoding process of CS shares strong connections with popular regularized least-squares methods, and the use of such numerical recovery techniques for mitigating artifacts and denoising in reconstructions of fully sampled datasets remains advantageous. This article provides a link to the software that has been developed for CS-ET reconstruction of electron tomographic data sets.

Applying compressive sensing to TEM video: a substantial frame rate increase on any camera

One of the main limitations of imaging at high spatial and temporal resolution during in-situ transmission electron microscopy (TEM) experiments is the frame rate of the camera being used to image the dynamic process. While the recent development of direct detectors has provided the hardware to achieve frame rates approaching 0.1 ms, the cameras are expensive and must replace existing detectors. In this paper, we examine the use of coded aperture compressive sensing (CS) methods to increase the frame rate of any camera with simple, low-cost hardware modifications. The coded aperture approach allows multiple sub-frames to be coded and integrated into a single camera frame during the acquisition process, and then extracted upon readout using statistical CS inversion. Here we describe the background of CS and statistical methods in depth and simulate the frame rates and efficiencies for in-situ TEM experiments. Depending on the resolution and signal/noise of the image, it should be possible to increase the speed of any camera by more than an order of magnitude using this approach.

Mathematics Subject Classification: (2010) 94A08 · 78A15