Low-dose phase retrieval of biological specimens using cryo-electron ptychography

Three-Dimensional Reconstruction-Characterization of Polymeric Membranes: A Review

Polymeric membranes serve as vital separation materials in diverse energy and environmental applications. A comprehensive understanding of three-dimensional (3D) structures of membranes is critical to performance evaluation and future design. Such quantitative 3D structural information is beyond the limit of most employed conventional two-dimentional characterization techniques such as scanning electron microscopy. In this review, we summarize eight types of 3D reconstruction-characterization techniques for membrane materials. Originated from life and materials science, these techniques have been optimized to reveal the 3D structures of membrane materials in the separation field. We systematically introduce the theories of each technique, summarize the sample preparation procedures developed for membrane materials, and demonstrate step-by-step data processing, including 3D model reconstruction and subsequent characterization. Representative case studies are introduced to show the progress of this field and how technical challenges have been overcome over the years. In the end, we share our perspectives and believe that this review can serve as a useful reference for 3D reconstruction-characterization techniques developed for membrane materials.

Retrieval of Phase Information from Low-Dose Electron Microscopy Experiments: Are We at the Limit Yet?

The challenge of imaging low-density objects in an electron microscope without causing beam damage is significant in modern transmission electron microscopy. This is especially true for life science imaging, where the sample, rather than the instrument, still determines the resolution limit. Here, we explore whether we have to accept this or can progress further in this area. To do this, we use numerical simulations to see how much information we can obtain from a weak phase object at different electron doses. Starting from a model with four phase values, we compare Zernike phase contrast with measuring diffracted intensity under multiple random phase illuminations to solve the inverse problem. Our simulations have shown that diffraction-based methods perform better than the Zernike method, as we have found and addressed a normalization issue that, in some other studies, led to an overly optimistic representation of the Zernike setup. We further validated this using more realistic 2D objects and found that random phase illuminated diffraction can be up to five times more efficient than an ideal Zernike implementation. These findings suggest that diffraction-based methods could be a promising approach for imaging beam-sensitive materials and that current low-dose imaging methods are not yet at the quantum limit.

Atomically resolved imaging of radiation-sensitive metal-organic frameworks via electron ptychography

Electron ptychography, recognized as an ideal technique for low-dose imaging, consistently achieves deep sub-angstrom resolution at electron doses of several thousand electrons per square angstrom (e-/Å2) or higher. Despite its proven efficacy, the application of electron ptychography at even lower doses-necessary for materials highly sensitive to electron beams-raises questions regarding its feasibility and the attainable resolution under such stringent conditions. Herein, we demonstrate the implementation of near-atomic-resolution ( ~ 2 Å) electron ptychography reconstruction at electron doses as low as ~100 e-/Å2, for metal-organic frameworks (MOFs), which are known for their extreme sensitivity. The reconstructed images clearly resolve organic linkers, metal clusters, and even atomic columns within these clusters, while unravelling various local structural features in MOFs, including missing linkers, extra clusters, and surface termination modes. By combining the findings from simulations and experiments, we have identified that employing a small convergence semi-angle during data acquisition is crucial for effective iterative ptychographic reconstruction under such low-dose conditions. This important insight advances our understanding of the rapidly evolving electron ptychography technique and provides a novel approach to high-resolution imaging of various sensitive materials.

Simulation Study of Low-Dose 4D-STEM Phase Contrast Techniques at the Nanoscale in SEM

Phase contrast imaging is well-suited for studying weakly scattering samples. Its strength lies in its ability to measure how the phase of the electron beam is affected by the sample, even when other imaging techniques yield low contrast. In this study, we explore via simulations two phase contrast techniques: integrated center of mass (iCOM) and ptychography, specifically using the extended ptychographical iterative engine (ePIE). We simulate the four-dimensional scanning transmission electron microscopy (4D-STEM) datasets for specific parameters corresponding to a scanning electron microscope (SEM) with an immersive objective and a given pixelated detector. The performance of these phase contrast techniques is analyzed using a contrast transfer function. Simulated datasets from a sample consisting of graphene sheets and carbon nanotubes are used for iCOM and ePIE reconstructions for two aperture sizes and two electron doses. We highlight the influence of aperture size, showing that for a smaller aperture, the radiation dose is spent mostly on larger sample features, which may aid in imaging sensitive samples while minimizing radiation damage.

Computational microscopy with coherent diffractive imaging and ptychography

Microscopy and crystallography are two essential experimental methodologies for advancing modern science. They complement one another, with microscopy typically relying on lenses to image the local structures of samples, and crystallography using diffraction to determine the global atomic structure of crystals. Over the past two decades, computational microscopy, encompassing coherent diffractive imaging (CDI) and ptychography, has advanced rapidly, unifying microscopy and crystallography to overcome their limitations. Here, I review the innovative developments in CDI and ptychography, which achieve exceptional imaging capabilities across nine orders of magnitude in length scales, from resolving atomic structures in materials at sub-ångstrom resolution to quantitative phase imaging of centimetre-sized tissues, using the same principle and similar computational algorithms. These methods have been applied to determine the 3D atomic structures of crystal defects and amorphous materials, visualize oxygen vacancies in high-temperature superconductors and capture ultrafast dynamics. They have also been used for nanoscale imaging of magnetic, quantum and energy materials, nanomaterials, integrated circuits and biological specimens. By harnessing fourth-generation synchrotron radiation, X-ray-free electron lasers, high-harmonic generation, electron microscopes, optical microscopes, cutting-edge detectors and deep learning, CDI and ptychography are poised to make even greater contributions to multidisciplinary sciences in the years to come.

Advanced Compressive Sensing and Dynamic Sampling for 4D-STEM Imaging of Interfaces

Interfaces in energy materials and devices often involve beam-sensitive materials such as fast ionic, soft, or liquid phases. 4D scanning transmission electron microscopy (4D-STEM) offers insights into local lattice, strain charge, and field distributions, but faces challenges in analyzing beam-sensitive interfaces at high spatial resolutions. Here, a 4D-STEM compressive sensing algorithm is introduced that significantly reduces data acquisition time and electron dose. This method autonomously allocates probe positions on interfaces and reconstructs missing information from datasets acquired via dynamic sampling. This algorithm allows for the integration of various scanning schemes and electron probe conditions to optimize data integrity. Its data reconstruction employs a neural network and an autoencoder to correlate diffraction pattern features with measured properties, significantly reducing training costs. The accuracy of the reconstructed 4D-STEM datasets is verified using a combination of explicitly and implicitly trained parameters from atomic resolution datasets. This method is broadly applicable for 4D-STEM imaging of any local features of interest and will be available on GitHub upon publication.

Advancing Single-Particle Analysis in Synthetic Chemical Systems: A Forward-Looking Discussion

Single-particle analysis (SPA) is a fundamental method of cryo-electron microscopy developed to resolve the structures of biological macromolecules. This method has seen significant success in structural biology, yet its potential applications in synthetic chemical systems remain underexplored. In this perspective article, SPA and associated electron microscopy techniques are first briefly introduced. It is then proposed that SPA is well-suited for structural analysis of chemical systems where discrete, identical macromolecules can be readily obtained. Applicable systems include various clusters such as coinage metal clusters, metal-oxo/sulfur clusters, metal-organic clusters, and supramolecular compounds like coordination cages and metallo-supramolecular cages. When high-quality large single crystals are unattainable, SPA provides an alternative method for determining their structures. Beyond these end products, it is suggested that SPA can be instrumental in studying synthetic intermediates of materials with specific building units, such as metal-organic frameworks and zeolites. Given that various intermediates coexist in the reaction system, a purification step is necessary before conducting SPA, which can be facilitated by soft-landing electrospray ionization mass spectrometry.

Ultra-fast Digital DPC Yielding High Spatio-temporal Resolution for Low-Dose Phase Characterization

In the scanning transmission electron microscope, both phase imaging of beam-sensitive materials and characterization of a material's functional properties using in situ experiments are becoming more widely available. As the practicable scan speed of 4D-STEM detectors improves, so too does the temporal resolution achievable for both differential phase contrast (DPC) and ptychography. However, the read-out burden of pixelated detectors, and the size of the gigabyte to terabyte sized data sets, remain a challenge for both temporal resolution and their practical adoption. In this work, we combine ultra-fast scan coils and detector signal digitization to show that a high-fidelity DPC phase reconstruction can be achieved from an annular segmented detector. Unlike conventional analog data phase reconstructions from digitized DPC-segment images yield reliable data, even at the fastest scan speeds. Finally, dose fractionation by fast scanning and multi-framing allows for postprocess binning of frame streams to balance signal-to-noise ratio and temporal resolution for low-dose phase imaging for in situ experiments.

Ptychographic phase retrieval via a deep-learning-assisted iterative algorithm

Ptychography is a powerful computational imaging technique with microscopic imaging capability and adaptability to various specimens. To obtain an imaging result, it requires a phase-retrieval algorithm whose performance directly determines the imaging quality. Recently, deep neural network (DNN)-based phase retrieval has been proposed to improve the imaging quality from the ordinary model-based iterative algorithms. However, the DNN-based methods have some limitations because of the sensitivity to changes in experimental conditions and the difficulty of collecting enough measured specimen images for training the DNN. To overcome these limitations, a ptychographic phase-retrieval algorithm that combines model-based and DNN-based approaches is proposed. This method exploits a DNN-based denoiser to assist an iterative algorithm like ePIE in finding better reconstruction images. This combination of DNN and iterative algorithms allows the measurement model to be explicitly incorporated into the DNN-based approach, improving its robustness to changes in experimental conditions. Furthermore, to circumvent the difficulty of collecting the training data, it is proposed that the DNN-based denoiser be trained without using actual measured specimen images but using a formula-driven supervised approach that systemically generates synthetic images. In experiments using simulation based on a hard X-ray ptychographic measurement system, the imaging capability of the proposed method was evaluated by comparing it with ePIE and rPIE. These results demonstrated that the proposed method was able to reconstruct higher-spatial-resolution images with half the number of iterations required by ePIE and rPIE, even for data with low illumination intensity. Also, the proposed method was shown to be robust to its hyperparameters. In addition, the proposed method was applied to ptychographic datasets of a Simens star chart and ink toner particles measured at SPring-8 BL24XU, which confirmed that it can successfully reconstruct images from measurement scans with a lower overlap ratio of the illumination regions than is required by ePIE and rPIE.

Low-dose cryo-electron ptychography of proteins at sub-nanometer resolution

Cryo-transmission electron microscopy (cryo-EM) of frozen hydrated specimens is an efficient method for the structural analysis of purified biological molecules. However, cryo-EM and cryo-electron tomography are limited by the low signal-to-noise ratio (SNR) of recorded images, making detection of smaller particles challenging. For dose-resilient samples often studied in the physical sciences, electron ptychography - a coherent diffractive imaging technique using 4D scanning transmission electron microscopy (4D-STEM) - has recently demonstrated excellent SNR and resolution down to tens of picometers for thin specimens imaged at room temperature. Here we apply 4D-STEM and ptychographic data analysis to frozen hydrated proteins, reaching sub-nanometer resolution 3D reconstructions. We employ low-dose cryo-EM with an aberration-corrected, convergent electron beam to collect 4D-STEM data for our reconstructions. The high frame rate of the electron detector allows us to record large datasets of electron diffraction patterns with substantial overlaps between the interaction volumes of adjacent scan positions, from which the scattering potentials of the samples are iteratively reconstructed. The reconstructed micrographs show strong SNR enabling the reconstruction of the structure of apoferritin protein at up to 5.8 Å resolution. We also show structural analysis of the Phi92 capsid and sheath, tobacco mosaic virus, and bacteriorhodopsin at slightly lower resolutions.

Ghostbuster: A phase retrieval diffraction tomography algorithm for cryo-EM

Ewald sphere curvature correction, which extends beyond the projection approximation, stretches the shallow depth of field in cryo-EM reconstructions of thick particles. Here we show that even for previously assumed thin particles, reconstruction artifacts which we refer to as ghosts can appear. By retrieving the lost phases of the electron exitwaves and accounting for the first Born approximation scattering within the particle, we show that these ghosts can be effectively eliminated. Our simulations demonstrate how such ghostbusting can improve reconstructions as compared to existing state-of-the-art software. Like ptychographic cryo-EM, our Ghostbuster algorithm uses phase retrieval to improve reconstructions, but unlike the former, we do not need to modify the existing data acquisition pipelines.

Optimizing Contrast in Automated 4D STEM Cryotomography

4D STEM is an emerging approach to electron microscopy. While it was developed principally for high-resolution studies in materials science, the possibility to collect the entire transmitted flux makes it attractive for cryomicroscopy in application to life science and radiation-sensitive materials where dose efficiency is of utmost importance. We present a workflow to acquire tomographic tilt series of 4D STEM data sets using a segmented diode and an ultrafast pixelated detector, demonstrating the methods using a specimen of a T4 bacteriophage. Full integration with the SerialEM platform conveniently provides all the tools for grid navigation and automation of the data collection. Scripts are provided to convert the raw data to mrc format files and further to generate a variety of modes representing both scattering and phase contrasts, including incoherent and annular bright field, integrated center of mass, and parallax decomposition of a simulated integrated differential phase contrast. Principal component analysis of virtual annular detectors proves particularly useful, and axial contrast is improved by 3D deconvolution with an optimized point spread function. Contrast optimization enables visualization of irregular features such as DNA strands and thin filaments of the phage tails, which would be lost upon averaging or imposition of an inappropriate symmetry.

Atomic-level imaging of beam-sensitive COFs and MOFs by low-dose electron microscopy

Electron microscopy, an important technique that allows for the precise determination of structural information with high spatiotemporal resolution, has become indispensable in unravelling the complex relationships between material structure and properties ranging from mesoscale morphology to atomic arrangement. However, beam-sensitive materials, particularly those comprising organic components such as metal-organic frameworks (MOFs) and covalent organic frameworks (COFs), would suffer catastrophic damage from the high energy electrons, hindering the determination of atomic structures. A low-dose approach has arisen as a possible solution to this problem based on the integration of advancements in several aspects: electron optical system, detector, image processing, and specimen preservation. This article summarizes the transmission electron microscopy characterization of MOFs and COFs, including local structures, host-guest interactions, and interfaces at the atomic level. Revolutions in advanced direct electron detectors, algorithms in image acquisition and processing, and emerging methodology for high quality low-dose imaging are also reviewed. Finally, perspectives on the future development of electron microscopy methodology with the support of computer science are presented.

Phase offset method of ptychographic contrast reversal correction

The contrast transfer function of direct ptychography methods such as the single side band (SSB) method are single signed, yet these methods still sometimes exhibit contrast reversals, most often where the projected potentials are strong. In thicker samples central focusing often provides the best ptychographic contrast as this leads to defocus variations within the sample canceling out. However focusing away from the entrance surface is often undesirable as this degrades the annular dark field (ADF) signal. Here we discuss how phase wrap asymptotes in the frequency response of SSB ptychography give rise to contrast reversals, without the need for dynamical scattering, and how these can be counteracted by manipulating the phases such that the asymptotes are either shifted to higher frequencies or damped via amplitude modulation. This is what enables post collection defocus correction of contrast reversals. However, the phase offset method of counteracting contrast reversals we introduce here is generally found to be superior to post collection application of defocus, with greater reliability and generally stronger contrast. Importantly, the phase offset method also works for thin and thick samples where central focusing does not. Finally, the independence of the method from focus is useful for optical sectioning involving ptychography, improving interpretability by better disentangling the effects of strong potentials and focus.

Modern approaches to improving phase contrast electron microscopy

Although defocus can be used to generate partial phase contrast in transmission electron microscope images, cryo-electron microscopy (cryo-EM) can be further improved by the development of phase plates which increase contrast by applying a phase shift to the unscattered part of the electron beam. Many approaches have been investigated, including the ponderomotive interaction between light and electrons. We review the recent successes achieved with this method in high-resolution, single-particle cryo-EM. We also review the status of using pulsed or near-field enhanced laser light as alternatives, along with approaches that use scanning transmission electron microscopy (STEM) with a segmented detector rather than a phase plate.

Achieving sub-0.5-angstrom-resolution ptychography in an uncorrected electron microscope

Subangstrom resolution has long been limited to aberration-corrected electron microscopy, where it is a powerful tool for understanding the atomic structure and properties of matter. Here, we demonstrate electron ptychography in an uncorrected scanning transmission electron microscope (STEM) with deep subangstrom spatial resolution down to 0.44 angstroms, exceeding the conventional resolution of aberration-corrected tools and rivaling their highest ptychographic resolutions​. Our approach, which we demonstrate on twisted two-dimensional materials in a widely available commercial microscope, far surpasses prior ptychographic resolutions (1 to 5 angstroms) of uncorrected STEMs. We further show how geometric aberrations can create optimized, structured beams for dose-efficient electron ptychography. Our results demonstrate that expensive aberration correctors are no longer required for deep subangstrom resolution.

Temporal structured illumination and vision-transformer enables large field-of-view binary snapshot ptychography

Ptychography, a widely used computational imaging method, generates images by processing coherent interference patterns scattered from an object of interest. In order to capture scenes with large field-of-view (FoV) and high spatial resolution simultaneously in a single shot, we propose a temporal-compressive structured-light Ptychography system. A novel three-step reconstruction algorithm composed of multi-frame spectra reconstruction, phase retrieval, and multi-frame image stitching is developed, where we employ the emerging Transformer-based network in the first step. Experimental results demonstrate that our system can expand the FoV by 20× without losing spatial resolution. Our results offer huge potential for enabling lensless imaging of molecules with large FoV as well as high spatial-temporal resolutions. We also notice that due to the loss of low-intensity information caused by the compressed sensing process, our method so far is only applicable to binary targets.

Design of Electrostatic Aberration Correctors for Scanning Transmission Electron Microscopy

In a scanning transmission electron microscope (STEM), producing a high-resolution image generally requires an electron beam focused to the smallest point possible. However, the magnetic lenses used to focus the beam are unavoidably imperfect, introducing aberrations that limit resolution. Modern STEMs overcome this by using hardware aberration correctors comprised of many multipole elements, but these devices are complex, expensive, and can be difficult to tune. We demonstrate a design for an electrostatic phase plate that can act as an aberration corrector. The corrector is comprised of annular segments, each of which is an independent two-terminal device that can apply a constant or ramped phase shift to a portion of the electron beam. We show the improvement in image resolution using an electrostatic corrector. Engineering criteria impose that much of the beam within the probe-forming aperture be blocked by support bars, leading to large probe tails for the corrected probe that sample the specimen beyond the central lobe. We also show how this device can be used to create other STEM beam profiles such as vortex beams and probes with a high degree of phase diversity, which improve information transfer in ptychographic reconstructions.

An efficient ptychography reconstruction strategy through fine-tuning of large pre-trained deep learning model

With pre-trained large models and their associated fine-tuning paradigms being constantly applied in deep learning, the performance of large models achieves a dramatic boost, mostly owing to the improvements on both data quantity and quality. Next-generation synchrotron light sources offer ultra-bright and highly coherent X-rays, which are becoming one of the largest data sources for scientific experiments. As one of the most data-intensive scanning-based imaging methodologies, ptychography produces an immense amount of data, making the adoption of large deep learning models possible. Here, we introduce and refine the architecture of a neural network model to improve the reconstruction performance, through fine-tuning large pre-trained model using a variety of datasets. The pre-trained model exhibits remarkable generalization capability, while the fine-tuning strategy enhances the reconstruction quality. We anticipate this work will contribute to the advancement of deep learning methods in ptychography, as well as in broader coherent diffraction imaging methodologies in future.

Solving complex nanostructures with ptychographic atomic electron tomography

Transmission electron microscopy (TEM) is essential for determining atomic scale structures in structural biology and materials science. In structural biology, three-dimensional structures of proteins are routinely determined from thousands of identical particles using phase-contrast TEM. In materials science, three-dimensional atomic structures of complex nanomaterials have been determined using atomic electron tomography (AET). However, neither of these methods can determine the three-dimensional atomic structure of heterogeneous nanomaterials containing light elements. Here, we perform ptychographic electron tomography from 34.5 million diffraction patterns to reconstruct an atomic resolution tilt series of a double wall-carbon nanotube (DW-CNT) encapsulating a complex ZrTe sandwich structure. Class averaging the resulting tilt series images and subpixel localization of the atomic peaks reveals a Zr11Te50 structure containing a previously unobserved ZrTe2 phase in the core. The experimental realization of atomic resolution ptychographic electron tomography will allow for the structural determination of a wide range of beam-sensitive nanomaterials containing light elements.

Hollow Ptychography: Toward Simultaneous 4D Scanning Transmission Electron Microscopy and Electron Energy Loss Spectroscopy

With the recent development of high-acquisition-speed pixelated detectors, 4D scanning transmission electron microscopy (4D-STEM) is becoming routinely available in high-resolution electron microscopy. 4D-STEM acts as a "universal" method that provides local information on materials that is challenging to extract from bulk techniques. It extends conventional STEM imaging to include super-resolution techniques and to provide quantitative phase-based information, such as differential phase contrast, ptychography, or Bloch wave phase retrieval. However, an important missing factor is the chemical and bonding information provided by electron energy loss spectroscopy (EELS). 4D-STEM and EELS cannot currently be acquired simultaneously due to the overlapping geometry of the detectors. Here, the feasibility of modifying the detector geometry to overcome this challenge for bulk specimens is demonstrated, and the use of a partial or defective detector for ptycholgaphic structural imaging is explored. Results show that structural information beyond the diffraction-limit and chemical information from the material can be extracted together, resulting in simultaneous multi-modal measurements, adding the additional dimensions of spectral information to 4D datasets.

Charging of Vitreous Samples in Cryogenic Electron Microscopy Mitigated by Graphene

Cryogenic electron microscopy can provide high-resolution reconstructions of macromolecules embedded in a thin layer of ice from which atomic models can be built de novo. However, the interaction between the ionizing electron beam and the sample results in beam-induced motion and image distortion, which limit the attainable resolutions. Sample charging is one contributing factor of beam-induced motions and image distortions, which is normally alleviated by including part of the supporting conducting film within the beam-exposed region. However, routine data collection schemes avoid strategies whereby the beam is not in contact with the supporting film, whose rationale is not fully understood. Here we characterize electrostatic charging of vitreous samples, both in imaging and in diffraction mode. We mitigate sample charging by depositing a single layer of conductive graphene on top of regular EM grids. We obtained high-resolution single-particle analysis (SPA) reconstructions at 2 Å when the electron beam only irradiates the middle of the hole on graphene-coated grids, using data collection schemes that previously failed to produce sub 3 Å reconstructions without the graphene layer. We also observe that the SPA data obtained with the graphene-coated grids exhibit a higher b factor and reduced particle movement compared to data obtained without the graphene layer. This mitigation of charging could have broad implications for various EM techniques, including SPA and cryotomography, and for the study of radiation damage and the development of future sample carriers. Furthermore, it may facilitate the exploration of more dose-efficient, scanning transmission EM based SPA techniques.

Resolution of Virtual Depth Sectioning from Four-Dimensional Scanning Transmission Electron Microscopy

One approach to three-dimensional structure determination using the wealth of scattering data in four-dimensional (4D) scanning transmission electron microscopy (STEM) is the parallax method proposed by Ophus et al. (2019. Advanced phase reconstruction methods enabled by 4D scanning transmission electron microscopy, Microsc Microanal25, 10-11), which determines the scattering matrix and uses it to synthesize a virtual depth-sectioning reconstruction of the sample structure. Drawing on an equivalence with a hypothetical confocal imaging mode, we derive contrast transfer and point spread functions for this parallax method applied to weakly scattering objects, showing them identical to earlier depth-sectioning STEM modes when only bright field signal is used, but that improved depth resolution is possible if dark field signal can be used. Through a simulation-based study of doped Si, we show that this depth resolution is preserved for thicker samples, explore the impact of shot noise on the parallax reconstructions, discuss challenges to making use of dark field signal, and identify cases where the interpretation of the parallax reconstruction breaks down.

Synchronization of scanning probe and pixelated sensor for image-guided diffraction microscopy

A 4-dimensional modality of a scanning transmission electron microscope (4D-STEM) acquires diffraction images formed by a coherent and focused electron beam scanning the specimen. Newly developed ultrafast detectors offer a possibility to acquire high throughput diffraction patterns at each pixel of the scan, enabling rapid tilt series acquisition for 4D-STEM tomography. Here we present a solution to the problem of synchronizing the electron probe scan with the diffraction image acquisition, and demonstrate on a fast hybrid-pixel detector camera (ARINA, DECTRIS). Image-guided tracking and autofocus corrections are handled by the freely-available microscope-control software SerialEM, in conjunction with a high angle annular dark field (HAADF) image acquired simultaneously. The open source SavvyScan system offers a versatile set of scanning patterns, operated by commercially available multi-channel acquisition and signal generator computer cards (Spectrum Instrumentation GmbH). Images are recorded only within a sub-region of the total field, so as to avoid spurious data collection during flyback and/or acceleration periods in the scan. Hence, the trigger of the fast camera follows selected pulses from the scan generator clock gated according to the chosen scan pattern. Software and protocol are provided for gating the trigger pulses via a microcontroller (ST Microelectronics ARM Cortex). We demonstrate the system on a standard replica grating and by diffraction imaging of a ferritin specimen.

Cryogenic electron ptychographic single particle analysis with wide bandwidth information transfer

Advances in cryogenic transmission electron microscopy have revolutionised the determination of many macromolecular structures at atomic or near-atomic resolution. This method is based on conventional defocused phase contrast imaging. However, it has limitations of weaker contrast for small biological molecules embedded in vitreous ice, in comparison with cryo-ptychography, which shows increased contrast. Here we report a single-particle analysis based on the use of ptychographic reconstruction data, demonstrating that three dimensional reconstructions with a wide information transfer bandwidth can be recovered by Fourier domain synthesis. Our work suggests future applications in otherwise challenging single particle analyses, including small macromolecules and heterogeneous or flexible particles. In addition structure determination in situ within cells without the requirement for protein purification and expression may be possible.

Near-field multi-slice ptychography: quantitative phase imaging of optically thick samples with visible light and X-rays

Ptychography is a form of lens-free coherent diffractive imaging now used extensively in electron and synchrotron-based X-ray microscopy. In its near-field implementation, it offers a route to quantitative phase imaging at an accuracy and resolution competitive with holography, with the added advantages of extended field of view and blind deconvolution of the illumination beam profile from the sample image. In this paper we show how near-field ptychography can be combined with a multi-slice model, adding to this list of advantages the unique ability to recover high-resolution phase images of larger samples, whose thickness places them beyond the depth of field of alternative methods.

An integrated constrained gradient descent (iCGD) protocol to correct scan-positional errors for electron ptychography with high accuracy and precision

Correcting scan-positional errors is critical in achieving electron ptychography with both high resolution and high precision. This is a demanding and challenging task due to the sheer number of parameters that need to be optimized. For atomic-resolution ptychographic reconstructions, we found classical refining methods for scan positions not satisfactory due to the inherent entanglement between the object and scan positions, which can produce systematic errors in the results. Here, we propose a new protocol consisting of a series of constrained gradient descent (CGD) methods to achieve better recovery of scan positions. The central idea of these CGD methods is to utilize a priori knowledge about the nature of STEM experiments and add necessary constraints to isolate different types of scan positional errors during the iterative reconstruction process. Each constraint will be introduced with the help of simulated 4D-STEM datasets with known positional errors. Then the integrated constrained gradient decent (iCGD) protocol will be demonstrated using an experimental 4D-STEM dataset of the 1H-MoS₂ monolayer. We will show that the iCGD protocol can effectively address the errors of scan positions across the spectrum and help to achieve electron ptychography with high accuracy and precision.

Atomic-Level Imaging of Zeolite Local Structures Using Electron Ptychography

Zeolites are among the most important heterogeneous catalysts, widely employed in separation reaction, fine chemical production, and petroleum refining. Through rational design of the frameworks, zeolites with versatile functions can be synthesized. Local imaging of zeolite structures at the atomic scale, including the basic framework atoms (Si, Al, and O) and extra-framework cations, is necessary to understand the structure-function relationship of zeolites. Herein, we implemented electron ptychography into direct imaging of local structures of two zeolites, Na-LTA and ZSM-5. Not only all the framework atoms but also extra-framework Na⁺ cations with only 1/4 occupation probabilities in Na-LTA were directly observed. Local structures of ZSM-5 zeolites having guest molecules among channels with different orientations were also unraveled using different reconstruction algorithms. The approach presented here provides a new way to locally image zeolites structure, and it is expected to be an essential key for further studying and tuning zeolites active sites at the atomic level.

Deep-Learning Electron Diffractive Imaging

We report the development of deep-learning coherent electron diffractive imaging at subangstrom resolution using convolutional neural networks (CNNs) trained with only simulated data. We experimentally demonstrate this method by applying the trained CNNs to recover the phase images from electron diffraction patterns of twisted hexagonal boron nitride, monolayer graphene, and a gold nanoparticle with comparable quality to those reconstructed by a conventional ptychographic algorithm. Fourier ring correlation between the CNN and ptychographic images indicates the achievement of a resolution in the range of 0.70 and 0.55 Å. We further develop CNNs to recover the probe function from the experimental data. The ability to replace iterative algorithms with CNNs and perform real-time atomic imaging from coherent diffraction patterns is expected to find applications in the physical and biological sciences.

4D-STEM Ptychography for Electron-Beam-Sensitive Materials

Recent advances in high-speed pixelated electron detectors have substantially facilitated the implementation of four-dimensional scanning transmission electron microscopy (4D-STEM). A critical application of 4D-STEM is electron ptychography, which reveals the atomic structure of a specimen by reconstructing its transmission function from redundant convergent-beam electron diffraction patterns. Although 4D-STEM ptychography offers many advantages over conventional imaging modes, this emerging technique has not been fully applied to materials highly sensitive to electron beams. In this Outlook, we introduce the fundamentals of 4D-STEM ptychography, focusing on data collection and processing methods, and present the current applications of 4D-STEM ptychography in various materials. Next, we discuss the potential advantages of imaging electron-beam-sensitive materials using 4D-STEM ptychography and explore its feasibility by performing simulations and experiments on a zeolite material. The preliminary results demonstrate that, at the low electron dose required to preserve the zeolite structure, 4D-STEM ptychography can reliably provide higher resolution and greater tolerance to the specimen thickness and probe defocus as compared to existing imaging techniques. In the final section, we discuss the challenges and possible strategies to further reduce the electron dose for 4D-STEM ptychography. If successful, it will be a game-changer for imaging extremely sensitive materials, such as metal-organic frameworks, hybrid halide perovskites, and supramolecular crystals.

Cryomicroscopy in situ: what is the smallest molecule that can be directly identified without labels in a cell?

Electron cryomicroscopy (cryoEM) has made great strides in the last decade, such that the atomic structure of most biological macromolecules can, at least in principle, be determined. Major technological advances - in electron imaging hardware, data analysis software, and cryogenic specimen preparation technology - continue at pace and contribute to the exponential growth in the number of atomic structures determined by cryoEM. It is now conceivable that within the next decade we will have structures for hundreds of thousands of unique protein and nucleic acid molecular complexes. But the answers to many important questions in biology would become obvious if we could identify these structures precisely inside cells with quantifiable error. In the context of an abundance of known structures, it is appropriate to consider the current state of electron cryomicroscopy for frozen specimens prepared directly from cells, and try to answer to the question of the title, both now and in the foreseeable future.

Single-particle cryo-EM structures from iDPC-STEM at near-atomic resolution

In electron cryomicroscopy (cryo-EM), molecular images of vitrified biological samples are obtained by conventional transmission microscopy (CTEM) using large underfocuses and subsequently computationally combined into a high-resolution three-dimensional structure. Here, we apply scanning transmission electron microscopy (STEM) using the integrated differential phase contrast mode also known as iDPC-STEM to two cryo-EM test specimens, keyhole limpet hemocyanin (KLH) and tobacco mosaic virus (TMV). The micrographs show complete contrast transfer to high resolution and enable the cryo-EM structure determination for KLH at 6.5 Å resolution, as well as for TMV at 3.5 Å resolution using single-particle reconstruction methods, which share identical features with maps obtained by CTEM of a previously acquired same-sized TMV data set. These data show that STEM imaging in general, and in particular the iDPC-STEM approach, can be applied to vitrified single-particle specimens to determine near-atomic resolution cryo-EM structures of biological macromolecules.

Three-dimensional electron ptychography of organic-inorganic hybrid nanostructures

Three dimensional scaffolded DNA origami with inorganic nanoparticles has been used to create tailored multidimensional nanostructures. However, the image contrast of DNA is poorer than those of the heavy nanoparticles in conventional transmission electron microscopy at high defocus so that the biological and non-biological components in 3D scaffolds cannot be simultaneously resolved using tomography of samples in a native state. We demonstrate the use of electron ptychography to recover high contrast phase information from all components in a DNA origami scaffold without staining. We further quantitatively evaluate the enhancement of contrast in comparison with conventional transmission electron microscopy. In addition, We show that for ptychography post-reconstruction focusing simplifies the workflow and reduces electron dose and beam damage.

The authors demonstrate electron ptychographic computed tomography by simultaneously recording high contrast data from both the organic- and inorganic components in a 3D DNA-origami framework hybrid nanostructure.

Scanning Transmission Electron Microscopy in a Scanning Electron Microscope for the High-Throughput Imaging of Biological Assemblies

Electron microscopy of soft and biological materials, or "soft electron microscopy", is essential to the characterization of macromolecules. Soft microscopy is governed by enhancing contrast while maintaining low electron doses, and sample preparation and imaging methodologies are driven by the length scale of features of interest. While cryo-electron microscopy offers the highest resolution, larger structures can be characterized efficiently and with high contrast using low-voltage electron microscopy by performing scanning transmission electron microscopy in a scanning electron microscope (STEM-in-SEM). Here, STEM-in-SEM is demonstrated for a four-lobed protein assembly where the arrangement of the proteins in the construct must be examined. STEM image simulations show the theoretical contrast enhancement at SEM-level voltages for unstained structures, and experimental images with multiple STEM modes exhibit the resolution possible for negative-stained proteins. This technique can be extended to complex protein assemblies, larger structures such as cell sections, and hybrid materials, making STEM-in-SEM a valuable high-throughput imaging method.

Automatic parameter selection for electron ptychography via Bayesian optimization

Electron ptychography provides new opportunities to resolve atomic structures with deep sub-angstrom spatial resolution and to study electron-beam sensitive materials with high dose efficiency. In practice, obtaining accurate ptychography images requires simultaneously optimizing multiple parameters that are often selected based on trial-and-error, resulting in low-throughput experiments and preventing wider adoption. Here, we develop an automatic parameter selection framework to circumvent this problem using Bayesian optimization with Gaussian processes. With minimal prior knowledge, the workflow efficiently produces ptychographic reconstructions that are superior to those processed by experienced experts. The method also facilitates better experimental designs by exploring optimized experimental parameters from simulated data.

High-performance lensless diffraction imaging from diverse holograms by three-dimensional scanning

For lensless diffraction imaging, it is a challenging dilemma to achieve a large field of view (FOV) and high resolution with a small amount of data at the same time. Ptychography can reconstruct the high-resolution image and illumination light simultaneously. But the illumination is limited to a small size by a probe in typical ptychography. For large samples, it takes much time to collect abundant patterns and has strict requirements for the computing power of computers. Another widely applied method, multi-height measurement, can realize a wide FOV with several holograms. But, the recovered image is easily destroyed by the background noise. In this Letter, a lensless diffraction imaging method by three-dimensional scanning is proposed. All positions of the object are different in three directions instead of scanning schemes only on a plane or along the optic axis, so more diversity of diffraction information is obtained. We apply the illumination without the limit of a confined aperture, which means that the imaging FOV of a pattern is equal to the size of the utilized image sensor. In comparison with the multi-height method, our method can separate the illumination background noise from the retrieved object. Consequently, the proposed method realized high resolution and contrast, large FOV, and the removal of background noise simultaneously. Experimental validations and comparisons with other methods are presented.

Deep sub-angstrom resolution imaging by electron ptychography with misorientation correction

Superresolution imaging of solids is essential to explore local symmetry breaking and derived material properties. Electron ptychography is one of the most promising schemes to realize superresolution imaging beyond aberration correction. However, to reach both deep sub-angstrom resolution imaging and accurate measurement of atomic structures, it is still required for the electron beam to be nearly parallel to the zone axis of crystals. Here, we report an efficient and robust method to correct the specimen misorientation in electron ptychography, giving deep sub-angstrom resolution for specimens with large misorientations. The method largely reduces the experimental difficulties of electron ptychography and paves the way for widespread applications of ptychographic deep sub-angstrom resolution imaging.

Accurate and Robust Calibration of the Uniform Affine Transformation Between Scan-Camera Coordinates for Atom-Resolved In-Focus 4D-STEM Datasets

Accurate geometrical calibration between the scan coordinates and the camera coordinates is critical in four-dimensional scanning transmission electron microscopy (4D-STEM) for both quantitative imaging and ptychographic reconstructions. For atomic-resolved, in-focus 4D-STEM datasets, we propose a hybrid method incorporating two sub-routines, namely a J-matrix method and a Fourier method, which can calibrate the uniform affine transformation between the scan-camera coordinates using raw data, without a priori knowledge of the crystal structure of the specimen. The hybrid method is found robust against scan distortions and residual probe aberrations. It is also effective even when defects are present in the specimen, or the specimen becomes relatively thick. We will demonstrate that a successful geometrical calibration with the hybrid method will lead to a more reliable recovery of both the specimen and the electron probe in a ptychographic reconstruction. We will also show that, although the elimination of local scan position errors still requires an iterative approach, the rate of convergence can be improved, and the residual errors can be further reduced if the hybrid method can be firstly applied for initial calibration. The code is made available as a simple-to-use tool to correct affine transformations of the scan-camera coordinates in 4D-STEM experiments.

Event driven 4D STEM acquisition with a Timepix3 detector: Microsecond dwell time and faster scans for high precision and low dose applications

Four dimensional scanning transmission electron microscopy (4D STEM) records the scattering of electrons in a material in great detail. The benefits offered by 4D STEM are substantial, with the wealth of data it provides facilitating for instance high precision, high electron dose efficiency phase imaging via centre of mass or ptychography based analysis. However the requirement for a 2D image of the scattering to be recorded at each probe position has long placed a severe bottleneck on the speed at which 4D STEM can be performed. Recent advances in camera technology have greatly reduced this bottleneck, with the detection efficiency of direct electron detectors being especially well suited to the technique. However even the fastest frame driven pixelated detectors still significantly limit the scan speed which can be used in 4D STEM, making the resulting data susceptible to drift and hampering its use for low dose beam sensitive applications. Here we report the development of the use of an event driven Timepix3 direct electron camera that allows us to overcome this bottleneck and achieve 4D STEM dwell times down to 100 ns; orders of magnitude faster than what has been possible with frame based readout. We characterize the detector for different acceleration voltages and show that the method is especially well suited for low dose imaging and promises rich datasets without compromising dwell time when compared to conventional STEM imaging.

Shaping of Electron Beams Using Sculpted Thin Films

Electron beam shaping by sculpted thin films relies on electron-matter interactions and the wave nature of electrons. It can be used to study physical phenomena of special electron beams and to develop technological applications in electron microscopy that offer new and improved measurement techniques and increased resolution in different imaging modes. In this Perspective, we review recent applications of sculpted thin films for electron orbital angular momentum sorting, improvements in phase contrast transmission electron microscopy, and aberration correction. For the latter, we also present new results of our work toward correction of the spherical aberration of Lorentz scanning transmission electron microscopes and suggest a method to correct chromatic aberration using thin films. This review provides practical insight for researchers in the field and motivates future progress in electron microscopy.

High temporal-resolution scanning transmission electron microscopy using sparse-serpentine scan pathways

Scanning transmission electron microscopy (STEM) provides structural analysis with sub-angstrom resolution. But the pixel-by-pixel scanning process is a limiting factor in acquiring high-speed data. Different strategies have been implemented to increase scanning speeds while at the same time minimizing beam damage via optimizing the scanning strategy. Here, we achieve the highest possible scanning speed by eliminating the image acquisition dead time induced by the beam flyback time combined with reducing the amount of scanning pixels via sparse imaging. A calibration procedure was developed to compensate for the hysteresis of the magnetic scan coils. A combination of sparse and serpentine scanning routines was tested for a crystalline thin film, gold nanoparticles, and in an in-situ liquid phase STEM experiment. Frame rates of 92, 23 and 5.8 s^-1 were achieved for images of a width of 128, 256, and 512 pixels, respectively. The methods described here can be applied to single-particle tracking and analysis of radiation sensitive materials.

Making the Most of your Electrons: Challenges and Opportunities in Characterizing Hybrid Interfaces with STEM

Inspired by the unique architectures composed of hard and soft materials in natural and biological systems, synthetic hybrid structures and associated soft-hard interfaces have recently evoked significant interest. Soft matter is typically dominated by fluctuations even at room temperature, while hard matter (which often serves as the substrate or anchor for the soft component) is governed by rigid mechanical behavior. This dichotomy offers considerable opportunities to leverage the disparate properties offered by these components across a wide spectrum spanning from basic science to engineering insights with significant technological overtones. Such hybrid structures, which include polymer nanocomposites, DNA functionalized nanoparticle superlattices and metal organic frameworks to name a few, have delivered promising insights into the areas of catalysis, environmental remediation, optoelectronics, medicine, and beyond. The interfacial structure between these hard and soft phases exists across a variety of length scales and often strongly influence the functionality of hybrid systems. While scanning/transmission electron microscopy (S/TEM) has proven to be a valuable tool for acquiring intricate molecular and nanoscale details of these interfaces, the unusual nature of hybrid composites presents a suite of challenges that make assessing or establishing the classical structure-property relationships especially difficult. These include challenges associated with preparing electron-transparent samples and obtaining sufficient contrast to resolve the interface between dissimilar materials given the dose sensitivity of soft materials. We discuss each of these challenges and supplement a review of recent developments in the field with additional experimental investigations and simulations to present solutions for attaining a nano or atomic-level understanding of these interfaces. These solutions present a host of opportunities for investigating and understanding the role interfaces play in this unique class of functional materials.

Toward Compositional Contrast by Cryo-STEM

Electron microscopy (EM) is the most versatile tool for the study of matter at scales ranging from subatomic to visible. The high vacuum environment and the charged irradiation require careful stabilization of many specimens of interest. Biological samples are particularly sensitive due to their composition of light elements suspended in an aqueous medium. Early investigators developed techniques of embedding and staining with heavy metal salts for contrast enhancement. Indeed, the Nobel Prize in 1974 recognized Claude, de Duve, and Palade for establishment of the field of cell biology, largely due to their developments in separation and preservation of cellular components for electron microscopy. A decade later, cryogenic fixation was introduced. Vitrification of the water avoids the need for dehydration and provides an ideal matrix in which the organic macromolecules are suspended; the specimen represents a native state, suddenly frozen in time at temperatures below -150 °C. The low temperature maintains a low vapor pressure for the electron microscope, and the amorphous nature of the medium avoids diffraction contrast from crystalline ice. Such samples are extremely delicate, however, and cryo-EM imaging is a race for information in the face of ongoing damage by electron irradiation. Through this journey, cryo-EM enhanced the resolution scale from membranes to molecules and most recently to atoms. Cryo-EM pioneers, Dubochet, Frank, and Henderson, were awarded the Nobel Prize in 2017 for high resolution structure determination of biological macromolecules.A relatively untapped feature of cryo-EM is its preservation of composition. Nothing is added and nothing removed. Analytical spectroscopies based on electron energy loss or X-ray emission can be applied, but the very small interaction cross sections conflict with the weak exposures required to preserve sample integrity. To what extent can we interpret quantitatively the pixel intensities in images themselves? Conventional cryo-transmission electron microscopy (TEM) is limited in this respect, due to the strong dependence of the contrast transfer on defocus and the absence of contrast at low spatial frequencies.Inspiration comes largely from a different modality for cryo-tomography, using soft X-rays. Contrast depends on the difference in atomic absorption between carbon and oxygen in a region of the spectrum between their core level ionization energies, the so-called water window. Three dimensional (3D) reconstruction provides a map of the local X-ray absorption coefficient. The quantitative contrast enables the visualization of organic materials without stain and measurement of their concentration quantitatively. We asked, what aspects of the quantitative contrast might be transferred to cryo-electron microscopy?Compositional contrast is accessible in scanning transmission EM (STEM) via incoherent elastic scattering, which is sensitive to the atomic number Z. STEM can be regarded as a high energy, low angle diffraction measurement performed pixel by pixel with a weakly convergent beam. When coherent diffraction effects are absent, that is, in amorphous materials, a dark field signal measures quantitatively the flux scattered from the specimen integrated over the detector area. Learning to interpret these signals will open a new dimension in cryo-EM. This Account describes our efforts so far to introduce STEM for cryo-EM and tomography of biological specimens. We conclude with some thoughts on further developments.

Single-particle cryo-EM: alternative schemes to improve dose efficiency

Imaging of biomolecules by ionizing radiation, such as electrons, causes radiation damage which introduces structural and compositional changes of the specimen. The total number of high-energy electrons per surface area that can be used for imaging in cryogenic electron microscopy (cryo-EM) is severely restricted due to radiation damage, resulting in low signal-to-noise ratios (SNR). High resolution details are dampened by the transfer function of the microscope and detector, and are the first to be lost as radiation damage alters the individual molecules which are presumed to be identical during averaging. As a consequence, radiation damage puts a limit on the particle size and sample heterogeneity with which electron microscopy (EM) can deal. Since a transmission EM (TEM) image is formed from the scattering process of the electron by the specimen interaction potential, radiation damage is inevitable. However, we can aim to maximize the information transfer for a given dose and increase the SNR by finding alternatives to the conventional phase-contrast cryo-EM techniques. Here some alternative transmission electron microscopy techniques are reviewed, including phase plate, multi-pass transmission electron microscopy, off-axis holography, ptychography and a quantum sorter. Their prospects for providing more or complementary structural information within the limited lifetime of the sample are discussed.

Advances and Applications of Atomic-Resolution Scanning Transmission Electron Microscopy

Although scanning transmission electron microscopy (STEM) images of individual heavy atoms were reported 50 years ago, the applications of atomic-resolution STEM imaging became wide spread only after the practical realization of aberration correctors on field-emission STEM/TEM instruments to form sub-Ångstrom electron probes. The innovative designs and advances of electron optical systems, the fundamental understanding of electron–specimen interaction processes, and the advances in detector technology all played a major role in achieving the goal of atomic-resolution STEM imaging of practical materials. It is clear that tremendous advances in computer technology and electronics, image acquisition and processing algorithms, image simulations, and precision machining synergistically made atomic-resolution STEM imaging routinely accessible. It is anticipated that further hardware/software development is needed to achieve three-dimensional atomic-resolution STEM imaging with single-atom chemical sensitivity, even for electron-beam-sensitive materials. Artificial intelligence, machine learning, and big-data science are expected to significantly enhance the impact of STEM and associated techniques on many research fields such as materials science and engineering, quantum and nanoscale science, physics and chemistry, and biology and medicine. This review focuses on advances of STEM imaging from the invention of the field-emission electron gun to the realization of aberration-corrected and monochromated atomic-resolution STEM and its broad applications.

A Close Look at Molecular Self-Assembly with the Transmission Electron Microscope

Molecular self-assembly is pervasive in the formation of living and synthetic materials. Knowledge gained from research into the principles of molecular self-assembly drives innovation in the biological, chemical, and materials sciences. Self-assembly processes span a wide range of temporal and spatial domains and are often unintuitive and complex. Studying such complex processes requires an arsenal of analytical and computational tools. Within this arsenal, the transmission electron microscope stands out for its unique ability to visualize and quantify self-assembly structures and processes. This review describes the contribution that the transmission electron microscope has made to the field of molecular self-assembly. An emphasis is placed on which TEM methods are applicable to different structures and processes and how TEM can be used in combination with other experimental or computational methods. Finally, we provide an outlook on the current challenges to, and opportunities for, increasing the impact that the transmission electron microscope can have on molecular self-assembly.