Sparse coding and dictionary learning for electron hologram denoising

Removal of phase residues in electron holography

Electron holography provides quantitative phase information regarding the electromagnetic fields and the morphology of micro- to nano-scale samples. A phase image reconstructed numerically from an electron hologram sometimes includes phase residues, i.e. origins of unremovable phase discontinuities, which make it much more difficult to quantitatively analyze local phase values. We developed a method to remove the residues in a phase image by a combination of patching local areas of a hologram and denoising based on machine learning. The small patches for a hologram, which were generated using the spatial frequency information of the own fringe patterns, were pasted at each residue point by an algorithm based on sparse modeling. After successive phase reconstruction, the phase components with no dependency on the vicinity were filtered out by Gaussian process regression. We determined that the phase discontinuities that appeared around phase residues were removed and the phase distributions of an atomic resolution phase image of a Pt nanoparticle were sufficiently restored.

Noise reduction of electron holography observations for a thin-foiled Nd-Fe-B specimen using the wavelet hidden Markov model

In this study, we investigate the effectiveness of noise reduction in electron holography, based on the wavelet hidden Markov model (WHMM), which allows the reasonable separation of weak signals from noise. Electron holography observations from a Nd2Fe14B thin foil showed that the noise reduction method suppressed artificial phase discontinuities generated by phase retrieval. From the peak signal-to-noise ratio, it was seen that the impact of denoising was significant for observations with a narrow spacing of interference fringes, which is a key parameter for the spatial resolution of electron holography. These results provide essential information for improving the precision of electron holography studies.

Low-dose measurement of electric potential distribution in organic light-emitting diode by phase-shifting electron holography with 3D tensor decomposition

To improve the performance of organic light-emitting diodes (OLEDs), it is essential to understand and control the electric potential in the organic semiconductor layers. Electron holography (EH) is a powerful technique for visualizing the potential distribution with a transmission electron microscope. However, it has a serious issue that high-energy electrons may damage the organic layers, meaning that a low-dose EH is required. Here, we used a machine learning technique, three-dimensional (3D) tensor decomposition, to denoise electron interference patterns (holograms) of bilayer OLEDs composed of N,N'-di-[(1-naphthyl)-N,N'-diphenyl]-(1,1'-biphenyl)-4,4'-diamine (α-NPD) and tris-(8-hydroxyquinoline)aluminum (Alq3), acquired under a low-dose rate of 130 e- nm-2 s-1. The effect of denoising on the phase images reconstructed from the holograms was evaluated in terms of both the phase measurement error and the peak signal-to-noise ratio. We achieved a precision equivalent to that of a conventional measurement that had an exposure time 60 times longer. The electric field within the Alq3 layer decreased as the cumulative dose increased, which indicates that the Alq3 layer was degraded by the electron irradiation. On the basis of the degradation of the electric field, we concluded that the tolerance dose without damaging the OLED sample is about 1.7 × 105 e- nm-2, which is about 0.6 times that of the conventional EH. The combination of EH and 3D tensor decomposition denoising is capable of making a time series measurement of an OLED sample without any effect from the electron irradiation.

Enhancing performance of electron holography with mathematical and machine learning-based denoising techniques

Electron holography is a useful tool for analyzing functional properties, such as electromagnetic fields and strains of materials and devices. The performance of electron holography is limited by the 'shot noise' inherent in electron micrographs (holograms), which are composed of a finite number of electrons. A promising approach for addressing this issue is to use mathematical and machine learning-based image-processing techniques for hologram denoising. With the advancement of information science, denoising methods have become capable of extracting signals that are completely buried in noise, and they are being applied to electron microscopy, including electron holography. However, these advanced denoising methods are complex and have many parameters to be tuned; therefore, it is necessary to understand their principles in depth and use them carefully. Herein, we present an overview of the principles and usage of sparse coding, the wavelet hidden Markov model and tensor decomposition, which have been applied to electron holography. We also present evaluation results for the denoising performance of these methods obtained through their application to simulated and experimentally recorded holograms. Our analysis, review and comparison of the methods clarify the impact of denoising on electron holography research.

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

In the last few years, electron microscopy has experienced a new methodological paradigm aimed to fix the bottlenecks and overcome the challenges of its analytical workflow. Machine learning and artificial intelligence are answering this call providing powerful resources towards automation, exploration, and development. In this review, we evaluate the state-of-the-art of machine learning applied to electron microscopy (and obliquely, to materials and nano-sciences). We start from the traditional imaging techniques to reach the newest higher-dimensionality ones, also covering the recent advances in spectroscopy and tomography. Additionally, the present review provides a practical guide for microscopists, and in general for material scientists, but not necessarily advanced machine learning practitioners, to straightforwardly apply the offered set of tools to their own research. To conclude, we explore the state-of-the-art of other disciplines with a broader experience in applying artificial intelligence methods to their research (e.g., high-energy physics, astronomy, Earth sciences, and even robotics, videogames, or marketing and finances), in order to narrow down the incoming future of electron microscopy, its challenges and outlook.

Early Detection of Nucleation Events From Solution in LC-TEM by Machine Learning

To support the detection, recording, and analysis of nucleation events during in situ observations, we developed an early detection system for nucleation events observed using a liquid-cell transmission electron microscope. Detectability was achieved using the machine learning equivalent of detection by humans watching a video numerous times. The detection system was applied to the nucleation of sodium chloride crystals from a saturated acetone solution of sodium chlorate. Nanoparticles with a radius of more greater than 150 nm were detected in a viewing area of 12 μm × 12 μm by the detection system. The analysis of the change in the size of the growing particles as a function of time suggested that the crystal phase of the particles with a radius smaller than 400 nm differed from that of the crystals larger than 400 nm. Moreover, the use of machine learning enabled the detection of numerous nanometer sized nuclei. The nucleation rate estimated from the machine-learning-based detection was of the same order as that estimated from the detection using manual procedures.

Computational Evaluation of Sparse Coding on off-axis Electron Holograms: Comparison Between Charge-Coupled Device and Direct-Detection Device Cameras

The effectiveness of sparse coding for image inpainting and denoising of off-axis electron holograms was examined computationally based on hologram simulations according to considerations of two types of electron detectors, namely, charge-coupled device (CCD) and direct-detection device (DDD) cameras. In this simulation, we used a simple-phase object with a phase step such as a semiconductor p-n junction and assumed that the holograms recorded by the CCD camera include shot noise, dark-current, and read-out noise, while those recorded by the DDD camera include only shot noise. Simulated holograms with various electron doses were sparsely coded. Even though interference fringes cannot be recognized in the simulated CCD and DDD holograms when subjected to electron doses (per pixel) equal to 1 and 0.01, respectively, both the corresponding sparse-coded holograms exhibit meaningful interference fringes. We demonstrate that a combination of the DDD camera and sparse coding reduces the requisite dose used to obtain holograms to values less than one-thousandth compared with the CCD camera without image postprocessing. This combination is expected to generate lower-dose and/or higher-speed electron holography.

Phase-shifting electron holography for accurate measurement of potential distributions in organic and inorganic semiconductors

Phase-shifting electron holography (PS-EH) is an interference transmission electron microscopy technique that accurately visualizes potential distributions in functional materials, such as semiconductors. In this paper, we briefly introduce the features of the PS-EH that overcome some of the issues facing the conventional EH based on Fourier transformation. Then, we present a high-precision PS-EH technique with multiple electron biprisms and a sample preparation technique using a cryo-focused-ion-beam, which are important techniques for the accurate phase measurement of semiconductors. We present several applications of PS-EH to demonstrate the potential in organic and inorganic semiconductors and then discuss the differences by comparing them with previous reports on the conventional EH. We show that in situ biasing PS-EH was able to observe not only electric potential distribution but also electric field and charge density at a GaAs p-n junction and clarify how local band structures, depletion layer widths and space charges changed depending on the biasing conditions. Moreover, the PS-EH clearly visualized the local potential distributions of two-dimensional electron gas layers formed at AlGaN/GaN interfaces with different Al compositions. We also report the results of our PS-EH application for organic electroluminescence multilayers and point out the significant potential changes in the layers. The proposed PS-EH enables more precise phase measurement compared to the conventional EH, and our findings introduced in this paper will contribute to the future research and development of high-performance semiconductor materials and devices.

Magnetic flux density measurements from narrow grain boundaries produced in sintered permanent magnets

This review examines methods of magnetic flux density measurements from the narrow grain boundary (GB) regions, the thickness of which is of the order of nanometers, produced in Nd-Fe-B-based sintered magnets. Despite of the complex crystallographic microstructure and the significant stray magnetic field of the sintered magnet, recent progress in electron holography allowed for the determination of the intrinsic magnetic flux density due to the GB which is embedded in the polycrystalline thin-foil. The methods appear to be useful as well for intensive studies about interface magnetism in a variety of systems.

Denoising of series electron holograms using tensor decomposition

In this study, a noise-reduction technique for series low-dose electron holograms using tensor decomposition is demonstrated through simulation. We treated an entire dataset of the series holograms with Poisson noise as a third-order tensor, which is a stack of 2D holograms. The third-order tensor, which is decomposed into a core tensor and three factor matrices, is approximated as a lower-rank tensor using only noise-free principal components. This technique is applied to simulated holograms by assuming a p-n junction in a semiconductor sample. The peak signal-to-noise ratios of the holograms and the reconstructed phase maps have been improved significantly using tensor decomposition. Moreover, the proposed method was applied to a more practical situation of time-resolved in situ electron holography by considering a nonuniform fringe contrast and fringe drift relative to the sample. The accuracy and precision of the reconstructed phase maps were quantitatively evaluated to demonstrate its effectiveness for in situ experiments and low-dose experiments on beam-sensitive materials.

Simulation-Trained Sparse Coding for High-Precision Phase Imaging in Low-Dose Electron Holography

We broaden the applicability of sparse coding, a machine learning method, to low-dose electron holography by using simulated holograms for learning and validation processes. The holograms, with shot noise, are prepared to generate a model, or a dictionary, that includes basic features representing interference fringes. The dictionary is applied to sparse representations of other simulated holograms with various signal-to-noise ratios (SNRs). Results demonstrate that this approach successfully removes noise for holograms with an extremely small SNR of 0.10, and that the denoised holograms provide the accurate phase distribution. Furthermore, this study demonstrates that the dictionary learned from the simulated holograms can be applied to denoising of experimental holograms of a p-n junction specimen recorded with different exposure times. The results indicate that the simulation-trained sparse coding is suitable for use over a wide range of imaging conditions, in particular for observing electron beam-sensitive materials.

Accuracy improvement of phase estimation in electron holography using noise reduction methods

We try to improve the limit of the phase estimation of the interference fringe at low electron dose levels in electron holography by a noise reduction method. In this paper, we focus on unsupervised approaches to apply it to electron beam-sensitive and unknown samples and describe an overview of denoising methods used widely in image processing, such as wiener filter, total variation denoising, nonlocal mean filters and wavelet thresholding. We compare the wavelet hidden Markov model (WHMM) denoising that we have studied so far with the other conventional noise reduction methods. We evaluate the denoise performance of each method using the peak signal-to-noise ratio between noise-free and the target holograms (noisy or denoised holograms) and the root mean-square error (RMSE) between the true phase of the fringe and the measured phase by the discrete Fourier transform phase estimator. We show the denoised holograms for simulation and experimental data by using each noise reduction method and then discuss evaluation indexes obtained from these denoised holograms. From experimental results, it can be seen that the WHMM denoising can reduce the RMSE of fringe phase to about 1/4.5 for noisy simulation holograms and it has stable and good performance for noise reduction of observed holograms with various image qualities.