Super-compression of large electron microscopy time series by deep compressive sensing learning

The development of ultrafast detectors for electron microscopy (EM) opens a new door to exploring dynamics of nanomaterials; however, it raises grand challenges for big data processing and storage. Here, we combine deep learning and temporal compressive sensing (TCS) to propose a novel EM big data compression strategy. Specifically, TCS is employed to compress sequential EM images into a single compressed measurement; an end-to-end deep learning network is leveraged to reconstruct the original images. Owing to the significantly improved compression efficiency and built-in denoising capability of the deep learning framework over conventional JPEG compression, compressed videos with a compression ratio of up to 30 can be reconstructed with high fidelity. Using this approach, considerable encoding power, memory, and transmission bandwidth can be saved, allowing it to be deployed to existing detectors. We anticipate the proposed technique will have far-reaching applications in edge computing for EM and other imaging techniques.

•

A novel framework, i.e., TCS-DL, has been proposed for big data compressing for EM

•

The proposed TCL-DL outperforms JPEG due to the built-in denoising capability

•

Considerable power, in situ memory, and transmission bandwidth could be saved

•

The proposed TCL-DL is a novel and promising way for EM data compressing

The rapid development of electron microscopy (EM) opens a new door to exploring physical sciences; however, it raises grand challenges and urgent needs for big data processing. Therefore, it is crucial to compress the EM data. But existing compression methods developed for natural images do not perform well in EM images. In this paper, by combining deep learning and temporal compressive sensing, we propose a novel compression strategy specifically for EM data processing. Owing to the improved compression efficiency and built-in denoising capability of our framework over JPEG compression, compressed videos with compression ratio of 30 can be reconstructed with high fidelity. Therefore, considerable (encoding) power, in situ memory, and transmission bandwidth are expected to be saved. In the future, we will strive to increase the compression ratio without reducing the reconstruction quality. And we believe our proposed EM compression method has a wide application for the EM community.

Elemental mapping of labelled biological specimens at intermediate energy loss in an energy-filtered TEM acquired using a direct detection device

The technique of colour EM that was recently developed enabled localisation of specific macromolecules/proteins of interest by the targeted deposition of diaminobenzidine (DAB) conjugated to lanthanide chelates. By acquiring lanthanide elemental maps by energy‐filtered transmission electron microscopy (EFTEM) and overlaying them in pseudo‐colour over the conventional greyscale TEM image, a colour EM image is generated. This provides a powerful tool for visualising subcellular component/s, by the ability to clearly distinguish them from the general staining of the endogenous cellular material. Previously, the lanthanide elemental maps were acquired at the high‐loss M_4,5 edge (excitation of 3d electrons), where the characteristic signal is extremely low and required considerably long exposures. In this paper, we explore the possibility of acquiring the elemental maps of lanthanides at their N_4,5 edge (excitation of 4d electrons), which occurring at a much lower energy‐loss regime, thereby contains significantly greater total characteristic signal owing to the higher inelastic scattering cross‐sections at the N_4,5 edge. Acquiring EFTEM lanthanide elemental maps at the N_4,5 edge instead of the M_4,5 edge, provides ∼4× increase in signal‐to‐noise and ∼2× increase in resolution. However, the interpretation of the lanthanide maps acquired at the N_4,5 edge by the traditional 3‐window method, is complicated due to the broad shape of the edge profile and the lower signal‐above‐background ratio. Most of these problems can be circumvented by the acquisition of elemental maps with the more sophisticated technique of EFTEM Spectrum Imaging (EFTEM SI). Here, we also report the chemical synthesis of novel second‐generation DAB lanthanide metal chelate conjugates that contain 2 lanthanide ions per DAB molecule in comparison with 0.5 lanthanide ion per DAB in the first generation. Thereby, fourfold more Ln³⁺ per oxidised DAB would be deposited providing significant amplification of signal. This paper applies the colour EM technique at the intermediate‐loss energy‐loss regime to three different cellular targets, namely using mitochondrial matrix‐directed APEX2, histone H2B‐Nucleosome and EdU‐DNA. All the examples shown in the paper are single colour EM images only.

Improving signal to noise in labeled biological specimens using energy-filtered TEM of sections with a drift correction strategy and a direct detection device

Energy filtered transmission electron microscopy techniques are regularly used to build elemental maps of spatially distributed nanoparticles in materials and biological specimens. When working with thick biological sections, electron energy loss spectroscopy techniques involving core-loss electrons often require exposures exceeding several minutes to provide sufficient signal to noise. Image quality with these long exposures is often compromised by specimen drift, which results in blurring and reduced resolution. To mitigate drift artifacts, a series of short exposure images can be acquired, aligned, and merged to form a single image. For samples where the target elements have extremely low signal yields, the use of charge coupled device (CCD)-based detectors for this purpose can be problematic. At short acquisition times, the images produced by CCDs can be noisy and may contain fixed pattern artifacts that impact subsequent correlative alignment. Here we report on the use of direct electron detection devices (DDD's) to increase the signal to noise as compared with CCD's. A 3× improvement in signal is reported with a DDD versus a comparably formatted CCD, with equivalent dose on each detector. With the fast rolling-readout design of the DDD, the duty cycle provides a major benefit, as there is no dead time between successive frames.