Validation of 3D EM Reconstructions: The Phantom in the Noise

Image processing tools for the validation of CryoEM maps

The number of maps deposited in public databases (Electron Microscopy Data Bank, EMDB) determined by cryo-electron microscopy has quickly grown in recent years. With this rapid growth, it is critical to guarantee their quality. So far, map validation has primarily focused on the agreement between maps and models. From the image processing perspective, the validation has been mostly restricted to using two half-maps and the measurement of their internal consistency. In this article, we suggest that map validation can be taken much further from the point of view of image processing if 2D classes, particles, angles, coordinates, defoci, and micrographs are also provided. We present a progressive validation scheme that qualifies a result validation status from 0 to 5 and offers three optional qualifiers (A, W, and O) that can be added. The simplest validation state is 0, while the most complete would be 5AWO. This scheme has been implemented in a website https://biocomp.cnb.csic.es/EMValidationService/ to which reconstructed maps and their ESI can be uploaded.

Emerging Themes in CryoEM─Single Particle Analysis Image Processing

Cryo-electron microscopy (CryoEM) has become a vital technique in structural biology. It is an interdisciplinary field that takes advantage of advances in biochemistry, physics, and image processing, among other disciplines. Innovations in these three basic pillars have contributed to the boosting of CryoEM in the past decade. This work reviews the main contributions in image processing to the current reconstruction workflow of single particle analysis (SPA) by CryoEM. Our review emphasizes the time evolution of the algorithms across the different steps of the workflow differentiating between two groups of approaches: analytical methods and deep learning algorithms. We present an analysis of the current state of the art. Finally, we discuss the emerging problems and challenges still to be addressed in the evolution of CryoEM image processing methods in SPA.

Bsoft: Image Processing for Structural Biology

Bsoft is a software package primarily developed for processing electron micrographs, with the goal of determining the structures of biologically relevant molecules, molecular assemblies, and parts of cells. However, it incorporates many ways to deal with images, from the mundane to very sophisticated algorithms. This article is an introduction into its use, illustrating that it is an extensive toolbox, for manipulating and understanding images. Bsoft has over 150 programs, allowing the user an infinite number of ways to process images. These programs can be executed on the command line, or through the interactive program called brun. The main visualization program is bshow, providing numerous ways to manipulate and interpret images. The primary aim is to provide the user with powerful capabilities, including processing large numbers of images. An important additional aim is to make it as accessible as possible, making it easier to deal with image formats and features, and enhance productivity.

Image Processing in Cryo-Electron Microscopy of Single Particles: The Power of Combining Methods

Cryo-electron microscopy has established as a mature structural biology technique to elucidate the three-dimensional structure of biological macromolecules. The Coulomb potential of the sample is imaged by an electron beam, and fast semi-conductor detectors produce movies of the sample under study. These movies have to be further processed by a whole pipeline of image-processing algorithms that produce the final structure of the macromolecule. In this chapter, we illustrate this whole processing pipeline putting in value the strength of "meta algorithms," which are the combination of several algorithms, each one with different mathematical rationale, in order to distinguish correctly from incorrectly estimated parameters. We show how this strategy leads to superior performance of the whole pipeline as well as more confident assessments about the reconstructed structures. The "meta algorithms" strategy is common to many fields and, in particular, it has provided excellent results in bioinformatics. We illustrate this combination using the workflow engine, Scipion.

Validation tests for cryo-EM maps using an independent particle set

Cryo-electron microscopy (cryo-EM) has revolutionized structural biology by providing 3D density maps of biomolecules at near-atomic resolution. However, map validation is still an open issue. Despite several efforts from the community, it is possible to overfit 3D maps to noisy data. Here, we develop a novel methodology that uses a small independent particle set (not used during the 3D refinement) to validate the maps. The main idea is to monitor how the map probability evolves over the control set during the 3D refinement. The method is complementary to the gold-standard procedure, which generates two reconstructions at each iteration. We low-pass filter the two reconstructions for different frequency cutoffs, and we calculate the probability of each filtered map given the control set. For high-quality maps, the probability should increase as a function of the frequency cutoff and the refinement iteration. We also compute the similarity between the densities of probability distributions of the two reconstructions. As higher frequencies are included, the distributions become more dissimilar. We optimized the BioEM package to perform these calculations, and tested it over systems ranging from quality data to pure noise. Our results show that with our methodology, it possible to discriminate datasets that are constructed from noise particles. We conclude that validation against a control particle set provides a powerful tool to assess the quality of cryo-EM maps.

Need for Cross-Validation of Single Particle Cryo-EM

Cross-validation is used to determine the validity of a model on unseen data by assessing if the model is overfitted to noise. It is widely used in many fields, from artificial intelligence to structural biology in X-ray crystallography and nuclear magnetic resonance. Although there are concerns of map overfitting in cryo-electron microscopy (cryo-EM), cross-validation is rarely used. The problem is that establishing a performance metric of the maps over unseen data (given by 2D-projection images) is difficult due to the low signal-to-noise ratios in the individual particles. Here, I present recent advances for cryo-EM map reconstruction. I highlight that the gold-standard procedure can fail to detect map overfitting in certain cases, showing the necessity of assessing the map quality on unbiased data. Finally, I describe the challenges and advantages of developing a robust cross-validation methodology for cryo-EM.

Structure Determination by Single-Particle Cryo-Electron Microscopy: Only the Sky (and Intrinsic Disorder) is the Limit

Traditionally, X-ray crystallography and NMR spectroscopy represent major workhorses of structural biologists, with the lion share of protein structures reported in protein data bank (PDB) being generated by these powerful techniques. Despite their wide utilization in protein structure determination, these two techniques have logical limitations, with X-ray crystallography being unsuitable for the analysis of highly dynamic structures and with NMR spectroscopy being restricted to the analysis of relatively small proteins. In recent years, we have witnessed an explosive development of the techniques based on Cryo-electron microscopy (Cryo-EM) for structural characterization of biological molecules. In fact, single-particle Cryo-EM is a special niche as it is a technique of choice for the structural analysis of large, structurally heterogeneous, and dynamic complexes. Here, sub-nanometer atomic resolution can be achieved (i.e., resolution below 10 Å) via single-particle imaging of non-crystalline specimens, with accurate 3D reconstruction being generated based on the computational averaging of multiple 2D projection images of the same particle that was frozen rapidly in solution. We provide here a brief overview of single-particle Cryo-EM and show how Cryo-EM has revolutionized structural investigations of membrane proteins. We also show that the presence of intrinsically disordered or flexible regions in a target protein represents one of the major limitations of this promising technique.

Single-particle reconstruction statistics: a diagnostic tool in solving biomolecular structures by cryo-EM

In single-particle analysis (SPA), the aim is to obtain a 3D reconstruction of a biological molecule from 2D electron micrographs to the highest level of detail or resolution as possible. Current practice is to collect large volumes of data, hoping to reach high-resolution maps through sheer numbers. However, adding more particles from a specific data set eventually leads to diminishing improvements in resolution. Understanding what these resolution limits are and how to deal with them are important in optimization and automation of SPA. This study revisits the theory of 3D reconstruction and demonstrates how the associated statistics can provide a diagnostic tool to improve SPA. Small numbers of images already give sufficient information on micrograph quality and the amount of data required to reach high resolution. Such feedback allows the microscopist to improve sample-preparation and imaging parameters before committing to extensive data collection. Once a larger data set is available, a B factor can be determined describing the suppression of the signal owing to one or more causes, such as specimen movement, radiation damage, alignment inaccuracy and structural variation. Insight into the causes of signal suppression can then guide the user to consider appropriate actions to obtain better reconstructions.

Advances in image processing for single-particle analysis by electron cryomicroscopy and challenges ahead

Electron cryomicroscopy (cryoEM) is essential for the study and functional understanding of non-crystalline macromolecules such as proteins. These molecules cannot be imaged using X-ray crystallography or other popular methods. CryoEM has been successfully used to visualize macromolecular complexes such as ribosomes, viruses, and ion channels. Determination of structural models of these at various conformational states leads to insight on how these molecules function. Recent advances in imaging technology have given cryoEM a scientific rebirth. As a result of these technological advances image processing and analysis have yielded molecular structures at atomic resolution. Nevertheless there continue to be challenges in image processing, and in this article we will touch on the most essential in order to derive an accurate three-dimensional model from noisy projection images. Traditional approaches, such as k-means clustering for class averaging, will be provided as background. We will then highlight new approaches for each image processing subproblem, including a 3D reconstruction method for asymmetric molecules using just two projection images and deep learning algorithms for automated particle picking.

The first single particle analysis Map Challenge: A summary of the assessments

The recent successes of cryo-electron microscopy fostered great expectation of solving many new and previously recalcitrant biomolecular structures. However, it also brings with it the danger of compromising the validity of the outcomes if not done properly. The Map Challenge is a first step in assessing the state of the art and to shape future developments in data processing. The organizers presented seven cases for single particle reconstruction, and 27 members of the community responded with 66 submissions. Seven groups analyzed these submissions, resulting in several assessment reports, summarized here. We devised a range of analyses to evaluate the submitted maps, including visual impressions, Fourier shell correlation, pairwise similarity and interpretation through modeling. Unfortunately, we did not find strong trends. We ascribe this to the complexity of the challenge, dealing with multiple cases, software packages and processing approaches. This puts the user in the spotlight, where his/her choices becomes the determinant of map quality. The future focus should therefore be on promulgating best practices and encapsulating these in the software. Such practices include adherence to validation principles, most notably the processing of independent sets, proper resolution-limited alignment, appropriate masking and map sharpening. We consider the Map Challenge to be a highly valuable exercise that should be repeated frequently or on an ongoing basis.

Map Challenge assessment: Fair comparison of single particle cryoEM reconstructions

Cryo-electron microscopy (cryoEM) is capable of achieving near-atomic resolution of biomolecular structures due to recent advances in hardware. Despite the long history of image processing software development for cryoEM, uncertainty about best practices and validation remains. The Map Challenge was therefore designed to test the current state of single particle reconstruction. As the first such challenge, the participants were given the freedom to analyze the cases in whichever way they wanted. Therefore, the maps submitted feature different sizes, sampling and orientations, making assessment non-trivial. To be fair, I developed a method to pose all maps in each case in the same configuration with minimal interpolation. I assessed the quality of these maps by visual inspection and Fourier shell correlation (FSC). Comparing the even-odd FSC with an FSC calculated against a reference structure analysis, I concluded that the quality of the maps related more to the user than to other factors, such as the software package used. Poor quality maps suffer either from lack of data or poor choices made by the user. Some maps appear significantly better than a reference or consensus of other maps, indicating overfitting. Best practices to avoid problems include an understanding of the effects of reference map modifications on particle image alignment, and generating appropriate masks. Ultimately, none of the issues revealed in the Map Challenge is insurmountable, as underscored by the excellent quality of reconstructions achieved by a significant number of participants.

Single particle reconstruction and validation using Bsoft for the map challenge

The Bsoft package is aimed at processing electron micrographs for the determination of the three-dimensional structures of biological specimens. Recent advances in hardware allow us to solve structures to near atomic resolution using single particle analysis (SPA). The Map Challenge offered me an opportunity to test the ability of Bsoft to produce reconstructions from cryo-electron micrographs at the best resolution. I also wanted to understand what needed to be done to work towards full automation with validation. Here, I present two cases for the Map Challenge using Bsoft: ß-galactosidase and GroEL. I processed two independent subsets in each case with resolution-limited alignment. In both cases the reconstructions approached the expected resolution within a few iterations of alignment. I further validated the results by coherency-testing: i.e., that the reconstructions from real particles give better resolutions than reconstructions from the same number of aligned noise images. The key operations requiring attention for full automation are: particle picking, faster accurate alignment, proper mask generation, appropriate map sharpening, and understanding the amount of data needed to reach a desired resolution.

Guidelines for using Bsoft for high resolution reconstruction and validation of biomolecular structures from electron micrographs

Cryo-electron microscopy (cryoEM) is becoming popular as a tool to solve biomolecular structures with the recent availability of direct electron detectors allowing automated acquisition of high resolution data. The Bsoft software package, developed over 20 years for analyzing electron micrographs, offers a full workflow for validated single particle analysis with extensive functionality, enabling customization for specific cases. With the increasing use of cryoEM and its automation, proper validation of the results is a bigger concern. The three major validation approaches, independent data sets, resolution-limited processing, and coherence testing, can be incorporated into any Bsoft workflow. Here, the main workflow is divided into four phases: (i) micrograph preprocessing, (ii) particle picking, (iii) particle alignment and reconstruction, and (iv) interpretation. Each of these phases represents a conceptual unit that can be automated, followed by a check point to assess the results. The aim in the first three phases is to reconstruct one or more validated maps at the best resolution possible. Map interpretation then involves identification of components, segmentation, quantification, and modeling. The algorithms in Bsoft are well established, with future plans focused on ease of use, automation and institutionalizing validation.

Structure of the intact ATM/Tel1 kinase

The ataxia-telangiectasia mutated (ATM) protein is an apical kinase that orchestrates the multifaceted DNA-damage response. Normally, ATM kinase is in an inactive, homodimer form and is transformed into monomers upon activation. Besides a conserved kinase domain at the C terminus, ATM contains three other structural modules, referred to as FAT, FATC and N-terminal helical solenoid. Here we report the first cryo-EM structure of ATM kinase, which is an intact homodimeric ATM/Tel1 from Schizosaccharomyces pombe. We show that two monomers directly contact head-to-head through the FAT and kinase domains. The tandem N-terminal helical solenoid tightly packs against the FAT and kinase domains. The structure suggests that ATM/Tel1 dimer interface and the consecutive HEAT repeats inhibit the binding of kinase substrates and regulators by steric hindrance. Our study provides a structural framework for understanding the mechanisms of ATM/Tel1 regulation as well as the development of new therapeutic agents.