DeepPicker: A deep learning approach for fully automated particle picking in cryo-EM

Accurate size-based protein localization from cryo-ET tomograms

Cryo-electron tomography (cryo-ET) combined with sub-tomogram averaging (STA) allows the determination of protein structures imaged within the native context of the cell at near-atomic resolution. Particle picking is an essential step in the cryo-ET/STA image analysis pipeline that consists in locating the position of proteins within crowded cellular tomograms so that they can be aligned and averaged in 3D to improve resolution. While extensive work in 2D particle picking has been done in the context of single-particle cryo-EM, comparatively fewer strategies have been proposed to pick particles from 3D tomograms, in part due to the challenges associated with working with noisy 3D volumes affected by the missing wedge. While strategies based on 3D template-matching and deep learning are commonly used, these methods are computationally expensive and require either an external template or manual labelling which can bias the results and limit their applicability. Here, we propose a size-based method to pick particles from tomograms that is fast, accurate, and does not require external templates or user provided labels. We compare the performance of our approach against a commonly used algorithm based on deep learning, crYOLO, and show that our method: i) has higher detection accuracy, ii) does not require user input for labeling or time-consuming training, and iii) runs efficiently on non-specialized CPU hardware. We demonstrate the effectiveness of our approach by automatically detecting particles from tomograms representing different types of samples and using these particles to determine the high-resolution structures of ribosomes imaged in vitro and in situ.

The advent of preventive high-resolution structural histopathology by artificial-intelligence-powered cryogenic electron tomography

Advances in cryogenic electron microscopy (cryoEM) single particle analysis have revolutionized structural biology by facilitating the in vitro determination of atomic- and near-atomic-resolution structures for fully hydrated macromolecular complexes exhibiting compositional and conformational heterogeneity across a wide range of sizes. Cryogenic electron tomography (cryoET) and subtomogram averaging are rapidly progressing toward delivering similar insights for macromolecular complexes in situ, without requiring tags or harsh biochemical purification. Furthermore, cryoET enables the visualization of cellular and tissue phenotypes directly at molecular, nanometric resolution without chemical fixation or staining artifacts. This forward-looking review covers recent developments in cryoEM/ET and related technologies such as cryogenic focused ion beam milling scanning electron microscopy and correlative light microscopy, increasingly enhanced and supported by artificial intelligence algorithms. Their potential application to emerging concepts is discussed, primarily the prospect of complementing medical histopathology analysis. Machine learning solutions are poised to address current challenges posed by "big data" in cryoET of tissues, cells, and macromolecules, offering the promise of enabling novel, quantitative insights into disease processes, which may translate into the clinic and lead to improved diagnostics and targeted therapeutics.

CryoSegNet: accurate cryo-EM protein particle picking by integrating the foundational AI image segmentation model and attention-gated U-Net

Picking protein particles in cryo-electron microscopy (cryo-EM) micrographs is a crucial step in the cryo-EM-based structure determination. However, existing methods trained on a limited amount of cryo-EM data still cannot accurately pick protein particles from noisy cryo-EM images. The general foundational artificial intelligence-based image segmentation model such as Meta's Segment Anything Model (SAM) cannot segment protein particles well because their training data do not include cryo-EM images. Here, we present a novel approach (CryoSegNet) of integrating an attention-gated U-shape network (U-Net) specially designed and trained for cryo-EM particle picking and the SAM. The U-Net is first trained on a large cryo-EM image dataset and then used to generate input from original cryo-EM images for SAM to make particle pickings. CryoSegNet shows both high precision and recall in segmenting protein particles from cryo-EM micrographs, irrespective of protein type, shape and size. On several independent datasets of various protein types, CryoSegNet outperforms two top machine learning particle pickers crYOLO and Topaz as well as SAM itself. The average resolution of density maps reconstructed from the particles picked by CryoSegNet is 3.33 Å, 7% better than 3.58 Å of Topaz and 14% better than 3.87 Å of crYOLO. It is publicly available at https://github.com/jianlin-cheng/CryoSegNet.

Phase Identification and Discovery of an Elusive Polymorph of Drug-Polymer Inclusion Complex Using Automated 3D Electron Diffraction

3D electron diffraction (3D ED) has shown great potential in crystal structure determination in materials, small organic molecules, and macromolecules. In this work, an automated, low-dose and low-bias 3D ED protocol has been implemented to identify six phases from a multiple-phase melt-crystallisation product of an active pharmaceutical ingredient, griseofulvin (GSF). Batch data collection under low-dose conditions using a widely available commercial software was combined with automated data analysis to collect and process over 230 datasets in three days. Accurate unit cell parameters obtained from 3D ED data allowed direct phase identification of GSF Forms III, I and the known GSF inclusion complex (IC) with polyethylene glycol (PEG) (GSF-PEG IC-I), as well as three minor phases, namely GSF Forms II, V and an elusive new phase, GSF-PEG IC-II. Their structures were then directly determined by 3D ED. Furthermore, we reveal how the stabilities of the two GSF-PEG IC polymorphs are closely related to their crystal structures. These results demonstrate the power of automated 3D ED for accurate phase identification and direct structure determination of complex, beam-sensitive crystallisation products, which is significant for drug development where solid form screening is crucial for the overall efficacy of the drug product.

Structural basis of U12-type intron engagement by the fully assembled human minor spliceosome

The minor spliceosome, which is responsible for the splicing of U12-type introns, comprises five small nuclear RNAs (snRNAs), of which only one is shared with the major spliceosome. In this work, we report the 3.3-angstrom cryo-electron microscopy structure of the fully assembled human minor spliceosome pre-B complex. The atomic model includes U11 small nuclear ribonucleoprotein (snRNP), U12 snRNP, and U4atac/U6atac.U5 tri-snRNP. U11 snRNA is recognized by five U11-specific proteins (20K, 25K, 35K, 48K, and 59K) and the heptameric Sm ring. The 3' half of the 5'-splice site forms a duplex with U11 snRNA; the 5' half is recognized by U11-35K, U11-48K, and U11 snRNA. Two proteins, CENATAC and DIM2/TXNL4B, specifically associate with the minor tri-snRNP. A structural analysis uncovered how two conformationally similar tri-snRNPs are differentiated by the minor and major prespliceosomes for assembly.

Joint micrograph denoising and protein localization in cryo-electron microscopy

Cryo-electron microscopy (cryo-EM) is an imaging technique that allows the visualization of proteins and macromolecular complexes at near-atomic resolution. The low electron doses used to prevent radiation damage to the biological samples result in images where the power of noise is 100 times stronger than that of the signal. Accurate identification of proteins from these low signal-to-noise ratio (SNR) images is a critical task, as the detected positions serve as inputs for the downstream 3D structure determination process. Current methods either fail to identify all true positives or result in many false positives, especially when analyzing images from smaller-sized proteins that exhibit extremely low contrast, or require manual labeling that can take days to complete. Acknowledging the fact that accurate protein identification is dependent upon the visual interpretability of micrographs, we propose a framework that can perform denoising and detection in a joint manner and enable particle localization under extremely low SNR conditions using self-supervised denoising and particle identification from sparsely annotated data. We validate our approach on three challenging single-particle cryo-EM datasets and projection images from one cryo-electron tomography dataset with extremely low SNR, showing that it outperforms existing state-of-the-art methods used for cryo-EM image analysis by a significant margin. We also evaluate the performance of our algorithm under decreasing SNR conditions and show that our method is more robust to noise than competing methods.

Cryo-electron microscopy-based drug design

Structure-based drug design (SBDD) has gained popularity owing to its ability to develop more potent drugs compared to conventional drug-discovery methods. The success of SBDD relies heavily on obtaining the three-dimensional structures of drug targets. X-ray crystallography is the primary method used for solving structures and aiding the SBDD workflow; however, it is not suitable for all targets. With the resolution revolution, enabling routine high-resolution reconstruction of structures, cryogenic electron microscopy (cryo-EM) has emerged as a promising alternative and has attracted increasing attention in SBDD. Cryo-EM offers various advantages over X-ray crystallography and can potentially replace X-ray crystallography in SBDD. To fully utilize cryo-EM in drug discovery, understanding the strengths and weaknesses of this technique and noting the key advancements in the field are crucial. This review provides an overview of the general workflow of cryo-EM in SBDD and highlights technical innovations that enable its application in drug design. Furthermore, the most recent achievements in the cryo-EM methodology for drug discovery are discussed, demonstrating the potential of this technique for advancing drug development. By understanding the capabilities and advancements of cryo-EM, researchers can leverage the benefits of designing more effective drugs. This review concludes with a discussion of the future perspectives of cryo-EM-based SBDD, emphasizing the role of this technique in driving innovations in drug discovery and development. The integration of cryo-EM into the drug design process holds great promise for accelerating the discovery of new and improved therapeutic agents to combat various diseases.

CryoTransformer: a transformer model for picking protein particles from cryo-EM micrographs

MOTIVATION

Cryo-electron microscopy (cryo-EM) is a powerful technique for determining the structures of large protein complexes. Picking single protein particles from cryo-EM micrographs (images) is a crucial step in reconstructing protein structures from them. However, the widely used template-based particle picking process requires some manual particle picking and is labor-intensive and time-consuming. Though machine learning and artificial intelligence (AI) can potentially automate particle picking, the current AI methods pick particles with low precision or low recall. The erroneously picked particles can severely reduce the quality of reconstructed protein structures, especially for the micrographs with low signal-to-noise ratio.

RESULTS

To address these shortcomings, we devised CryoTransformer based on transformers, residual networks, and image processing techniques to accurately pick protein particles from cryo-EM micrographs. CryoTransformer was trained and tested on the largest labeled cryo-EM protein particle dataset-CryoPPP. It outperforms the current state-of-the-art machine learning methods of particle picking in terms of the resolution of 3D density maps reconstructed from the picked particles as well as F1-score, and is poised to facilitate the automation of the cryo-EM protein particle picking.

AVAILABILITY AND IMPLEMENTATION

The source code and data for CryoTransformer are openly available at: https://github.com/jianlin-cheng/CryoTransformer.

Scaling up cryo-EM for biology and chemistry: The journey from niche technology to mainstream method

Cryoelectron microscopy (cryo-EM) methods have made meaningful contributions in a wide variety of scientific research fields. In structural biology, cryo-EM routinely elucidates molecular structure from isolated biological macromolecular complexes or in a cellular context by harnessing the high-resolution power of the electron in order to image samples in a frozen, hydrated environment. For structural chemistry, the cryo-EM method popularly known as microcrystal electron diffraction (MicroED) has facilitated atomic structure generation of peptides and small molecules from their three-dimensional crystal forms. As cryo-EM has grown from an emerging technology, it has undergone modernization to enable multimodal transmission electron microscopy (TEM) techniques becoming more routine, reproducible, and accessible to accelerate research across multiple disciplines. We review recent advances in modern cryo-EM and assess how they are contributing to the future of the field with an eye to the past.

CryoTransformer: A Transformer Model for Picking Protein Particles from Cryo-EM Micrographs

Cryo-electron microscopy (cryo-EM) is a powerful technique for determining the structures of large protein complexes. Picking single protein particles from cryo-EM micrographs (images) is a crucial step in reconstructing protein structures from them. However, the widely used template-based particle picking process requires some manual particle picking and is labor-intensive and time-consuming. Though machine learning and artificial intelligence (AI) can potentially automate particle picking, the current AI methods pick particles with low precision or low recall. The erroneously picked particles can severely reduce the quality of reconstructed protein structures, especially for the micrographs with low signal-to-noise (SNR) ratios. To address these shortcomings, we devised CryoTransformer based on transformers, residual networks, and image processing techniques to accurately pick protein particles from cryo-EM micrographs. CryoTransformer was trained and tested on the largest labelled cryo-EM protein particle dataset - CryoPPP. It outperforms the current state-of-the-art machine learning methods of particle picking in terms of the resolution of 3D density maps reconstructed from the picked particles as well as F1-score and is poised to facilitate the automation of the cryo-EM protein particle picking.

Big data in contemporary electron microscopy: challenges and opportunities in data transfer, compute and management

The second decade of the twenty-first century witnessed a new challenge in the handling of microscopy data. Big data, data deluge, large data, data compliance, data analytics, data integrity, data interoperability, data retention and data lifecycle are terms that have introduced themselves to the electron microscopy sciences. This is largely attributed to the booming development of new microscopy hardware tools. As a result, large digital image files with an average size of one terabyte within one single acquisition session is not uncommon nowadays, especially in the field of cryogenic electron microscopy. This brings along numerous challenges in data transfer, compute and management. In this review, we will discuss in detail the current state of international knowledge on big data in contemporary electron microscopy and how big data can be transferred, computed and managed efficiently and sustainably. Workflows, solutions, approaches and suggestions will be provided, with the example of the latest experiences in Australia. Finally, important principles such as data integrity, data lifetime and the FAIR and CARE principles will be considered.

Enabling cryo-EM density interpretation from yeast native cell extracts by proteomics data and AlphaFold structures

In the cellular context, proteins participate in communities to perform their function. The detection and identification of these communities as well as in-community interactions has long been the subject of investigation, mainly through proteomics analysis with mass spectrometry. With the advent of cryogenic electron microscopy and the "resolution revolution," their visualization has recently been made possible, even in complex, native samples. The advances in both fields have resulted in the generation of large amounts of data, whose analysis requires advanced computation, often employing machine learning approaches to reach the desired outcome. In this work, we first performed a robust proteomics analysis of mass spectrometry (MS) data derived from a yeast native cell extract and used this information to identify protein communities and inter-protein interactions. Cryo-EM analysis of the cell extract provided a reconstruction of a biomolecule at medium resolution (∼8 Å (FSC = 0.143)). Utilizing MS-derived proteomics data and systematic fitting of AlphaFold-predicted atomic models, this density was assigned to the 2.6 MDa complex of yeast fatty acid synthase. Our proposed workflow identifies protein complexes in native cell extracts from Saccharomyces cerevisiae by combining proteomics, cryo-EM, and AI-guided protein structure prediction.

Novel Artificial Intelligence-Based Approaches for Ab Initio Structure Determination and Atomic Model Building for Cryo-Electron Microscopy

Single particle cryo-electron microscopy (cryo-EM) has emerged as the prevailing method for near-atomic structure determination, shedding light on the important molecular mechanisms of biological macromolecules. However, the inherent dynamics and structural variability of biological complexes coupled with the large number of experimental images generated by a cryo-EM experiment make data processing nontrivial. In particular, ab initio reconstruction and atomic model building remain major bottlenecks that demand substantial computational resources and manual intervention. Approaches utilizing recent innovations in artificial intelligence (AI) technology, particularly deep learning, have the potential to overcome the limitations that cannot be adequately addressed by traditional image processing approaches. Here, we review newly proposed AI-based methods for ab initio volume generation, heterogeneous 3D reconstruction, and atomic model building. We highlight the advancements made by the implementation of AI methods, as well as discuss remaining limitations and areas for future development.

A large expert-curated cryo-EM image dataset for machine learning protein particle picking

Cryo-electron microscopy (cryo-EM) is a powerful technique for determining the structures of biological macromolecular complexes. Picking single-protein particles from cryo-EM micrographs is a crucial step in reconstructing protein structures. However, the widely used template-based particle picking process is labor-intensive and time-consuming. Though machine learning and artificial intelligence (AI) based particle picking can potentially automate the process, its development is hindered by lack of large, high-quality labelled training data. To address this bottleneck, we present CryoPPP, a large, diverse, expert-curated cryo-EM image dataset for protein particle picking and analysis. It consists of labelled cryo-EM micrographs (images) of 34 representative protein datasets selected from the Electron Microscopy Public Image Archive (EMPIAR). The dataset is 2.6 terabytes and includes 9,893 high-resolution micrographs with labelled protein particle coordinates. The labelling process was rigorously validated through 2D particle class validation and 3D density map validation with the gold standard. The dataset is expected to greatly facilitate the development of both AI and classical methods for automated cryo-EM protein particle picking.

Computational Methods Toward Unbiased Pattern Mining and Structure Determination in Cryo-Electron Tomography Data

Cryo-electron tomography can uniquely probe the native cellular environment for macromolecular structures. Tomograms feature complex data with densities of diverse, densely crowded macromolecular complexes, low signal-to-noise, and artifacts such as the missing wedge effect. Post-processing of this data generally involves isolating regions or particles of interest from tomograms, organizing them into related groups, and rendering final structures through subtomogram averaging. Template-matching and reference-based structure determination are popular analysis methods but are vulnerable to biases and can often require significant user input. Most importantly, these approaches cannot identify novel complexes that reside within the imaged cellular environment. To reliably extract and resolve structures of interest, efficient and unbiased approaches are therefore of great value. This review highlights notable computational software and discusses how they contribute to making automated structural pattern discovery a possibility. Perspectives emphasizing the importance of features for user-friendliness and accessibility are also presented.

From Deep Mutational Mapping of Allosteric Protein Landscapes to Deep Learning of Allostery and Hidden Allosteric Sites: Zooming in on "Allosteric Intersection" of Biochemical and Big Data Approaches

The recent advances in artificial intelligence (AI) and machine learning have driven the design of new expert systems and automated workflows that are able to model complex chemical and biological phenomena. In recent years, machine learning approaches have been developed and actively deployed to facilitate computational and experimental studies of protein dynamics and allosteric mechanisms. In this review, we discuss in detail new developments along two major directions of allosteric research through the lens of data-intensive biochemical approaches and AI-based computational methods. Despite considerable progress in applications of AI methods for protein structure and dynamics studies, the intersection between allosteric regulation, the emerging structural biology technologies and AI approaches remains largely unexplored, calling for the development of AI-augmented integrative structural biology. In this review, we focus on the latest remarkable progress in deep high-throughput mining and comprehensive mapping of allosteric protein landscapes and allosteric regulatory mechanisms as well as on the new developments in AI methods for prediction and characterization of allosteric binding sites on the proteome level. We also discuss new AI-augmented structural biology approaches that expand our knowledge of the universe of protein dynamics and allostery. We conclude with an outlook and highlight the importance of developing an open science infrastructure for machine learning studies of allosteric regulation and validation of computational approaches using integrative studies of allosteric mechanisms. The development of community-accessible tools that uniquely leverage the existing experimental and simulation knowledgebase to enable interrogation of the allosteric functions can provide a much-needed boost to further innovation and integration of experimental and computational technologies empowered by booming AI field.

High-throughput cryo-ET structural pattern mining by unsupervised deep iterative subtomogram clustering

Cryoelectron tomography directly visualizes heterogeneous macromolecular structures in their native and complex cellular environments. However, existing computer-assisted structure sorting approaches are low throughput or inherently limited due to their dependency on available templates and manual labels. Here, we introduce a high-throughput template-and-label-free deep learning approach, Deep Iterative Subtomogram Clustering Approach (DISCA), that automatically detects subsets of homogeneous structures by learning and modeling 3D structural features and their distributions. Evaluation on five experimental cryo-ET datasets shows that an unsupervised deep learning based method can detect diverse structures with a wide range of molecular sizes. This unsupervised detection paves the way for systematic unbiased recognition of macromolecular complexes in situ.

MULTI-TARGET DETECTION WITH ROTATIONS

We consider the multi-target detection problem of estimating a two-dimensional target image from a large noisy measurement image that contains many randomly rotated and translated copies of the target image. Motivated by single-particle cryo-electron microscopy, we focus on the low signal-to-noise regime, where it is difficult to estimate the locations and orientations of the target images in the measurement. Our approach uses autocorrelation analysis to estimate rotationally and translationally invariant features of the target image. We demonstrate that, regardless of the level of noise, our technique can be used to recover the target image when the measurement is sufficiently large.

CryoPPP: A Large Expert-Labelled Cryo-EM Image Dataset for Machine Learning Protein Particle Picking

Cryo-electron microscopy (cryo-EM) is currently the most powerful technique for determining the structures of large protein complexes and assemblies. Picking single-protein particles from cryo-EM micrographs (images) is a key step in reconstructing protein structures. However, the widely used template-based particle picking process is labor-intensive and time-consuming. Though the emerging machine learning-based particle picking can potentially automate the process, its development is severely hindered by lack of large, high-quality, manually labelled training data. Here, we present CryoPPP, a large, diverse, expert-curated cryo-EM image dataset for single protein particle picking and analysis to address this bottleneck. It consists of manually labelled cryo-EM micrographs of 32 non-redundant, representative protein datasets selected from the Electron Microscopy Public Image Archive (EMPIAR). It includes 9,089 diverse, high-resolution micrographs (∼300 cryo-EM images per EMPIAR dataset) in which the coordinates of protein particles were labelled by human experts. The protein particle labelling process was rigorously validated by both 2D particle class validation and 3D density map validation with the gold standard. The dataset is expected to greatly facilitate the development of machine learning and artificial intelligence methods for automated cryo-EM protein particle picking. The dataset and data processing scripts are available at https://github.com/BioinfoMachineLearning/cryoppp.

Multiple-image super-resolution of cryo-electron micrographs based on deep internal learning

Single-particle cryo-electron microscopy (cryo-EM) is a powerful imaging modality capable of visualizing proteins and macromolecular complexes at near-atomic resolution. The low electron-doses used to prevent radiation damage to the biological samples, however, result in images where the power of the noise is 100 times greater than the power of the signal. To overcome these low signal-to-noise ratios (SNRs), hundreds of thousands of particle projections are averaged to determine the three-dimensional structure of the molecule of interest. The sampling requirements of high-resolution imaging impose limitations on the pixel sizes that can be used for acquisition, limiting the size of the field of view and requiring data collection sessions of several days to accumulate sufficient numbers of particles. Meanwhile, recent image super-resolution (SR) techniques based on neural networks have shown state-of-the-art performance on natural images. Building on these advances, here, we present a multiple-image SR algorithm based on deep internal learning designed specifically to work under low-SNR conditions. Our approach leverages the internal image statistics of cryo-EM movies and does not require training on ground-truth data. When applied to single-particle datasets of apoferritin and T20S proteasome, we show that the resolution of the 3D structure obtained from SR micrographs can surpass the limits imposed by the imaging system. Our results indicate that the combination of low magnification imaging with in silico image SR has the potential to accelerate cryo-EM data collection by virtue of including more particles in each exposure and doing so without sacrificing resolution.

Exploring the Structural Variability of Dynamic Biological Complexes by Single-Particle Cryo-Electron Microscopy

Biological macromolecules and assemblies precisely rearrange their atomic 3D structures to execute cellular functions. Understanding the mechanisms by which these molecular machines operate requires insight into the ensemble of structural states they occupy during the functional cycle. Single-particle cryo-electron microscopy (cryo-EM) has become the preferred method to provide near-atomic resolution, structural information about dynamic biological macromolecules elusive to other structure determination methods. Recent advances in cryo-EM methodology have allowed structural biologists not only to probe the structural intermediates of biochemical reactions, but also to resolve different compositional and conformational states present within the same dataset. This article reviews newly developed sample preparation and single-particle analysis (SPA) techniques for high-resolution structure determination of intrinsically dynamic and heterogeneous samples, shedding light upon the intricate mechanisms employed by molecular machines and helping to guide drug discovery efforts.

Bending forces and nucleotide state jointly regulate F-actin structure

ATP-hydrolysis-coupled actin polymerization is a fundamental mechanism of cellular force generation1-3. In turn, force4,5 and actin filament (F-actin) nucleotide state6 regulate actin dynamics by tuning F-actin's engagement of actin-binding proteins through mechanisms that are unclear. Here we show that the nucleotide state of actin modulates F-actin structural transitions evoked by bending forces. Cryo-electron microscopy structures of ADP-F-actin and ADP-Pi-F-actin with sufficient resolution to visualize bound solvent reveal intersubunit interfaces bridged by water molecules that could mediate filament lattice flexibility. Despite extensive ordered solvent differences in the nucleotide cleft, these structures feature nearly identical lattices and essentially indistinguishable protein backbone conformations that are unlikely to be discriminable by actin-binding proteins. We next introduce a machine-learning-enabled pipeline for reconstructing bent filaments, enabling us to visualize both continuous structural variability and side-chain-level detail. Bent F-actin structures reveal rearrangements at intersubunit interfaces characterized by substantial alterations of helical twist and deformations in individual protomers, transitions that are distinct in ADP-F-actin and ADP-Pi-F-actin. This suggests that phosphate rigidifies actin subunits to alter the bending structural landscape of F-actin. As bending forces evoke nucleotide-state dependent conformational transitions of sufficient magnitude to be detected by actin-binding proteins, we propose that actin nucleotide state can serve as a co-regulator of F-actin mechanical regulation.

Bioinformatics in bioscience and bioengineering: Recent advances, applications, and perspectives

Recent advances have led to the emergence of highly comprehensive and analytical approaches, such as omics analysis and high-resolution, time-resolved bioimaging analysis. These technologies have made it possible to obtain vast data from a single measurement. Subsequently, large datasets have pioneered the data-driven approach, an alternative to the traditional hypothesis-testing system, for researchers. However, processing, interpreting, and elucidating enormous datasets is no longer possible without computation. Bioinformatics is a field that has developed over long periods, intending to understand biological phenomena using methods collected from information science and statistics, thus solving this proposed research challenge. This review presents the latest methodologies and applications in sequencing, imaging, and mass spectrometry that were developed using bioinformatics. We presented the features of individual techniques and outlines in each part, avoiding the use of complex algorithms and formulas to allow beginning researchers to understand an overview. In the section on sequencing, we focused on comparative genomic, transcriptomic, and bacterial microbiome analyses, which are frequently used as applications of next-generation sequencing. Bioinformatic methods for handling sequence data and case studies were described. In the section on imaging, we introduced the analytical methods and microscopy imaging informatics techniques used in animal cell biology and plant physiology. We introduce informatics technologies for maximizing the value of measured data, including predicting the structure of unknown molecules and untargeted analysis in the section on mass spectrometry. Finally, we discuss the future outlook of this field. We anticipate that this review will assist biologists in using bioinformatics more effectively.

Emerging Themes in CryoEM─Single Particle Analysis Image Processing

Cryo-electron microscopy (CryoEM) has become a vital technique in structural biology. It is an interdisciplinary field that takes advantage of advances in biochemistry, physics, and image processing, among other disciplines. Innovations in these three basic pillars have contributed to the boosting of CryoEM in the past decade. This work reviews the main contributions in image processing to the current reconstruction workflow of single particle analysis (SPA) by CryoEM. Our review emphasizes the time evolution of the algorithms across the different steps of the workflow differentiating between two groups of approaches: analytical methods and deep learning algorithms. We present an analysis of the current state of the art. Finally, we discuss the emerging problems and challenges still to be addressed in the evolution of CryoEM image processing methods in SPA.

Higher resolution in cryo-EM by the combination of macromolecular prior knowledge and image-processing tools

Single-particle cryo-electron microscopy has become a powerful technique for the 3D structure determination of biological molecules. The last decade has seen an astonishing development of both hardware and software, and an exponential growth of new structures obtained at medium-high resolution. However, the knowledge accumulated in this field over the years has hardly been utilized as feedback in the reconstruction of new structures. In this context, this article explores the use of the deep-learning approach deepEMhancer as a regularizer in the RELION refinement process. deepEMhancer introduces prior information derived from macromolecular structures, and contributes to noise reduction and signal enhancement, as well as a higher degree of isotropy. These features have a direct effect on image alignment and reduction of overfitting during iterative refinement. The advantages of this combination are demonstrated for several membrane proteins, for which it is especially useful because of their high disorder and flexibility.

Volumetric macromolecule identification in cryo-electron tomograms using capsule networks

Background

Despite recent advances in cellular cryo-electron tomography (CET), developing automated tools for macromolecule identification in submolecular resolution remains challenging due to the lack of annotated data and high structural complexities. To date, the extent of the deep learning methods constructed for this problem is limited to conventional Convolutional Neural Networks (CNNs). Identifying macromolecules of different types and sizes is a tedious and time-consuming task. In this paper, we employ a capsule-based architecture to automate the task of macromolecule identification, that we refer to as 3D-UCaps. In particular, the architecture is composed of three components: feature extractor, capsule encoder, and CNN decoder. The feature extractor converts voxel intensities of input sub-tomograms to activities of local features. The encoder is a 3D Capsule Network (CapsNet) that takes local features to generate a low-dimensional representation of the input. Then, a 3D CNN decoder reconstructs the sub-tomograms from the given representation by upsampling.

Results

We performed binary and multi-class localization and identification tasks on synthetic and experimental data. We observed that the 3D-UNet and the 3D-UCaps had an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1-$$\end{document}F1-score mostly above 60% and 70%, respectively, on the test data. In both network architectures, we observed degradation of at least 40% in the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1-score when identifying very small particles (PDB entry 3GL1) compared to a large particle (PDB entry 4D8Q). In the multi-class identification task of experimental data, 3D-UCaps had an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1-score of 91% on the test data in contrast to 64% of the 3D-UNet. The better \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1-score of 3D-UCaps compared to 3D-UNet is obtained by a higher precision score. We speculate this to be due to the capsule network employed in the encoder. To study the effect of the CapsNet-based encoder architecture further, we performed an ablation study and perceived that the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1-score is boosted as network depth is increased which is in contrast to the previously reported results for the 3D-UNet. To present a reproducible work, source code, trained models, data as well as visualization results are made publicly available.

Conclusion

Quantitative and qualitative results show that 3D-UCaps successfully perform various downstream tasks including identification and localization of macromolecules and can at least compete with CNN architectures for this task. Given that the capsule layers extract both the existence probability and the orientation of the molecules, this architecture has the potential to lead to representations of the data that are better interpretable than those of 3D-UNet.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-022-04901-w.

Artificial Intelligence in Cryo-Electron Microscopy

Cryo-electron microscopy (cryo-EM) has become an unrivaled tool for determining the structure of macromolecular complexes. The biological function of macromolecular complexes is inextricably tied to the flexibility of these complexes. Single particle cryo-EM can reveal the conformational heterogeneity of a biochemically pure sample, leading to well-founded mechanistic hypotheses about the roles these complexes play in biology. However, the processing of increasingly large, complex datasets using traditional data processing strategies is exceedingly expensive in both user time and computational resources. Current innovations in data processing capitalize on artificial intelligence (AI) to improve the efficiency of data analysis and validation. Here, we review new tools that use AI to automate the data analysis steps of particle picking, 3D map reconstruction, and local resolution determination. We discuss how the application of AI moves the field forward, and what obstacles remain. We also introduce potential future applications of AI to use cryo-EM in understanding protein communities in cells.

A Transfer Learning-Based Classification Model for Particle Pruning in Cryo-Electron Microscopy

The cryo-electron microscopy (cryo-EM) single-particle analysis requires tens of thousands of particle projections to reveal structural information of macromolecular complexes. However, due to the low signal-to-noise ratio and the presence of high contrast artifacts and contaminants in the micrographs, the semiautomatic and fully automatic particle picking algorithms tend to suffer from high false-positive rates, which degrades the confidence of structure determination. In this study, we introduce PickerOptimizer (PO), a transfer learning-based classification neural network for particle pruning in cryo-EM, as an additional strategy to complement the current automated particle picking algorithms. To achieve high classification performance with minimal human intervention, we adopted two key strategies: (1) utilizing the transfer learning techniques to train the convolutional neural network, where the knowledge gained from public classification datasets is applied to the field of cryo-EM. (2) Designing a multiloss strategy, a combination of multiple loss functions, to guide the optimization of the network parameters. To reduce the domain shift between cryo-EM images and natural images for pretraining, we build the first image classification dataset for cryo-EM, which contains positive and negative samples collected from EMPIAR entries. The PO is tested on 14 public experimental datasets, achieving accuracy and F1 scores above 95% in most cases. Furthermore, three case studies are provided to verify the model performance by applying PO on problematic particle selections, showing that our algorithm achieved better or comparable performance compared with other particle pruning strategies.

Recent Technical Advances in Sample Preparation for Single-Particle Cryo-EM

Cryo-sample preparation is a vital step in the process of obtaining high-resolution structures of macromolecules by using the single-particle cryo–electron microscopy (cryo-EM) method; however, cryo-sample preparation is commonly hampered by high uncertainty and low reproducibility. Specifically, the existence of air-water interfaces during the sample vitrification process could cause protein denaturation and aggregation, complex disassembly, adoption of preferred orientations, and other serious problems affecting the protein particles, thereby making it challenging to pursue high-resolution 3D reconstruction. Therefore, sample preparation has emerged as a critical research topic, and several new methods for application at various preparation stages have been proposed to overcome the aforementioned hurdles. Here, we summarize the methods developed for enhancing the quality of cryo-samples at distinct stages of sample preparation, and we offer insights for developing future strategies based on diverse viewpoints. We anticipate that cryo-sample preparation will no longer be a limiting step in the single-particle cryo-EM field as increasing numbers of methods are developed in the near future, which will ultimately benefit the entire research community.

VP-Detector: A 3D multi-scale dense convolutional neural network for macromolecule localization and classification in cryo-electron tomograms

BACKGROUND AND OBJECTIVE

Cryo-electron tomography (cryo-ET) with subtomogram averaging (STA) is indispensable when studying macromolecule structures and functions in their native environments. Due to the low signal-to-noise ratio, the missing wedge artifacts in tomographic reconstructions, and multiple macromolecules of varied shapes and sizes, macromolecule localization and classification remain challenging. To tackle this bottleneck problem for structural determination by STA, we design an accurate macromolecule localization and classification method named voxelwise particle detector (VP-Detector).

METHODS

VP-Detector is a two-stage particle detection method based on a 3D multiscale dense convolutional neural network (3D MSDNet). The proposed network uses 3D hybrid dilated convolution (3D HDC) to avoid the resolution loss caused by scaling operations. Meanwhile, it uses 3D dense connectivity to encourage the reuse of feature maps to reduce trainable parameters. In addition, the weighted focal loss is proposed to focus more attention on difficult samples and rare classes, which relieves the class imbalance caused by multiple particles of various sizes. The performance of VP-Detector is evaluated on both simulated and real-world tomograms, and it shows that VP-Detector outperforms state-of-the-art methods.

RESULTS

The experiments show that VP-Detector outperforms the state-of-the-art methods on particle localization with an F1-score of 0.951 and a precision of 0.978. In addition, VP-Detector can replace manual particle picking in experiment on the real-world tomograms. Furthermore, it performs well in classifying large-, medium-, and small-weight proteins with accuracies of 1, 0.95, and 0.82, respectively. Finally, ablation studies demonstrate the effectiveness of 3D HDC, 3D dense connectivity, weighted focal loss, and training on small training sets.

CONCLUSIONS

VP-Detector can achieve high accuracy in particle detection with few trainable parameters and support training on small datasets. It can also relieve the class imbalance caused by multiple particles with various shapes and sizes.

EPicker is an exemplar-based continual learning approach for knowledge accumulation in cryoEM particle picking

Deep learning is a popular method for facilitating particle picking in single-particle cryo-electron microscopy (cryo-EM), which is essential for developing automated processing pipelines. Most existing deep learning algorithms for particle picking rely on supervised learning where the features to be identified must be provided through a training procedure. However, the generalization performance of these algorithms on unseen datasets with different features is often unpredictable. In addition, while they perform well on the latest training datasets, these algorithms often fail to maintain the knowledge of old particles. Here, we report an exemplar-based continual learning approach, which can accumulate knowledge from the new dataset into the model by training an existing model on only a few new samples without catastrophic forgetting of old knowledge, implemented in a program called EPicker. Therefore, the ability of EPicker to identify bio-macromolecules can be expanded by continuously learning new knowledge during routine particle picking applications. Powered by the improved training strategy, EPicker is designed to pick not only protein particles but also general biological objects such as vesicles and fibers.

Many existing deep learning algorithms for particle picking are not predictable on unseen datasets. Here the authors report an exemplar-based continual learning approach, EPicker, enabling accumulation of new knowledge of cryoEM particle picking without catastrophic forgetting of old knowledge.

Applications of deep learning in electron microscopy

We review the growing use of machine learning in electron microscopy (EM) driven in part by the availability of fast detectors operating at kiloHertz frame rates leading to large data sets that cannot be processed using manually implemented algorithms. We summarize the various network architectures and error metrics that have been applied to a range of EM-related problems including denoising and inpainting. We then provide a review of the application of these in both physical and life sciences, highlighting how conventional networks and training data have been specifically modified for EM.

Machine Learning for Structure Determination in Single-Particle Cryo-Electron Microscopy: A Systematic Review

Recently, single-particle cryo-electron microscopy (cryo-EM) has become an indispensable method for determining macromolecular structures at high resolution to deeply explore the relevant molecular mechanism. Its recent breakthrough is mainly because of the rapid advances in hardware and image processing algorithms, especially machine learning. As an essential support of single-particle cryo-EM, machine learning has powered many aspects of structure determination and greatly promoted its development. In this article, we provide a systematic review of the applications of machine learning in this field. Our review begins with a brief introduction of single-particle cryo-EM, followed by the specific tasks and challenges of its image processing. Then, focusing on the workflow of structure determination, we describe relevant machine learning algorithms and applications at different steps, including particle picking, 2-D clustering, 3-D reconstruction, and other steps. As different tasks exhibit distinct characteristics, we introduce the evaluation metrics for each task and summarize their dynamics of technology development. Finally, we discuss the open issues and potential trends in this promising field.

End-to-end orientation estimation from 2D cryo-EM images

Cryo-electron microscopy (cryo-EM) is a Nobel Prize-winning technique for determining high-resolution 3D structures of biological macromolecules. A 3D structure is reconstructed from hundreds of thousands of noisy 2D projection images. However, existing 3D reconstruction methods are still time-consuming, and one of the major computational bottlenecks is recovering the unknown orientation of the particle in each 2D image. The dominant methods typically exploit an expensive global search on each image to estimate the missing orientations. Here, a novel end-to-end supervised learning method is introduced to directly recover the missing orientations from 2D cryo-EM images. A neural network is used to approximate the mapping from images to orientations. A robust loss function is proposed for optimizing the parameters of the network, which can handle both asymmetric and symmetric 3D structures. Experiments on synthetic data sets with various symmetry types confirm that the neural network is capable of recovering orientations from 2D cryo-EM images, and the results on a real cryo-EM data set further demonstrate its potential under more challenging imaging conditions.

Applications of Cryo-EM in small molecule and biologics drug design

Electron cryo-microscopy (cryo-EM) is a powerful technique for the structural characterization of biological macromolecules, enabling high-resolution analysis of targets once inaccessible to structural interrogation. In recent years, pharmaceutical companies have begun to utilize cryo-EM for structure-based drug design. Structural analysis of integral membrane proteins, which comprise a large proportion of druggable targets and pose particular challenges for X-ray crystallography, by cryo-EM has enabled insights into important drug target families such as G protein-coupled receptors (GPCRs), ion channels, and solute carrier (SLCs) proteins. Structural characterization of biologics, such as vaccines, viral vectors, and gene therapy agents, has also become significantly more tractable. As a result, cryo-EM has begun to make major impacts in bringing critical therapeutics to market. In this review, we discuss recent instructive examples of impacts from cryo-EM in therapeutics design, focusing largely on its implementation at Pfizer. We also discuss the opportunities afforded by emerging technological advances in cryo-EM, and the prospects for future development of the technique.

Deep learning improves macromolecule identification in 3D cellular cryo-electron tomograms

Cryogenic electron tomography (cryo-ET) visualizes the 3D spatial distribution of macromolecules at nanometer resolution inside native cells. However, automated identification of macromolecules inside cellular tomograms is challenged by noise and reconstruction artifacts, as well as the presence of many molecular species in the crowded volumes. Here, we present DeepFinder, a computational procedure that uses artificial neural networks to simultaneously localize multiple classes of macromolecules. Once trained, the inference stage of DeepFinder is faster than template matching and performs better than other competitive deep learning methods at identifying macromolecules of various sizes in both synthetic and experimental datasets. On cellular cryo-ET data, DeepFinder localized membrane-bound and cytosolic ribosomes (roughly 3.2 MDa), ribulose 1,5-bisphosphate carboxylase-oxygenase (roughly 560 kDa soluble complex) and photosystem II (roughly 550 kDa membrane complex) with an accuracy comparable to expert-supervised ground truth annotations. DeepFinder is therefore a promising algorithm for the semiautomated analysis of a wide range of molecular targets in cellular tomograms.

Challenges in sample preparation and structure determination of amyloids by cryo-EM

Amyloids share a common architecture but play disparate biological roles in processes ranging from bacterial defense mechanisms to protein misfolding diseases. Their structures are highly polymorphic, which makes them difficult to study by X-ray diffraction or NMR spectroscopy. Our understanding of amyloid structures is due in large part to recent advances in the field of cryo-EM, which allows for determining the polymorphs separately. In this review, we highlight the main stepping stones leading to the substantial number of high-resolution amyloid fibril structures known today as well as recent developments regarding automation and software in cryo-EM. We discuss that sample preparation should move closer to physiological conditions to understand how amyloid aggregation and disease are linked. We further highlight new approaches to address heterogeneity and polymorphism of amyloid fibrils in EM image processing and give an outlook to the upcoming challenges in researching the structural biology of amyloids.

DeepEMhancer: a deep learning solution for cryo-EM volume post-processing

Cryo-EM maps are valuable sources of information for protein structure modeling. However, due to the loss of contrast at high frequencies, they generally need to be post-processed to improve their interpretability. Most popular approaches, based on global B-factor correction, suffer from limitations. For instance, they ignore the heterogeneity in the map local quality that reconstructions tend to exhibit. Aiming to overcome these problems, we present DeepEMhancer, a deep learning approach designed to perform automatic post-processing of cryo-EM maps. Trained on a dataset of pairs of experimental maps and maps sharpened using their respective atomic models, DeepEMhancer has learned how to post-process experimental maps performing masking-like and sharpening-like operations in a single step. DeepEMhancer was evaluated on a testing set of 20 different experimental maps, showing its ability to reduce noise levels and obtain more detailed versions of the experimental maps. Additionally, we illustrated the benefits of DeepEMhancer on the structure of the SARS-CoV-2 RNA polymerase.

Sanchez-Garcia et al. present DeepEMhancer, a deep learning-based method that can automatically perform post-processing of raw cryo-electron microscopy density maps. The authors report that DeepEMhancer globally improves local quality of density maps, and may represent a useful tool for novel structures where PDB models are not readily available.

Synthetic Image Rendering Solves Annotation Problem in Deep Learning Nanoparticle Segmentation

Nanoparticles occur in various environments as a consequence of man-made processes, which raises concerns about their impact on the environment and human health. To allow for proper risk assessment, a precise and statistically relevant analysis of particle characteristics (such as size, shape, and composition) is required that would greatly benefit from automated image analysis procedures. While deep learning shows impressive results in object detection tasks, its applicability is limited by the amount of representative, experimentally collected and manually annotated training data. Here, an elegant, flexible, and versatile method to bypass this costly and tedious data acquisition process is presented. It shows that using a rendering software allows to generate realistic, synthetic training data to train a state-of-the art deep neural network. Using this approach, a segmentation accuracy can be derived that is comparable to man-made annotations for toxicologically relevant metal-oxide nanoparticle ensembles which were chosen as examples. The presented study paves the way toward the use of deep learning for automated, high-throughput particle detection in a variety of imaging techniques such as in microscopies and spectroscopies, for a wide range of applications, including the detection of micro- and nanoplastic particles in water and tissue samples.

Evaluation of automated particle picking for cryogenic electron microscopy using high-precision transmission electron microscope simulation based on a multi-slice method

This work describes the GRIPS automated particle-picking software for cryogenic electron microscopy and the evaluation of this software using elbis, a high-precision transmission electron microscope (TEM) image simulator. The goal was to develop a method that can pick particles under a small defocus condition where the particles are not clearly visible or under a condition where the particles are exhibiting preferred orientation. The proposed method handles these issues by repeatedly performing three processes, namely extraction, two-dimensional classification and positioning, and by introducing mask processing to exclude areas with particles that have already been picked. TEM images for evaluation were generated with a high-precision TEM image simulator. TEM images containing both particles and amorphous ice were simulated by randomly placing O atoms in the specimen. The experimental results indicate that the proposed method can be used to pick particles correctly under a relatively small defocus condition. Moreover, the results show that the mask processing introduced in the proposed method is valid for particles exhibiting preferred orientation. It is further shown that the proposed method is applicable to data collected from real samples.

Deep Learning-Based Advances in Protein Structure Prediction

Obtaining an accurate description of protein structure is a fundamental step toward understanding the underpinning of biology. Although recent advances in experimental approaches have greatly enhanced our capabilities to experimentally determine protein structures, the gap between the number of protein sequences and known protein structures is ever increasing. Computational protein structure prediction is one of the ways to fill this gap. Recently, the protein structure prediction field has witnessed a lot of advances due to Deep Learning (DL)-based approaches as evidenced by the success of AlphaFold2 in the most recent Critical Assessment of protein Structure Prediction (CASP14). In this article, we highlight important milestones and progresses in the field of protein structure prediction due to DL-based methods as observed in CASP experiments. We describe advances in various steps of protein structure prediction pipeline viz. protein contact map prediction, protein distogram prediction, protein real-valued distance prediction, and Quality Assessment/refinement. We also highlight some end-to-end DL-based approaches for protein structure prediction approaches. Additionally, as there have been some recent DL-based advances in protein structure determination using Cryo-Electron (Cryo-EM) microscopy based, we also highlight some of the important progress in the field. Finally, we provide an outlook and possible future research directions for DL-based approaches in the protein structure prediction arena.

Detecting Protein Communities in Native Cell Extracts by Machine Learning: A Structural Biologist's Perspective

Native cell extracts hold great promise for understanding the molecular structure of ordered biological systems at high resolution. This is because higher-order biomolecular interactions, dubbed as protein communities, may be retained in their (near-)native state, in contrast to extensively purifying or artificially overexpressing the proteins of interest. The distinct machine-learning approaches are applied to discover protein-protein interactions within cell extracts, reconstruct dedicated biological networks, and report on protein community members from various organisms. Their validation is also important, e.g., by the cross-linking mass spectrometry or cell biology methods. In addition, the cell extracts are amenable to structural analysis by cryo-electron microscopy (cryo-EM), but due to their inherent complexity, sorting structural signatures of protein communities derived by cryo-EM comprises a formidable task. The application of image-processing workflows inspired by machine-learning techniques would provide improvements in distinguishing structural signatures, correlating proteomic and network data to structural signatures and subsequently reconstructed cryo-EM maps, and, ultimately, characterizing unidentified protein communities at high resolution. In this review article, we summarize recent literature in detecting protein communities from native cell extracts and identify the remaining challenges and opportunities. We argue that the progress in, and the integration of, machine learning, cryo-EM, and complementary structural proteomics approaches would provide the basis for a multi-scale molecular description of protein communities within native cell extracts.

Deep learning in electron microscopy

Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy.

DeepAlign, a 3D alignment method based on regionalized deep learning for Cryo-EM

Cryo Electron Microscopy (Cryo-EM) is currently one of the main tools to reveal the structural information of biological specimens at high resolution. Despite the great development of the techniques involved to solve the biological structures with Cryo-EM in the last years, the reconstructed 3D maps can present lower resolution due to errors committed while processing the information acquired by the microscope. One of the main problems comes from the 3D alignment step, which is an error-prone part of the reconstruction workflow due to the very low signal-to-noise ratio (SNR) common in Cryo-EM imaging. In fact, as we will show in this work, it is not unusual to find a disagreement in the alignment parameters in approximately 20-40% of the processed images, when outputs of different alignment algorithms are compared. In this work, we present a novel method to align sets of single particle images in the 3D space, called DeepAlign. Our proposal is based on deep learning networks that have been successfully used in plenty of problems in image classification. Specifically, we propose to design several deep neural networks on a regionalized basis to classify the particle images in sub-regions and, then, make a refinement of the 3D alignment parameters only inside that sub-region. We show that this method results in accurately aligned images, improving the Fourier shell correlation (FSC) resolution obtained with other state-of-the-art methods while decreasing computational time.

Workflow towards automated segmentation of agglomerated, non-spherical particles from electron microscopy images using artificial neural networks

We present a workflow for obtaining fully trained artificial neural networks that can perform automatic particle segmentations of agglomerated, non-spherical nanoparticles from scanning electron microscopy images "from scratch", without the need for large training data sets of manually annotated images. The whole process only requires about 15 min of hands-on time by a user and can typically be finished within less than 12 h when training on a single graphics card (GPU). After training, SEM image analysis can be carried out by the artificial neural network within seconds. This is achieved by using unsupervised learning for most of the training dataset generation, making heavy use of generative adversarial networks and especially unpaired image-to-image translation via cycle-consistent adversarial networks. We compare the segmentation masks obtained with our suggested workflow qualitatively and quantitatively to state-of-the-art methods using various metrics. Finally, we used the segmentation masks for automatically extracting particle size distributions from the SEM images of TiO₂ particles, which were in excellent agreement with particle size distributions obtained manually but could be obtained in a fraction of the time.

CASSPER is a semantic segmentation-based particle picking algorithm for single-particle cryo-electron microscopy

Particle identification and selection, which is a prerequisite for high-resolution structure determination of biological macromolecules via single-particle cryo-electron microscopy poses a major bottleneck for automating the steps of structure determination. Here, we present a generalized deep learning tool, CASSPER, for the automated detection and isolation of protein particles in transmission microscope images. This deep learning tool uses Semantic Segmentation and a collection of visually prepared training samples to capture the differences in the transmission intensities of protein, ice, carbon, and other impurities found in the micrograph. CASSPER is a semantic segmentation based method that does pixel-level classification and completely eliminates the need for manual particle picking. Integration of Contrast Limited Adaptive Histogram Equalization (CLAHE) in CASSPER enables high-fidelity particle detection in micrographs with variable ice thickness and contrast. A generalized CASSPER model works with high efficiency on unseen datasets and can potentially pick particles on-the-fly, enabling data processing automation.

DRPnet: automated particle picking in cryo-electron micrographs using deep regression

BACKGROUND

Identification and selection of protein particles in cryo-electron micrographs is an important step in single particle analysis. In this study, we developed a deep learning-based particle picking network to automatically detect particle centers from cryoEM micrographs. This is a challenging task due to the nature of cryoEM data, having low signal-to-noise ratios with variable particle sizes, shapes, distributions, grayscale variations as well as other undesirable artifacts.

RESULTS

We propose a double convolutional neural network (CNN) cascade for automated detection of particles in cryo-electron micrographs. This approach, entitled Deep Regression Picker Network or "DRPnet", is simple but very effective in recognizing different particle sizes, shapes, distributions and grayscale patterns corresponding to 2D views of 3D particles. Particles are detected by the first network, a fully convolutional regression network (FCRN), which maps the particle image to a continuous distance map that acts like a probability density function of particle centers. Particles identified by FCRN are further refined to reduce false particle detections by the second classification CNN. DRPnet's first CNN pretrained with only a single cryoEM dataset can be used to detect particles from different datasets without retraining. Compared to RELION template-based autopicking, DRPnet results in better particle picking performance with drastically reduced user interactions and processing time. DRPnet also outperforms the state-of-the-art particle picking networks in terms of the supervised detection evaluation metrics recall, precision, and F-measure. To further highlight quality of the picked particle sets, we compute and present additional performance metrics assessing the resulting 3D reconstructions such as number of 2D class averages, efficiency/angular coverage, Rosenthal-Henderson plots and local/global 3D reconstruction resolution.

CONCLUSION

DRPnet shows greatly improved time-savings to generate an initial particle dataset compared to manual picking, followed by template-based autopicking. Compared to other networks, DRPnet has equivalent or better performance. DRPnet excels on cryoEM datasets that have low contrast or clumped particles. Evaluating other performance metrics, DRPnet is useful for higher resolution 3D reconstructions with decreased particle numbers or unknown symmetry, detecting particles with better angular orientation coverage.