Automatic localization and identification of mitochondria in cellular electron cryo-tomography using faster-RCNN

Semi-automated, high-content imaging of drug transporter knockout sea urchin (Lytechinus pictus) embryos

A defining feature of sea urchins is their extreme fecundity. Urchins produce millions of transparent, synchronously developing embryos, ideal for spatial and temporal analysis of development. This biological feature has been effectively utilized for ensemble measurement of biochemical changes. However, it has been underutilized in imaging studies, where single embryo measurements are used. Here we present an example of how stable genetics and high content imaging, along with machine learning-based image analysis, can be used to exploit the fecundity and synchrony of sea urchins in imaging-based drug screens. Building upon our recently created sea urchin ABCB1 knockout line, we developed a high-throughput assay to probe the role of this drug transporter in embryos. We used high content imaging to compare accumulation and toxicity of canonical substrates and inhibitors of the transporter, including fluorescent molecules and antimitotic cancer drugs, in homozygous knockout and wildtype embryos. To measure responses from the resulting image data, we used a nested convolutional neural network, which rapidly classified embryos according to fluorescence or cell division. This approach identified sea urchin embryos with 99.8% accuracy and determined two-cell and aberrant embryos with 96.3% and 89.1% accuracy, respectively. The results revealed that ABCB1 knockout embryos accumulated the transporter substrate calcein 3.09 times faster than wildtypes. Similarly, knockouts were 4.71 and 3.07 times more sensitive to the mitotic poisons vinblastine and taxol. This study paves the way for large scale pharmacological screens in the sea urchin embryo.

DeepETPicker: Fast and accurate 3D particle picking for cryo-electron tomography using weakly supervised deep learning

To solve three-dimensional structures of biological macromolecules in situ, large numbers of particles often need to be picked from cryo-electron tomograms. However, adoption of automated particle-picking methods remains limited because of their technical limitations. To overcome the limitations, we develop DeepETPicker, a deep learning model for fast and accurate picking of particles from cryo-electron tomograms. Training of DeepETPicker requires only weak supervision with low numbers of simplified labels, reducing the burden of manual annotation. The simplified labels combined with the customized and lightweight model architecture of DeepETPicker and accelerated pooling enable substantial performance improvement. When tested on simulated and real tomograms, DeepETPicker outperforms the competing state-of-the-art methods by achieving the highest overall accuracy and speed, which translate into higher authenticity and coordinates accuracy of picked particles and higher resolutions of final reconstruction maps. DeepETPicker is provided in open source with a user-friendly interface to support cryo-electron tomography in situ.

Diagnosis of primary clear cell carcinoma of the liver based on Faster region-based convolutional neural network

BACKGROUND

Distinguishing between primary clear cell carcinoma of the liver (PCCCL) and common hepatocellular carcinoma (CHCC) through traditional inspection methods before the operation is difficult. This study aimed to establish a Faster region-based convolutional neural network (RCNN) model for the accurate differential diagnosis of PCCCL and CHCC.

METHODS

In this study, we collected the data of 62 patients with PCCCL and 1079 patients with CHCC in Beijing YouAn Hospital from June 2012 to May 2020. A total of 109 patients with CHCC and 42 patients with PCCCL were randomly divided into the training validation set and the test set in a ratio of 4:1.The Faster RCNN was used for deep learning of patients' data in the training validation set, and established a convolutional neural network model to distinguish PCCCL and CHCC. The accuracy, average precision, and the recall of the model for diagnosing PCCCL and CHCC were used to evaluate the detection performance of the Faster RCNN algorithm.

RESULTS

A total of 4392 images of 121 patients (1032 images of 33 patients with PCCCL and 3360 images of 88 patients with CHCC) were uesd in test set for deep learning and establishing the model, and 1072 images of 30 patients (320 images of nine patients with PCCCL and 752 images of 21 patients with CHCC) were used to test the model. The accuracy of the model for accurately diagnosing PCCCL and CHCC was 0.962 (95% confidence interval [CI]: 0.931-0.992). The average precision of the model for diagnosing PCCCL was 0.908 (95% CI: 0.823-0.993) and that for diagnosing CHCC was 0.907 (95% CI: 0.823-0.993). The recall of the model for diagnosing PCCCL was 0.951 (95% CI: 0.916-0.985) and that for diagnosing CHCC was 0.960 (95% CI: 0.854-0.962). The time to make a diagnosis using the model took an average of 4 s for each patient.

CONCLUSION

The Faster RCNN model can accurately distinguish PCCCL and CHCC. This model could be important for clinicians to make appropriate treatment plans for patients with PCCCL or CHCC.

Deep Learning-Based Precision Analysis for Acrosome Reaction by Modification of Plasma Membrane in Boar Sperm

The analysis of AR is widely used to detect loss of acrosome in sperm, but the subjective decisions of experts affect the accuracy of the examination. Therefore, we develop an ARCS for objectivity and consistency of analysis using convolutional neural networks (CNNs) trained with various magnification images. Our models were trained on 215 microscopic images at 400× and 438 images at 1000× magnification using the ResNet 50 and Inception-ResNet v2 architectures. These models distinctly recognized micro-changes in the PM of AR sperms. Moreover, the Inception-ResNet v2-based ARCS achieved a mean average precision of over 97%. Our system's calculation of the AR ratio on the test dataset produced results similar to the work of the three experts and could do so more quickly. Our model streamlines sperm detection and AR status determination using a CNN-based approach, replacing laborious tasks and expert assessments. The ARCS offers consistent AR sperm detection, reduced human error, and decreased working time. In conclusion, our study suggests the feasibility and benefits of using a sperm diagnosis artificial intelligence assistance system in routine practice scenarios.

Organoids revealed: morphological analysis of the profound next generation in-vitro model with artificial intelligence

In modern terminology, "organoids" refer to cells that grow in a specific three-dimensional (3D) environment in vitro, sharing similar structures with their source organs or tissues. Observing the morphology or growth characteristics of organoids through a microscope is a commonly used method of organoid analysis. However, it is difficult, time-consuming, and inaccurate to screen and analyze organoids only manually, a problem which cannot be easily solved with traditional technology. Artificial intelligence (AI) technology has proven to be effective in many biological and medical research fields, especially in the analysis of single-cell or hematoxylin/eosin stained tissue slices. When used to analyze organoids, AI should also provide more efficient, quantitative, accurate, and fast solutions. In this review, we will first briefly outline the application areas of organoids and then discuss the shortcomings of traditional organoid measurement and analysis methods. Secondly, we will summarize the development from machine learning to deep learning and the advantages of the latter, and then describe how to utilize a convolutional neural network to solve the challenges in organoid observation and analysis. Finally, we will discuss the limitations of current AI used in organoid research, as well as opportunities and future research directions.

Graphic abstract

Structure Detection in Three-Dimensional Cellular Cryoelectron Tomograms by Reconstructing Two-Dimensional Annotated Tilt Series

The revolutionary technique cryoelectron tomography (cryo-ET) enables imaging of cellular structure and organization in a near-native environment at submolecular resolution, which is vital to subsequent data analysis and modeling. The conventional structure detection process first reconstructs the three-dimensional (3D) tomogram from a series of two-dimensional (2D) projections and then directly detects subcellular components found within the tomogram. However, this process is challenging due to potential structural information loss during the tomographic reconstruction and the limited scope of existing methods since most major state-of-the-art object detection methods are designed for 2D rather than 3D images. Therefore, in this article, as an alternative approach to complement the conventional process, we propose a novel 2D-to-3D framework that detects structures within 2D projection images before reconstructing the results back to 3D. We implemented the proposed framework as three specific algorithms for three individual tasks: semantic segmentation, edge detection, and object localization. As experimental validation of the 2D-to-3D framework for cryo-ET data, we applied the algorithms to the segmentation of mitochondrial calcium phosphate granules, detection of spherical edges, and localization of mitochondria. Quantitative and qualitative results show better performance for prediction tasks of segmentation on the 2D projections and promising performance on object localization and edge detection, paving the way for future studies in the exploration of cryo-ET for in situ structural biology.

VP-Detector: A 3D multi-scale dense convolutional neural network for macromolecule localization and classification in cryo-electron tomograms

BACKGROUND AND OBJECTIVE

Cryo-electron tomography (cryo-ET) with subtomogram averaging (STA) is indispensable when studying macromolecule structures and functions in their native environments. Due to the low signal-to-noise ratio, the missing wedge artifacts in tomographic reconstructions, and multiple macromolecules of varied shapes and sizes, macromolecule localization and classification remain challenging. To tackle this bottleneck problem for structural determination by STA, we design an accurate macromolecule localization and classification method named voxelwise particle detector (VP-Detector).

METHODS

VP-Detector is a two-stage particle detection method based on a 3D multiscale dense convolutional neural network (3D MSDNet). The proposed network uses 3D hybrid dilated convolution (3D HDC) to avoid the resolution loss caused by scaling operations. Meanwhile, it uses 3D dense connectivity to encourage the reuse of feature maps to reduce trainable parameters. In addition, the weighted focal loss is proposed to focus more attention on difficult samples and rare classes, which relieves the class imbalance caused by multiple particles of various sizes. The performance of VP-Detector is evaluated on both simulated and real-world tomograms, and it shows that VP-Detector outperforms state-of-the-art methods.

RESULTS

The experiments show that VP-Detector outperforms the state-of-the-art methods on particle localization with an F1-score of 0.951 and a precision of 0.978. In addition, VP-Detector can replace manual particle picking in experiment on the real-world tomograms. Furthermore, it performs well in classifying large-, medium-, and small-weight proteins with accuracies of 1, 0.95, and 0.82, respectively. Finally, ablation studies demonstrate the effectiveness of 3D HDC, 3D dense connectivity, weighted focal loss, and training on small training sets.

CONCLUSIONS

VP-Detector can achieve high accuracy in particle detection with few trainable parameters and support training on small datasets. It can also relieve the class imbalance caused by multiple particles with various shapes and sizes.

High-Resolution Computer Tomography Image Features of Lungs for Patients with Type 2 Diabetes under the Faster-Region Recurrent Convolutional Neural Network Algorithm

The objective of this study was to adopt the high-resolution computed tomography (HRCT) technology based on the faster-region recurrent convolutional neural network (Faster-RCNN) algorithm to evaluate the lung infection in patients with type 2 diabetes, so as to analyze the application value of imaging features in the assessment of pulmonary disease in type 2 diabetes. In this study, 176 patients with type 2 diabetes were selected as the research objects, and they were divided into different groups based on gender, course of disease, age, glycosylated hemoglobin level (HbA1c), 2 h C peptide (2 h C-P) after meal, fasting C peptide (FC-P), and complications. The research objects were performed with HRCT scan, and the Faster-RCNN algorithm model was built to obtain the imaging features. The relationships between HRCT imaging features and 2 h C-P, FC-P, HbA1c, gender, course of disease, age, and complications were analyzed comprehensively. The results showed that there were no significant differences in HRCT scores between male and female patients, patients of various ages, and patients with different HbA1c contents (P > 0.05). As the course of disease and complications increased, HRCT scores of patients increased obviously (P < 0.05). The HRCT score decreased dramatically with the increase in the contents of 2 h C-P and FC-P after the meal (P < 0.05). In addition, the results of the Spearman rank correlation analysis showed that the course of disease and complications were positively correlated with the HRCT scores, while the 2 h C-P and FC-P levels after meal were negatively correlated with the HRCT scores. The receiver operating curve (ROC) showed that the accuracy, specificity, and sensitivity of HRCT imaging based on Faster-RCNN algorithm were 90.12%, 90.43%, and 83.64%, respectively, in diagnosing lung infection of patients with type 2 diabetes. In summary, the HRCT imaging features based on the Faster-RCNN algorithm can provide effective reference information for the diagnosis and condition assessment of lung infection in patients with type 2 diabetes.

Quantifying Regulated Mitochondrial Fission in Macrophages

Mitochondria have co-evolved with eukaryotic cells for more than a billion years, becoming an important cog in their machinery. They are best known for being tasked with energy generation through the production of adenosine triphosphate, but they also have roles in several other cellular processes, for example, immune and inflammatory responses. Mitochondria have important functions in macrophages, key innate immune cells that detect pathogens and drive inflammation. Mitochondrial activity is influenced by the highly dynamic nature of the mitochondrial network, which alternates between interconnected tubular and fragmented forms. The dynamic balance between this interconnected fused network and fission-mediated mitochondrial fragmentation modulates inflammatory responses such as production of cytokines and mitochondrial reactive oxygen species. Here we describe methods to differentiate mouse bone marrow cells into macrophages and the use of light microscopy, electron microscopy, flow cytometry, and Western blotting to quantify regulated mitochondrial dynamics in these differentiated macrophages.

Application of computer tongue image analysis technology in the diagnosis of NAFLD

Nonalcoholic fatty liver disease (NAFLD), a leading cause of chronic hepatic disease, can progress to liver fibrosis, cirrhosis, and hepatocellular carcinoma. Therefore, it is extremely important to explore early diagnosis and screening methods. In this study, we developed models based on computer tongue image analysis technology to observe the tongue characteristics of 1778 participants (831 cases of NAFLD and 947 cases of non-NAFLD). Combining quantitative tongue image features, basic information, and serological indexes, including the hepatic steatosis index (HSI) and fatty liver index (FLI), we utilized machine learning methods, including Logistic Regression, Support Vector Machine (SVM), Random Forest (RF), Gradient Boosting Decision Tree (GBDT), Adaptive Boosting Algorithm (AdaBoost), Naïve Bayes, and Neural Network for NAFLD diagnosis. The best fusion model for diagnosing NAFLD by Logistic Regression, which contained the tongue image parameters, waist circumference, BMI, GGT, TG, and ALT/AST, achieved an AUC of 0.897 (95% CI, 0.882-0.911), an accuracy of 81.70% with a sensitivity of 77.62% and a specificity of 85.22%; in addition, the positive likelihood ratio and negative likelihood ratio were 5.25 and 0.26, respectively. The application of computer intelligent tongue diagnosis technology can improve the accuracy of NAFLD diagnosis and may provide a convenient technical reference for the establishment of early screening methods for NAFLD, which is worth further research and verification.

One-Shot Learning With Attention-Guided Segmentation in Cryo-Electron Tomography

Cryo-electron Tomography (cryo-ET) generates 3D visualization of cellular organization that allows biologists to analyze cellular structures in a near-native state with nano resolution. Recently, deep learning methods have demonstrated promising performance in classification and segmentation of macromolecule structures captured by cryo-ET, but training individual deep learning models requires large amounts of manually labeled and segmented data from previously observed classes. To perform classification and segmentation in the wild (i.e., with limited training data and with unseen classes), novel deep learning model needs to be developed to classify and segment unseen macromolecules captured by cryo-ET. In this paper, we develop a one-shot learning framework, called cryo-ET one-shot network (COS-Net), for simultaneous classification of macromolecular structure and generation of the voxel-level 3D segmentation, using only one training sample per class. Our experimental results on 22 macromolecule classes demonstrated that our COS-Net could efficiently classify macromolecular structures with small amounts of samples and produce accurate 3D segmentation at the same time.

Few-shot learning for classification of novel macromolecular structures in cryo-electron tomograms

Cryo-electron tomography (cryo-ET) provides 3D visualization of subcellular components in the near-native state and at sub-molecular resolutions in single cells, demonstrating an increasingly important role in structural biology in situ. However, systematic recognition and recovery of macromolecular structures in cryo-ET data remain challenging as a result of low signal-to-noise ratio (SNR), small sizes of macromolecules, and high complexity of the cellular environment. Subtomogram structural classification is an essential step for such task. Although acquisition of large amounts of subtomograms is no longer an obstacle due to advances in automation of data collection, obtaining the same number of structural labels is both computation and labor intensive. On the other hand, existing deep learning based supervised classification approaches are highly demanding on labeled data and have limited ability to learn about new structures rapidly from data containing very few labels of such new structures. In this work, we propose a novel approach for subtomogram classification based on few-shot learning. With our approach, classification of unseen structures in the training data can be conducted given few labeled samples in test data through instance embedding. Experiments were performed on both simulated and real datasets. Our experimental results show that we can make inference on new structures given only five labeled samples for each class with a competitive accuracy (> 0.86 on the simulated dataset with SNR = 0.1), or even one sample with an accuracy of 0.7644. The results on real datasets are also promising with accuracy > 0.9 on both conditions and even up to 1 on one of the real datasets. Our approach achieves significant improvement compared with the baseline method and has strong capabilities of generalizing to other cellular components.

Spark-based parallel calculation of 3D fourier shell correlation for macromolecule structure local resolution estimation

Background

Resolution estimation is the main evaluation criteria for the reconstruction of macromolecular 3D structure in the field of cryoelectron microscopy (cryo-EM). At present, there are many methods to evaluate the 3D resolution for reconstructed macromolecular structures from Single Particle Analysis (SPA) in cryo-EM and subtomogram averaging (SA) in electron cryotomography (cryo-ET). As global methods, they measure the resolution of the structure as a whole, but they are inaccurate in detecting subtle local changes of reconstruction. In order to detect the subtle changes of reconstruction of SPA and SA, a few local resolution methods are proposed. The mainstream local resolution evaluation methods are based on local Fourier shell correlation (FSC), which is computationally intensive. However, the existing resolution evaluation methods are based on multi-threading implementation on a single computer with very poor scalability.

Results

This paper proposes a new fine-grained 3D array partition method by key-value format in Spark. Our method first converts 3D images to key-value data (K-V). Then the K-V data is used for 3D array partitioning and data exchange in parallel. So Spark-based distributed parallel computing framework can solve the above scalability problem. In this distributed computing framework, all 3D local FSC tasks are simultaneously calculated across multiple nodes in a computer cluster. Through the calculation of experimental data, 3D local resolution evaluation algorithm based on Spark fine-grained 3D array partition has a magnitude change in computing speed compared with the mainstream FSC algorithm under the condition that the accuracy remains unchanged, and has better fault tolerance and scalability.

Conclusions

In this paper, we proposed a K-V format based fine-grained 3D array partition method in Spark to parallel calculating 3D FSC for getting a 3D local resolution density map. 3D local resolution density map evaluates the three-dimensional density maps reconstructed from single particle analysis and subtomogram averaging. Our proposed method can significantly increase the speed of the 3D local resolution evaluation, which is important for the efficient detection of subtle variations among reconstructed macromolecular structures.

A unified framework for packing deformable and non-deformable subcellular structures in crowded cryo-electron tomogram simulation

BACKGROUND

Cryo-electron tomography is an important and powerful technique to explore the structure, abundance, and location of ultrastructure in a near-native state. It contains detailed information of all macromolecular complexes in a sample cell. However, due to the compact and crowded status, the missing edge effect, and low signal to noise ratio (SNR), it is extremely challenging to recover such information with existing image processing methods. Cryo-electron tomogram simulation is an effective solution to test and optimize the performance of the above image processing methods. The simulated images could be regarded as the labeled data which covers a wide range of macromolecular complexes and ultrastructure. To approximate the crowded cellular environment, it is very important to pack these heterogeneous structures as tightly as possible. Besides, simulating non-deformable and deformable components under a unified framework also need to be achieved.

RESULT

In this paper, we proposed a unified framework for simulating crowded cryo-electron tomogram images including non-deformable macromolecular complexes and deformable ultrastructures. A macromolecule was approximated using multiple balls with fixed relative positions to reduce the vacuum volume. A ultrastructure, such as membrane and filament, was approximated using multiple balls with flexible relative positions so that this structure could deform under force field. In the experiment, 400 macromolecules of 20 representative types were packed into simulated cytoplasm by our framework, and numerical verification proved that our method has a smaller volume and higher compression ratio than the baseline single-ball model. We also packed filaments, membranes and macromolecules together, to obtain a simulated cryo-electron tomogram image with deformable structures. The simulated results are closer to the real Cryo-ET, making the analysis more difficult. The DOG particle picking method and the image segmentation method are tested on our simulation data, and the experimental results show that these methods still have much room for improvement.

CONCLUSION

The proposed multi-ball model can achieve more crowded packaging results and contains richer elements with different properties to obtain more realistic cryo-electron tomogram simulation. This enables users to simulate cryo-electron tomogram images with non-deformable macromolecular complexes and deformable ultrastructures under a unified framework. To illustrate the advantages of our framework in improving the compression ratio, we calculated the volume of simulated macromolecular under our multi-ball method and traditional single-ball method. We also performed the packing experiment of filaments and membranes to demonstrate the simulation ability of deformable structures. Our method can be used to do a benchmark by generating large labeled cryo-ET dataset and evaluating existing image processing methods. Since the content of the simulated cryo-ET is more complex and crowded compared with previous ones, it will pose a greater challenge to existing image processing methods.

Deep learning in fracture detection: a narrative review

Artificial intelligence (AI) is a general term that implies the use of a computer to model intelligent behavior with minimal human intervention. AI, particularly deep learning, has recently made substantial strides in perception tasks allowing machines to better represent and interpret complex data. Deep learning is a subset of AI represented by the combination of artificial neuron layers. In the last years, deep learning has gained great momentum. In the field of orthopaedics and traumatology, some studies have been done using deep learning to detect fractures in radiographs. Deep learning studies to detect and classify fractures on computed tomography (CT) scans are even more limited. In this narrative review, we provide a brief overview of deep learning technology: we (1) describe the ways in which deep learning until now has been applied to fracture detection on radiographs and CT examinations; (2) discuss what value deep learning offers to this field; and finally (3) comment on future directions of this technology.

Fine-grained alignment of cryo-electron subtomograms based on MPI parallel optimization

BACKGROUND

Cryo-electron tomography (Cryo-ET) is an imaging technique used to generate three-dimensional structures of cellular macromolecule complexes in their native environment. Due to developing cryo-electron microscopy technology, the image quality of three-dimensional reconstruction of cryo-electron tomography has greatly improved. However, cryo-ET images are characterized by low resolution, partial data loss and low signal-to-noise ratio (SNR). In order to tackle these challenges and improve resolution, a large number of subtomograms containing the same structure needs to be aligned and averaged. Existing methods for refining and aligning subtomograms are still highly time-consuming, requiring many computationally intensive processing steps (i.e. the rotations and translations of subtomograms in three-dimensional space).

RESULTS

In this article, we propose a Stochastic Average Gradient (SAG) fine-grained alignment method for optimizing the sum of dissimilarity measure in real space. We introduce a Message Passing Interface (MPI) parallel programming model in order to explore further speedup.

CONCLUSIONS

We compare our stochastic average gradient fine-grained alignment algorithm with two baseline methods, high-precision alignment and fast alignment. Our SAG fine-grained alignment algorithm is much faster than the two baseline methods. Results on simulated data of GroEL from the Protein Data Bank (PDB ID:1KP8) showed that our parallel SAG-based fine-grained alignment method could achieve close-to-optimal rigid transformations with higher precision than both high-precision alignment and fast alignment at a low SNR (SNR=0.003) with tilt angle range ±60∘ or ±40∘. For the experimental subtomograms data structures of GroEL and GroEL/GroES complexes, our parallel SAG-based fine-grained alignment can achieve higher precision and fewer iterations to converge than the two baseline methods.

Adversarial domain adaptation for cross data source macromolecule in situ structural classification in cellular electron cryo-tomograms

MOTIVATION

Since 2017, an increasing amount of attention has been paid to the supervised deep learning-based macromolecule in situ structural classification (i.e. subtomogram classification) in cellular electron cryo-tomography (CECT) due to the substantially higher scalability of deep learning. However, the success of such supervised approach relies heavily on the availability of large amounts of labeled training data. For CECT, creating valid training data from the same data source as prediction data is usually laborious and computationally intensive. It would be beneficial to have training data from a separate data source where the annotation is readily available or can be performed in a high-throughput fashion. However, the cross data source prediction is often biased due to the different image intensity distributions (a.k.a. domain shift).

RESULTS

We adapt a deep learning-based adversarial domain adaptation (3D-ADA) method to timely address the domain shift problem in CECT data analysis. 3D-ADA first uses a source domain feature extractor to extract discriminative features from the training data as the input to a classifier. Then it adversarially trains a target domain feature extractor to reduce the distribution differences of the extracted features between training and prediction data. As a result, the same classifier can be directly applied to the prediction data. We tested 3D-ADA on both experimental and realistically simulated subtomogram datasets under different imaging conditions. 3D-ADA stably improved the cross data source prediction, as well as outperformed two popular domain adaptation methods. Furthermore, we demonstrate that 3D-ADA can improve cross data source recovery of novel macromolecular structures.

AVAILABILITY AND IMPLEMENTATION

https://github.com/xulabs/projects.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.