Segmentation of virus particle candidates in transmission electron microscopy images

Pushing Optical Virus Detection to a Single Particle through a High-Q Quasi-bound State in the Continuum in an All-dielectric Metasurface

Bound states in the continuum (BICs) have emerged as a powerful platform for boosting light-matter interactions because they provide an alternative way of realizing optical resonances with ultrahigh quality factors, accompanied by extreme field confinement. In this work, we realized an optical biosensor by harnessing a quasi-BIC (qBIC) supported by an all-dielectric metasurface with broken symmetry, whose unit cell is composed of a silicon cuboid with two asymmetric air holes. Thanks to the excellent field confinement within the air gap of a metasurface enabled by such a high-Q qBIC, the figure of merit (FOM) of the biosensor is up to 2136.35 RIU-1. Futhermore, we demonstrated that such a high-Q metasurface can push the detection limit to a few virus particles. Our results may find exciting applications in extreme biochemical sensing like COVID-19 with ultralow concentrations.

Segmentation in large-scale cellular electron microscopy with deep learning: A literature survey

Electron microscopy (EM) enables high-resolution imaging of tissues and cells based on 2D and 3D imaging techniques. Due to the laborious and time-consuming nature of manual segmentation of large-scale EM datasets, automated segmentation approaches are crucial. This review focuses on the progress of deep learning-based segmentation techniques in large-scale cellular EM throughout the last six years, during which significant progress has been made in both semantic and instance segmentation. A detailed account is given for the key datasets that contributed to the proliferation of deep learning in 2D and 3D EM segmentation. The review covers supervised, unsupervised, and self-supervised learning methods and examines how these algorithms were adapted to the task of segmenting cellular and sub-cellular structures in EM images. The special challenges posed by such images, like heterogeneity and spatial complexity, and the network architectures that overcame some of them are described. Moreover, an overview of the evaluation measures used to benchmark EM datasets in various segmentation tasks is provided. Finally, an outlook of current trends and future prospects of EM segmentation is given, especially with large-scale models and unlabeled images to learn generic features across EM datasets.

VISN: virus instance segmentation network for TEM images using deep attention transformer

The identification of viruses from negative staining transmission electron microscopy (TEM) images has mainly depended on experienced experts. Recent advances in artificial intelligence have enabled virus recognition using deep learning techniques. However, most of the existing methods only perform virus classification or semantic segmentation, and few studies have addressed the challenge of virus instance segmentation in TEM images. In this paper, we focus on the instance segmentation of severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) and other respiratory viruses and provide experts with more effective information about viruses. We propose an effective virus instance segmentation network based on the You Only Look At CoefficienTs backbone, which integrates the Swin Transformer, dense connections and the coordinate-spatial attention mechanism, to identify SARS-CoV-2, H1N1 influenza virus, respiratory syncytial virus, Herpes simplex virus-1, Human adenovirus type 5 and Vaccinia virus. We also provide a public TEM virus dataset and conduct extensive comparative experiments. Our method achieves a mean average precision score of 83.8 and F1 score of 0.920, outperforming other state-of-the-art instance segmentation algorithms. The proposed automated method provides virologists with an effective approach for recognizing and identifying SARS-CoV-2 and assisting in the diagnosis of viruses. Our dataset and code are accessible at https://github.com/xiaochiHNU/Virus-Instance-Segmentation-Transformer-Network.

Applications of artificial neural networks in microorganism image analysis: a comprehensive review from conventional multilayer perceptron to popular convolutional neural network and potential visual transformer

Microorganisms are widely distributed in the human daily living environment. They play an essential role in environmental pollution control, disease prevention and treatment, and food and drug production. The analysis of microorganisms is essential for making full use of different microorganisms. The conventional analysis methods are laborious and time-consuming. Therefore, the automatic image analysis based on artificial neural networks is introduced to optimize it. However, the automatic microorganism image analysis faces many challenges, such as the requirement of a robust algorithm caused by various application occasions, insignificant features and easy under-segmentation caused by the image characteristic, and various analysis tasks. Therefore, we conduct this review to comprehensively discuss the characteristics of microorganism image analysis based on artificial neural networks. In this review, the background and motivation are introduced first. Then, the development of artificial neural networks and representative networks are presented. After that, the papers related to microorganism image analysis based on classical and deep neural networks are reviewed from the perspectives of different tasks. In the end, the methodology analysis and potential direction are discussed.

Learning Log-Determinant Divergences for Positive Definite Matrices

Representations in the form of Symmetric Positive Definite (SPD) matrices have been popularized in a variety of visual learning applications due to their demonstrated ability to capture rich second-order statistics of visual data. There exist several similarity measures for comparing SPD matrices with documented benefits. However, selecting an appropriate measure for a given problem remains a challenge and in most cases, is the result of a trial-and-error process. In this paper, we propose to learn similarity measures in a data-driven manner. To this end, we capitalize on the αβ-log-det divergence, which is a meta-divergence parametrized by scalars α and β, subsuming a wide family of popular information divergences on SPD matrices for distinct and discrete values of these parameters. Our key idea is to cast these parameters in a continuum and learn them from data. We systematically extend this idea to learn vector-valued parameters, thereby increasing the expressiveness of the underlying non-linear measure. We conjoin the divergence learning problem with several standard tasks in machine learning, including supervised discriminative dictionary learning and unsupervised SPD matrix clustering. We present Riemannian gradient descent schemes for optimizing our formulations efficiently, and show the usefulness of our method on eight standard computer vision tasks.

Gold-viral particle identification by deep learning in wide-field photon scattering parametric images

The ability to identify virus particles is important for research and clinical applications. Because of the optical diffraction limit, conventional optical microscopes are generally not suitable for virus particle detection, and higher resolution instruments such as transmission electron microscopy (TEM) and scanning electron microscopy (SEM) are required. In this paper, we propose a new method for identifying virus particles based on polarization parametric indirect microscopic imaging (PIMI) and deep learning techniques. By introducing an abrupt change of refractivity at the virus particle using antibody-conjugated gold nanoparticles (AuNPs), the strength of the photon scattering signal can be magnified. After acquiring the PIMI images, a deep learning method was applied to identify discriminating features and classify the virus particles, using electron microscopy (EM) images as the ground truth. Experimental results confirm that gold-virus particles can be identified in PIMI images with a high level of confidence.

TEM virus images: Benchmark dataset and deep learning classification

BACKGROUND AND OBJECTIVE

To achieve the full potential of deep learning (DL) models, such as understanding the interplay between model (size), training strategy, and amount of training data, researchers and developers need access to new dedicated image datasets; i.e., annotated collections of images representing real-world problems with all their variations, complexity, limitations, and noise. Here, we present, describe and make freely available an annotated transmission electron microscopy (TEM) image dataset. It constitutes an interesting challenge for many practical applications in virology and epidemiology; e.g., virus detection, segmentation, classification, and novelty detection. We also present benchmarking results for virus detection and recognition using some of the top-performing (large and small) networks as well as a handcrafted very small network. We compare and evaluate transfer learning and training from scratch hypothesizing that with a limited dataset, transfer learning is crucial for good performance of a large network whereas our handcrafted small network performs relatively well when training from scratch. This is one step towards understanding how much training data is needed for a given task.

METHODS

The benchmark dataset contains 1245 images of 22 virus classes. We propose a representative data split into training, validation, and test sets for this dataset. Moreover, we compare different established DL networks and present a baseline DL solution for classifying a subset of the 14 most-represented virus classes in the dataset.

RESULTS

Our best model, DenseNet201 pre-trained on ImageNet and fine-tuned on the training set, achieved a 0.921 F1-score and 93.1% accuracy on the proposed representative test set.

CONCLUSIONS

Public and real biomedical datasets are an important contribution and a necessity to increase the understanding of shortcomings, requirements, and potential improvements for deep learning solutions on biomedical problems or deploying solutions in clinical settings. We compared transfer learning to learning from scratch on this dataset and hypothesize that for limited-sized datasets transfer learning is crucial for achieving good performance for large models. Last but not least, we demonstrate the importance of application knowledge in creating datasets for training DL models and analyzing their results.

Deep Learning for Imaging and Detection of Microorganisms

Despite tremendous recent interest, the application of deep learning in microbiology has still not reached its full potential. To tackle the challenges faced by human-operated microscopy, deep-learning-based methods have been proposed for microscopic image analysis of a wide range of microorganisms, including viruses, bacteria, fungi, and parasites. We believe that deep-learning technology-based systems will be on the front line of monitoring and investigation of microorganisms.

Virus identification in electron microscopy images by residual mixed attention network

BACKGROUND AND OBJECTIVE

Virus identification in electron microscopy (EM) images is considered as one of the front-line method in pathogen diagnosis and re-emerging infectious agents. However, the existing methods either focused on the detection of a single virus or required large amounts of manual labeling work to segment virus. In this work, we focus on the task of virus classification and propose an effective and simple method to identify different viruses.

METHODS

We put forward a residual mixed attention network (RMAN) for virus classification. The proposed network uses channel attention, bottom-up and top-down attention, and incorporates a residual architecture in an end-to-end training manner, which is suitable for dealing with EM virus images and reducing the burden of manual annotation.

RESULTS

We validate the proposed network through extensive experiments on a transmission electron microscopy virus image dataset. The top-1 error rate of our RMAN on 12 virus classes is 4.285%, which surpasses that of state-of-the-art networks and even human experts. In addition, the ablation study and the visualization of class activation mapping (CAM) further demonstrate the effectiveness of our method.

CONCLUSIONS

The proposed automated method contributes to the development of medical virology, which provides virologists with a high-accuracy approach to recognize viruses as well as assist in the diagnosis of viruses.

Optimal Transport in Reproducing Kernel Hilbert Spaces: Theory and Applications

In this paper, we present a mathematical and computational framework for comparing and matching distributions in reproducing kernel Hilbert spaces (RKHS). This framework, called optimal transport in RKHS, is a generalization of the optimal transport problem in input spaces to (potentially) infinite-dimensional feature spaces. We provide a computable formulation of Kantorovich's optimal transport in RKHS. In particular, we explore the case in which data distributions in RKHS are Gaussian, obtaining closed-form expressions of both the estimated Wasserstein distance and optimal transport map via kernel matrices. Based on these expressions, we generalize the Bures metric on covariance matrices to infinite-dimensional settings, providing a new metric between covariance operators. Moreover, we extend the correlation alignment problem to Hilbert spaces, giving a new strategy for matching distributions in RKHS. Empirically, we apply the derived formulas under the Gaussianity assumption to image classification and domain adaptation. In both tasks, our algorithms yield state-of-the-art performances, demonstrating the effectiveness and potential of our framework.

Reducing the U-Net size for practical scenarios: Virus recognition in electron microscopy images

BACKGROUND AND OBJECTIVE

Convolutional neural networks (CNNs) offer human experts-like performance and in the same time they are faster and more consistent in their prediction. However, most of the proposed CNNs require an expensive state-of-the-art hardware which substantially limits their use in practical scenarios and commercial systems, especially for clinical, biomedical and other applications that require on-the-fly analysis. In this paper, we investigate the possibility of making CNNs lighter by parametrizing the architecture and decreasing the number of trainable weights of a popular CNN: U-Net.

METHODS

In order to demonstrate that comparable results can be achieved with substantially less trainable weights than the original U-Net we used a challenging application of a pixel-wise virus classification in Transmission Electron Microscopy images with minimal annotations (i.e. consisting only of the virus particle centers or centerlines). We explored 4 U-Net hyper-parameters: the number of base feature maps, the feature maps multiplier, the number of the encoding-decoding levels and the number of feature maps in the last 2 convolutional layers.

RESULTS

Our experiments lead to two main conclusions: 1) the architecture hyper-parameters are pivotal if less trainable weights are to be used, and 2) if there is no restriction on the trainable weights number using a deeper network generally gives better results. However, training larger networks takes longer, typically requires more data and such networks are also more prone to overfitting. Our best model achieved an accuracy of 82.2% which is similar to the original U-Net while using nearly 4 times less trainable weights (7.8 M in comparison to 31.0 M). We also present a network with  < 2 M trainable weights that achieved an accuracy of 76.4%.

CONCLUSIONS

The proposed U-Net hyper-parameter exploration can be adapted to other CNNs and other applications. It allows a comprehensive CNN architecture designing with the aim of a more efficient trainable weight use. Making the networks faster and lighter is crucial for their implementation in many practical applications. In addition, a lighter network ought to be less prone to over-fitting and hence generalize better.

Detection of herpesvirus capsids in transmission electron microscopy images using transfer learning

The detailed analysis of secondary envelopment of the Human betaherpesvirus 5/human cytomegalovirus (HCMV) from transmission electron microscopy (TEM) images is an important step towards understanding the mechanisms underlying the formation of infectious virions. As a step towards a software-based quantification of different stages of HCMV virion morphogenesis in TEM, we developed a transfer learning approach based on convolutional neural networks (CNNs) that automatically detects HCMV nucleocapsids in TEM images. In contrast to existing image analysis techniques that require time-consuming manual definition of structural features, our method automatically learns discriminative features from raw images without the need for extensive pre-processing. For this a constantly growing TEM image database of HCMV infected cells was available which is unique regarding image quality and size in the terms of virological EM. From the two investigated types of transfer learning approaches, namely feature extraction and fine-tuning, the latter enabled us to successfully detect HCMV nucleocapsids in TEM images. Our detection method has outperformed some of the existing image analysis methods based on discriminative textural indicators and radial density profiles for virus detection in TEM images. In summary, we could show that the method of transfer learning can be used for an automated detection of viral capsids in TEM images with high specificity using standard computers. This method is highly adaptable and in future could be easily extended to automatically detect and classify virions of other viruses and even distinguish different virion maturation stages.

Virus Particle Detection by Convolutional Neural Network in Transmission Electron Microscopy Images

A new computational method for the detection of virus particles in transmission electron microscopy (TEM) images is presented. Our approach is to use a convolutional neural network that transforms a TEM image to a probabilistic map that indicates where virus particles exist in the image. Our proposed approach automatically and simultaneously learns both discriminative features and classifier for virus particle detection by machine learning, in contrast to existing methods that are based on handcrafted features that yield many false positives and require several postprocessing steps. The detection performance of the proposed method was assessed against a dataset of TEM images containing feline calicivirus particles and compared with several existing detection methods, and the state-of-the-art performance of the developed method for detecting virus was demonstrated. Since our method is based on supervised learning that requires both the input images and their corresponding annotations, it is basically used for detection of already-known viruses. However, the method is highly flexible, and the convolutional networks can adapt themselves to any virus particles by learning automatically from an annotated dataset.

Toward a General-Purpose Heterogeneous Ensemble for Pattern Classification

We perform an extensive study of the performance of different classification approaches on twenty-five datasets (fourteen image datasets and eleven UCI data mining datasets). The aim is to find General-Purpose (GP) heterogeneous ensembles (requiring little to no parameter tuning) that perform competitively across multiple datasets. The state-of-the-art classifiers examined in this study include the support vector machine, Gaussian process classifiers, random subspace of adaboost, random subspace of rotation boosting, and deep learning classifiers. We demonstrate that a heterogeneous ensemble based on the simple fusion by sum rule of different classifiers performs consistently well across all twenty-five datasets. The most important result of our investigation is demonstrating that some very recent approaches, including the heterogeneous ensemble we propose in this paper, are capable of outperforming an SVM classifier (implemented with LibSVM), even when both kernel selection and SVM parameters are carefully tuned for each dataset.

Single virus detection by means of atomic force microscopy in combination with advanced image analysis

In the present contribution virions of five different virus species, namely Varicella-zoster virus, Porcine teschovirus, Tobacco mosaic virus, Coliphage M13 and Enterobacteria phage PsP3, are investigated using atomic force microscopy (AFM). From the resulting height images quantitative features like maximal height, area and volume of the viruses could be extracted and compared to reference values. Subsequently, these features were accompanied by image moments, which quantify the morphology of the virions. Both types of features could be utilized for an automatic discrimination of the five virus species. The accuracy of this classification model was 96.8%. Thus, a virus detection on a single-particle level using AFM images is possible. Due to the application of advanced image analysis the morphology could be quantified and used for further analysis. Here, an automatic recognition by means of a classification model could be achieved in a reliable and objective manner.

Texture indicators for segmentation of polyomavirus particles in transmission electron microscopy images

A fully automatic approach to locate polyomavirus particles in transmission electron microscopy images is presented that can localize intact particles, many damaged capsids, and an acceptable percentage of superposed ones. Performance of the approach is quantified in 25 electron micrographs containing nearly 390 particles and compared with the interpretation of the micrographs by two independent electron microscopy experts. All parameterization is based on the particle expected dimensions. This approach uses indicators calculated from the local co-occurrence matrix of gray levels to assess the textured pattern typical of polyomavirus and prune the initial set of candidates. In more complicated backgrounds, about 2-10% of the elements survive. A restricted set of the accepted points is used to evaluate the typical average and variance and to reduce the set of survivors accordingly. These intermediate points are evaluated using (i) a statistical index concerning the radiometric distribution of a circular neighborhood around the centroid of each candidate and (ii) a structural index resuming the expected morphological characteristics of eight radial intensity profiles encompassing the area of the possible particle. This hierarchical approach attains 90% efficiency in the detection of entire virus particles, tolerating a certain lack of differentiation in the borders and a certain amount of shape alterations.

Automatic virus particle selection--the entropy approach

This paper describes a fully automatic approach to locate icosahedral virus particles in transmission electron microscopy images. The initial detection of the particles takes place through automatic segmentation of the entropy-proportion image; this image is computed in particular regions of interest defined by two concentric structuring elements contained in a small overlapping window running over all the image. Morphological features help to select the candidates, as the threshold is kept low enough to avoid false negatives. The candidate points are subject to a credibility test based on features extracted from eight radial intensity profiles in each point from a texture image. A candidate is accepted if these features meet the set of acceptance conditions describing the typical intensity profiles of these kinds of particles. The set of points accepted is subjected to a last validation in a three-parameter space using a discrimination plan that is a function of the input image to separate possible outliers.