1
|
Li Z, Xie M, Nie G, Wang J, Huang L. Pushing Optical Virus Detection to a Single Particle through a High- Q Quasi-bound State in the Continuum in an All-dielectric Metasurface. J Phys Chem Lett 2023; 14:10762-10768. [PMID: 38010952 DOI: 10.1021/acs.jpclett.3c02763] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
Bound states in the continuum (BICs) have emerged as a powerful platform for boosting light-matter interactions because they provide an alternative way of realizing optical resonances with ultrahigh quality factors, accompanied by extreme field confinement. In this work, we realized an optical biosensor by harnessing a quasi-BIC (qBIC) supported by an all-dielectric metasurface with broken symmetry, whose unit cell is composed of a silicon cuboid with two asymmetric air holes. Thanks to the excellent field confinement within the air gap of a metasurface enabled by such a high-Q qBIC, the figure of merit (FOM) of the biosensor is up to 2136.35 RIU-1. Futhermore, we demonstrated that such a high-Q metasurface can push the detection limit to a few virus particles. Our results may find exciting applications in extreme biochemical sensing like COVID-19 with ultralow concentrations.
Collapse
Affiliation(s)
- Zonglin Li
- School of Physics and Electronic Science, Hunan University of Science and Technology, Xiangtan 411201, Hunan, China
- Hunan Provincial Key Laboratory of Intelligent Sensors and New Sensor Materials, Xiangtan 411201, Hunan, China
| | - Mingxin Xie
- School of Microelectronics and Physics, Hunan University of Technology and Business, Changsha 410205, China
| | - Guozheng Nie
- School of Physics and Electronic Science, Hunan University of Science and Technology, Xiangtan 411201, Hunan, China
- Hunan Provincial Key Laboratory of Intelligent Sensors and New Sensor Materials, Xiangtan 411201, Hunan, China
- School of Microelectronics and Physics, Hunan University of Technology and Business, Changsha 410205, China
| | - Junhui Wang
- School of Physics and Electronic Science, Hunan University of Science and Technology, Xiangtan 411201, Hunan, China
| | - Lujun Huang
- School of Physics and Electronic Science, East China Normal University, Shanghai 200241, China
| |
Collapse
|
2
|
Aswath A, Alsahaf A, Giepmans BNG, Azzopardi G. Segmentation in large-scale cellular electron microscopy with deep learning: A literature survey. Med Image Anal 2023; 89:102920. [PMID: 37572414 DOI: 10.1016/j.media.2023.102920] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2022] [Revised: 07/05/2023] [Accepted: 07/31/2023] [Indexed: 08/14/2023]
Abstract
Electron microscopy (EM) enables high-resolution imaging of tissues and cells based on 2D and 3D imaging techniques. Due to the laborious and time-consuming nature of manual segmentation of large-scale EM datasets, automated segmentation approaches are crucial. This review focuses on the progress of deep learning-based segmentation techniques in large-scale cellular EM throughout the last six years, during which significant progress has been made in both semantic and instance segmentation. A detailed account is given for the key datasets that contributed to the proliferation of deep learning in 2D and 3D EM segmentation. The review covers supervised, unsupervised, and self-supervised learning methods and examines how these algorithms were adapted to the task of segmenting cellular and sub-cellular structures in EM images. The special challenges posed by such images, like heterogeneity and spatial complexity, and the network architectures that overcame some of them are described. Moreover, an overview of the evaluation measures used to benchmark EM datasets in various segmentation tasks is provided. Finally, an outlook of current trends and future prospects of EM segmentation is given, especially with large-scale models and unlabeled images to learn generic features across EM datasets.
Collapse
Affiliation(s)
- Anusha Aswath
- Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University Groningen, Groningen, The Netherlands; Department of Biomedical Sciences of Cells and Systems, University Groningen, University Medical Center Groningen, Groningen, The Netherlands.
| | - Ahmad Alsahaf
- Department of Biomedical Sciences of Cells and Systems, University Groningen, University Medical Center Groningen, Groningen, The Netherlands
| | - Ben N G Giepmans
- Department of Biomedical Sciences of Cells and Systems, University Groningen, University Medical Center Groningen, Groningen, The Netherlands
| | - George Azzopardi
- Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University Groningen, Groningen, The Netherlands
| |
Collapse
|
3
|
Xiao C, Wang J, Yang S, Heng M, Su J, Xiao H, Song J, Li W. VISN: virus instance segmentation network for TEM images using deep attention transformer. Brief Bioinform 2023; 24:bbad373. [PMID: 37903415 DOI: 10.1093/bib/bbad373] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Revised: 08/02/2023] [Accepted: 09/29/2023] [Indexed: 11/01/2023] Open
Abstract
The identification of viruses from negative staining transmission electron microscopy (TEM) images has mainly depended on experienced experts. Recent advances in artificial intelligence have enabled virus recognition using deep learning techniques. However, most of the existing methods only perform virus classification or semantic segmentation, and few studies have addressed the challenge of virus instance segmentation in TEM images. In this paper, we focus on the instance segmentation of severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) and other respiratory viruses and provide experts with more effective information about viruses. We propose an effective virus instance segmentation network based on the You Only Look At CoefficienTs backbone, which integrates the Swin Transformer, dense connections and the coordinate-spatial attention mechanism, to identify SARS-CoV-2, H1N1 influenza virus, respiratory syncytial virus, Herpes simplex virus-1, Human adenovirus type 5 and Vaccinia virus. We also provide a public TEM virus dataset and conduct extensive comparative experiments. Our method achieves a mean average precision score of 83.8 and F1 score of 0.920, outperforming other state-of-the-art instance segmentation algorithms. The proposed automated method provides virologists with an effective approach for recognizing and identifying SARS-CoV-2 and assisting in the diagnosis of viruses. Our dataset and code are accessible at https://github.com/xiaochiHNU/Virus-Instance-Segmentation-Transformer-Network.
Collapse
Affiliation(s)
- Chi Xiao
- State key laboratory of digital medical engineering, School of Biomedical Engineering, Hainan University, 570228, Haikou, China
- Key Laboratory of Biomedical Engineering of Hainan Province, One Health Institute, Hainan University, 570228, Haikou, China
| | - Jun Wang
- Laboratory of Microelectronic Devices & Integrated Technology, Institute of Microelectronics, Chinese Academy of Sciences, 100029, Beijing, China
| | - Shenrong Yang
- State key laboratory of digital medical engineering, School of Biomedical Engineering, Hainan University, 570228, Haikou, China
- Key Laboratory of Biomedical Engineering of Hainan Province, One Health Institute, Hainan University, 570228, Haikou, China
| | - Minxin Heng
- State key laboratory of digital medical engineering, School of Biomedical Engineering, Hainan University, 570228, Haikou, China
- Key Laboratory of Biomedical Engineering of Hainan Province, One Health Institute, Hainan University, 570228, Haikou, China
| | - Junyi Su
- State key laboratory of digital medical engineering, School of Biomedical Engineering, Hainan University, 570228, Haikou, China
- Key Laboratory of Biomedical Engineering of Hainan Province, One Health Institute, Hainan University, 570228, Haikou, China
| | - Hao Xiao
- Key Laboratory for Matter Microstructure and Function of Hunan Province, Key Laboratory of Low-dimensional Quantum Structures and Quantum Control, School of Physics and Electronics, Hunan Normal University, 410081, Changsha, China
- National Key Laboratory of Intelligent Tracking and Forecasting for Infectious Diseases, National Institute for Viral Disease Control and Prevention, Chinese Center for Disease Control and Prevention, 102206, Beijing, China
| | - Jingdong Song
- National Key Laboratory of Intelligent Tracking and Forecasting for Infectious Diseases, National Institute for Viral Disease Control and Prevention, Chinese Center for Disease Control and Prevention, 102206, Beijing, China
- NHC Key Laboratory of Biosafety, National Institute for Viral Disease Control and Prevention, Chinese Center for Disease Control and Prevention, 102206, Beijing, China
| | - Weifu Li
- College of Informatics, Huazhong Agricultural University, 430070, Wuhan, China
| |
Collapse
|
4
|
Applications of artificial neural networks in microorganism image analysis: a comprehensive review from conventional multilayer perceptron to popular convolutional neural network and potential visual transformer. Artif Intell Rev 2023; 56:1013-1070. [PMID: 35528112 PMCID: PMC9066147 DOI: 10.1007/s10462-022-10192-7] [Citation(s) in RCA: 32] [Impact Index Per Article: 32.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
Abstract
Microorganisms are widely distributed in the human daily living environment. They play an essential role in environmental pollution control, disease prevention and treatment, and food and drug production. The analysis of microorganisms is essential for making full use of different microorganisms. The conventional analysis methods are laborious and time-consuming. Therefore, the automatic image analysis based on artificial neural networks is introduced to optimize it. However, the automatic microorganism image analysis faces many challenges, such as the requirement of a robust algorithm caused by various application occasions, insignificant features and easy under-segmentation caused by the image characteristic, and various analysis tasks. Therefore, we conduct this review to comprehensively discuss the characteristics of microorganism image analysis based on artificial neural networks. In this review, the background and motivation are introduced first. Then, the development of artificial neural networks and representative networks are presented. After that, the papers related to microorganism image analysis based on classical and deep neural networks are reviewed from the perspectives of different tasks. In the end, the methodology analysis and potential direction are discussed.
Collapse
|
5
|
Cherian A, Stanitsas P, Wang J, Harandi M, Morellas V, Papanikolopoulos N. Learning Log-Determinant Divergences for Positive Definite Matrices. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:5088-5102. [PMID: 33856984 DOI: 10.1109/tpami.2021.3073588] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Representations in the form of Symmetric Positive Definite (SPD) matrices have been popularized in a variety of visual learning applications due to their demonstrated ability to capture rich second-order statistics of visual data. There exist several similarity measures for comparing SPD matrices with documented benefits. However, selecting an appropriate measure for a given problem remains a challenge and in most cases, is the result of a trial-and-error process. In this paper, we propose to learn similarity measures in a data-driven manner. To this end, we capitalize on the αβ-log-det divergence, which is a meta-divergence parametrized by scalars α and β, subsuming a wide family of popular information divergences on SPD matrices for distinct and discrete values of these parameters. Our key idea is to cast these parameters in a continuum and learn them from data. We systematically extend this idea to learn vector-valued parameters, thereby increasing the expressiveness of the underlying non-linear measure. We conjoin the divergence learning problem with several standard tasks in machine learning, including supervised discriminative dictionary learning and unsupervised SPD matrix clustering. We present Riemannian gradient descent schemes for optimizing our formulations efficiently, and show the usefulness of our method on eight standard computer vision tasks.
Collapse
|
6
|
Zhao H, Ni B, Jin X, Zhang H, Hou JJ, Hou L, Marsh JH, Dong L, Li S, Gao XW, Shi D, Liu X, Xiong J. Gold-viral particle identification by deep learning in wide-field photon scattering parametric images. APPLIED OPTICS 2022; 61:546-553. [PMID: 35200896 DOI: 10.1364/ao.445953] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/13/2021] [Accepted: 12/06/2021] [Indexed: 06/14/2023]
Abstract
The ability to identify virus particles is important for research and clinical applications. Because of the optical diffraction limit, conventional optical microscopes are generally not suitable for virus particle detection, and higher resolution instruments such as transmission electron microscopy (TEM) and scanning electron microscopy (SEM) are required. In this paper, we propose a new method for identifying virus particles based on polarization parametric indirect microscopic imaging (PIMI) and deep learning techniques. By introducing an abrupt change of refractivity at the virus particle using antibody-conjugated gold nanoparticles (AuNPs), the strength of the photon scattering signal can be magnified. After acquiring the PIMI images, a deep learning method was applied to identify discriminating features and classify the virus particles, using electron microscopy (EM) images as the ground truth. Experimental results confirm that gold-virus particles can be identified in PIMI images with a high level of confidence.
Collapse
|
7
|
Matuszewski DJ, Sintorn IM. TEM virus images: Benchmark dataset and deep learning classification. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 209:106318. [PMID: 34375851 DOI: 10.1016/j.cmpb.2021.106318] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/11/2021] [Accepted: 07/24/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND AND OBJECTIVE To achieve the full potential of deep learning (DL) models, such as understanding the interplay between model (size), training strategy, and amount of training data, researchers and developers need access to new dedicated image datasets; i.e., annotated collections of images representing real-world problems with all their variations, complexity, limitations, and noise. Here, we present, describe and make freely available an annotated transmission electron microscopy (TEM) image dataset. It constitutes an interesting challenge for many practical applications in virology and epidemiology; e.g., virus detection, segmentation, classification, and novelty detection. We also present benchmarking results for virus detection and recognition using some of the top-performing (large and small) networks as well as a handcrafted very small network. We compare and evaluate transfer learning and training from scratch hypothesizing that with a limited dataset, transfer learning is crucial for good performance of a large network whereas our handcrafted small network performs relatively well when training from scratch. This is one step towards understanding how much training data is needed for a given task. METHODS The benchmark dataset contains 1245 images of 22 virus classes. We propose a representative data split into training, validation, and test sets for this dataset. Moreover, we compare different established DL networks and present a baseline DL solution for classifying a subset of the 14 most-represented virus classes in the dataset. RESULTS Our best model, DenseNet201 pre-trained on ImageNet and fine-tuned on the training set, achieved a 0.921 F1-score and 93.1% accuracy on the proposed representative test set. CONCLUSIONS Public and real biomedical datasets are an important contribution and a necessity to increase the understanding of shortcomings, requirements, and potential improvements for deep learning solutions on biomedical problems or deploying solutions in clinical settings. We compared transfer learning to learning from scratch on this dataset and hypothesize that for limited-sized datasets transfer learning is crucial for achieving good performance for large models. Last but not least, we demonstrate the importance of application knowledge in creating datasets for training DL models and analyzing their results.
Collapse
Affiliation(s)
| | - Ida-Maria Sintorn
- Department of Information Technology, Uppsala University, Uppsala, Sweden; Vironova AB, Gävlegatan 22, Stockholm, Sweden
| |
Collapse
|
8
|
Deep Learning for Imaging and Detection of Microorganisms. Trends Microbiol 2021; 29:569-572. [PMID: 33531192 DOI: 10.1016/j.tim.2021.01.006] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2020] [Revised: 01/12/2021] [Accepted: 01/13/2021] [Indexed: 01/03/2023]
Abstract
Despite tremendous recent interest, the application of deep learning in microbiology has still not reached its full potential. To tackle the challenges faced by human-operated microscopy, deep-learning-based methods have been proposed for microscopic image analysis of a wide range of microorganisms, including viruses, bacteria, fungi, and parasites. We believe that deep-learning technology-based systems will be on the front line of monitoring and investigation of microorganisms.
Collapse
|
9
|
Xiao C, Chen X, Xie Q, Li G, Xiao H, Song J, Han H. Virus identification in electron microscopy images by residual mixed attention network. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 198:105766. [PMID: 33059061 DOI: 10.1016/j.cmpb.2020.105766] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/14/2020] [Accepted: 09/15/2020] [Indexed: 06/11/2023]
Abstract
BACKGROUND AND OBJECTIVE Virus identification in electron microscopy (EM) images is considered as one of the front-line method in pathogen diagnosis and re-emerging infectious agents. However, the existing methods either focused on the detection of a single virus or required large amounts of manual labeling work to segment virus. In this work, we focus on the task of virus classification and propose an effective and simple method to identify different viruses. METHODS We put forward a residual mixed attention network (RMAN) for virus classification. The proposed network uses channel attention, bottom-up and top-down attention, and incorporates a residual architecture in an end-to-end training manner, which is suitable for dealing with EM virus images and reducing the burden of manual annotation. RESULTS We validate the proposed network through extensive experiments on a transmission electron microscopy virus image dataset. The top-1 error rate of our RMAN on 12 virus classes is 4.285%, which surpasses that of state-of-the-art networks and even human experts. In addition, the ablation study and the visualization of class activation mapping (CAM) further demonstrate the effectiveness of our method. CONCLUSIONS The proposed automated method contributes to the development of medical virology, which provides virologists with a high-accuracy approach to recognize viruses as well as assist in the diagnosis of viruses.
Collapse
Affiliation(s)
- Chi Xiao
- School of Biomedical Engineering, Hainan University, Haikou, China; National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Xi Chen
- National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Qiwei Xie
- Data Mining Lab, Beijing University of Technology, Beijing, China; National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Guoqing Li
- National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Hao Xiao
- State Key Laboratory of Infectious Disease Prevention and Control, National Institute for Viral Disease Control and Prevention, Chinese Center for Disease Control and Prevention, Beijing, China; College of Physics and Information Science, Key Laboratory of Low-dimensional Quantum Structures, And Quantum Control of the Ministry of Education, Synergetic Innovation Center for Quantum Effects and Applications, Hunan Normal University, Changsha, China
| | - Jingdong Song
- State Key Laboratory of Infectious Disease Prevention and Control, National Institute for Viral Disease Control and Prevention, Chinese Center for Disease Control and Prevention, Beijing, China; Collaborative Innovation Center for Diagnosis and Treatment of Infectious Diseases, Hangzhou, China.
| | - Hua Han
- National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China; Chinese Academy of Sciences Center for Excellence in Brain Science and Intelligence Technology, Shanghai, China; School of Future Technology, University of Chinese Academy of Sciences, Beijing, China.
| |
Collapse
|
10
|
Zhang Z, Wang M, Nehorai A. Optimal Transport in Reproducing Kernel Hilbert Spaces: Theory and Applications. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2020; 42:1741-1754. [PMID: 30843802 DOI: 10.1109/tpami.2019.2903050] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/20/2023]
Abstract
In this paper, we present a mathematical and computational framework for comparing and matching distributions in reproducing kernel Hilbert spaces (RKHS). This framework, called optimal transport in RKHS, is a generalization of the optimal transport problem in input spaces to (potentially) infinite-dimensional feature spaces. We provide a computable formulation of Kantorovich's optimal transport in RKHS. In particular, we explore the case in which data distributions in RKHS are Gaussian, obtaining closed-form expressions of both the estimated Wasserstein distance and optimal transport map via kernel matrices. Based on these expressions, we generalize the Bures metric on covariance matrices to infinite-dimensional settings, providing a new metric between covariance operators. Moreover, we extend the correlation alignment problem to Hilbert spaces, giving a new strategy for matching distributions in RKHS. Empirically, we apply the derived formulas under the Gaussianity assumption to image classification and domain adaptation. In both tasks, our algorithms yield state-of-the-art performances, demonstrating the effectiveness and potential of our framework.
Collapse
|
11
|
Matuszewski DJ, Sintorn IM. Reducing the U-Net size for practical scenarios: Virus recognition in electron microscopy images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2019; 178:31-39. [PMID: 31416558 DOI: 10.1016/j.cmpb.2019.05.026] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/11/2019] [Revised: 05/13/2019] [Accepted: 05/28/2019] [Indexed: 05/10/2023]
Abstract
BACKGROUND AND OBJECTIVE Convolutional neural networks (CNNs) offer human experts-like performance and in the same time they are faster and more consistent in their prediction. However, most of the proposed CNNs require an expensive state-of-the-art hardware which substantially limits their use in practical scenarios and commercial systems, especially for clinical, biomedical and other applications that require on-the-fly analysis. In this paper, we investigate the possibility of making CNNs lighter by parametrizing the architecture and decreasing the number of trainable weights of a popular CNN: U-Net. METHODS In order to demonstrate that comparable results can be achieved with substantially less trainable weights than the original U-Net we used a challenging application of a pixel-wise virus classification in Transmission Electron Microscopy images with minimal annotations (i.e. consisting only of the virus particle centers or centerlines). We explored 4 U-Net hyper-parameters: the number of base feature maps, the feature maps multiplier, the number of the encoding-decoding levels and the number of feature maps in the last 2 convolutional layers. RESULTS Our experiments lead to two main conclusions: 1) the architecture hyper-parameters are pivotal if less trainable weights are to be used, and 2) if there is no restriction on the trainable weights number using a deeper network generally gives better results. However, training larger networks takes longer, typically requires more data and such networks are also more prone to overfitting. Our best model achieved an accuracy of 82.2% which is similar to the original U-Net while using nearly 4 times less trainable weights (7.8 M in comparison to 31.0 M). We also present a network with < 2 M trainable weights that achieved an accuracy of 76.4%. CONCLUSIONS The proposed U-Net hyper-parameter exploration can be adapted to other CNNs and other applications. It allows a comprehensive CNN architecture designing with the aim of a more efficient trainable weight use. Making the networks faster and lighter is crucial for their implementation in many practical applications. In addition, a lighter network ought to be less prone to over-fitting and hence generalize better.
Collapse
Affiliation(s)
| | - Ida-Maria Sintorn
- Department of Information Technology, Uppsala University, Uppsala, Sweden; Vironova AB, Gävlegatan 22, Stockholm, Sweden.
| |
Collapse
|
12
|
Devan KS, Walther P, von Einem J, Ropinski T, Kestler HA, Read C. Detection of herpesvirus capsids in transmission electron microscopy images using transfer learning. Histochem Cell Biol 2018; 151:101-114. [PMID: 30488339 DOI: 10.1007/s00418-018-1759-5] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/24/2018] [Indexed: 10/27/2022]
Abstract
The detailed analysis of secondary envelopment of the Human betaherpesvirus 5/human cytomegalovirus (HCMV) from transmission electron microscopy (TEM) images is an important step towards understanding the mechanisms underlying the formation of infectious virions. As a step towards a software-based quantification of different stages of HCMV virion morphogenesis in TEM, we developed a transfer learning approach based on convolutional neural networks (CNNs) that automatically detects HCMV nucleocapsids in TEM images. In contrast to existing image analysis techniques that require time-consuming manual definition of structural features, our method automatically learns discriminative features from raw images without the need for extensive pre-processing. For this a constantly growing TEM image database of HCMV infected cells was available which is unique regarding image quality and size in the terms of virological EM. From the two investigated types of transfer learning approaches, namely feature extraction and fine-tuning, the latter enabled us to successfully detect HCMV nucleocapsids in TEM images. Our detection method has outperformed some of the existing image analysis methods based on discriminative textural indicators and radial density profiles for virus detection in TEM images. In summary, we could show that the method of transfer learning can be used for an automated detection of viral capsids in TEM images with high specificity using standard computers. This method is highly adaptable and in future could be easily extended to automatically detect and classify virions of other viruses and even distinguish different virion maturation stages.
Collapse
Affiliation(s)
- K Shaga Devan
- Central Facility for Electron Microscopy, Ulm University, Ulm, Germany
| | - P Walther
- Central Facility for Electron Microscopy, Ulm University, Ulm, Germany.
| | - J von Einem
- Institute of Virology, Ulm University Medical Center, Ulm, Germany
| | - T Ropinski
- Institute of Media Informatics, Ulm University, Ulm, Germany
| | - H A Kestler
- Institute of Medical Systems Biology, Ulm University, Ulm, Germany
| | - C Read
- Central Facility for Electron Microscopy, Ulm University, Ulm, Germany.,Institute of Virology, Ulm University Medical Center, Ulm, Germany
| |
Collapse
|
13
|
Ito E, Sato T, Sano D, Utagawa E, Kato T. Virus Particle Detection by Convolutional Neural Network in Transmission Electron Microscopy Images. FOOD AND ENVIRONMENTAL VIROLOGY 2018; 10:201-208. [PMID: 29352405 DOI: 10.1007/s12560-018-9335-7] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/28/2017] [Accepted: 01/12/2018] [Indexed: 05/21/2023]
Abstract
A new computational method for the detection of virus particles in transmission electron microscopy (TEM) images is presented. Our approach is to use a convolutional neural network that transforms a TEM image to a probabilistic map that indicates where virus particles exist in the image. Our proposed approach automatically and simultaneously learns both discriminative features and classifier for virus particle detection by machine learning, in contrast to existing methods that are based on handcrafted features that yield many false positives and require several postprocessing steps. The detection performance of the proposed method was assessed against a dataset of TEM images containing feline calicivirus particles and compared with several existing detection methods, and the state-of-the-art performance of the developed method for detecting virus was demonstrated. Since our method is based on supervised learning that requires both the input images and their corresponding annotations, it is basically used for detection of already-known viruses. However, the method is highly flexible, and the convolutional networks can adapt themselves to any virus particles by learning automatically from an annotated dataset.
Collapse
Affiliation(s)
- Eisuke Ito
- Division of Electronics and Informatics, Faculty of Science and Technology, Gunma University, Tenjin-cho 1-5-1, Kiryu, Gunma, 376-8515, Japan
| | - Takaaki Sato
- Division of Electronics and Informatics, Faculty of Science and Technology, Gunma University, Tenjin-cho 1-5-1, Kiryu, Gunma, 376-8515, Japan
| | - Daisuke Sano
- Department of Civil and Environmental Engineering, Graduate School of Engineering, Tohoku University, Aoba 6-6-06, Aramaki, Aoba-ku, Sendai, Miyagi, 980-8579, Japan
| | - Etsuko Utagawa
- Laboratory of Viral Infection I, Graduate School of Infection Control Sciences, Kitasato Institute for Life Sciences, Kitasato University, 5-9-1 Shirokane, Minato-ku, Tokyo, 108-8641, Japan
| | - Tsuyoshi Kato
- Division of Electronics and Informatics, Faculty of Science and Technology, Gunma University, Tenjin-cho 1-5-1, Kiryu, Gunma, 376-8515, Japan.
- Center for Research on Adoption of NextGen Transportation Systems (CRANTS), Gunma University, Tenjin-cho 1-5-1, Kiryu, Gunma, 376-8515, Japan.
- Integrated Institute for Regulatory Science, Waseda University, Tsurumaki-cho 513, Shinjuku-ku, Tokyo, 162-0041, Japan.
| |
Collapse
|
14
|
Explicit discriminative representation for improved classification of manifold features. Pattern Recognit Lett 2016. [DOI: 10.1016/j.patrec.2016.06.006] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
|
15
|
Wen Z, Li Z, Peng Y, Ying S. Virus image classification using multi-scale completed local binary pattern features extracted from filtered images by multi-scale principal component analysis. Pattern Recognit Lett 2016. [DOI: 10.1016/j.patrec.2016.04.022] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
16
|
Toward a General-Purpose Heterogeneous Ensemble for Pattern Classification. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2015; 2015:909123. [PMID: 26413089 PMCID: PMC4564633 DOI: 10.1155/2015/909123] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 04/09/2015] [Accepted: 07/27/2015] [Indexed: 12/03/2022]
Abstract
We perform an extensive study of the performance of different classification approaches on twenty-five datasets (fourteen image datasets and eleven UCI data mining datasets). The aim is to find General-Purpose (GP) heterogeneous ensembles (requiring little to no parameter tuning) that perform competitively across multiple datasets. The state-of-the-art classifiers examined in this study include the support vector machine, Gaussian process classifiers, random subspace of adaboost, random subspace of rotation boosting, and deep learning classifiers. We demonstrate that a heterogeneous ensemble based on the simple fusion by sum rule of different classifiers performs consistently well across all twenty-five datasets. The most important result of our investigation is demonstrating that some very recent approaches, including the heterogeneous ensemble we propose in this paper, are capable of outperforming an SVM classifier (implemented with LibSVM), even when both kernel selection and SVM parameters are carefully tuned for each dataset.
Collapse
|
17
|
Bocklitz T, Kämmer E, Stöckel S, Cialla-May D, Weber K, Zell R, Deckert V, Popp J. Single virus detection by means of atomic force microscopy in combination with advanced image analysis. J Struct Biol 2014; 188:30-8. [PMID: 25196422 DOI: 10.1016/j.jsb.2014.08.008] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2014] [Accepted: 08/25/2014] [Indexed: 11/25/2022]
Abstract
In the present contribution virions of five different virus species, namely Varicella-zoster virus, Porcine teschovirus, Tobacco mosaic virus, Coliphage M13 and Enterobacteria phage PsP3, are investigated using atomic force microscopy (AFM). From the resulting height images quantitative features like maximal height, area and volume of the viruses could be extracted and compared to reference values. Subsequently, these features were accompanied by image moments, which quantify the morphology of the virions. Both types of features could be utilized for an automatic discrimination of the five virus species. The accuracy of this classification model was 96.8%. Thus, a virus detection on a single-particle level using AFM images is possible. Due to the application of advanced image analysis the morphology could be quantified and used for further analysis. Here, an automatic recognition by means of a classification model could be achieved in a reliable and objective manner.
Collapse
Affiliation(s)
- Thomas Bocklitz
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany; InfectoGnostics Forschungscampus Jena e.V., Zentrum für Angewandte Forschung, Philosophenweg 7, 07743 Jena, Germany; Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany.
| | - Evelyn Kämmer
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany; InfectoGnostics Forschungscampus Jena e.V., Zentrum für Angewandte Forschung, Philosophenweg 7, 07743 Jena, Germany; Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany
| | - Stephan Stöckel
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany; InfectoGnostics Forschungscampus Jena e.V., Zentrum für Angewandte Forschung, Philosophenweg 7, 07743 Jena, Germany; Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany
| | - Dana Cialla-May
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany; InfectoGnostics Forschungscampus Jena e.V., Zentrum für Angewandte Forschung, Philosophenweg 7, 07743 Jena, Germany; Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany
| | - Karina Weber
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany; InfectoGnostics Forschungscampus Jena e.V., Zentrum für Angewandte Forschung, Philosophenweg 7, 07743 Jena, Germany; Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany
| | - Roland Zell
- Department of Virology and Antiviral Therapy, Jena University Hospital, Friedrich Schiller University Jena, Hans-Knöll-Strasse 2, 07745 Jena, Germany
| | - Volker Deckert
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany; InfectoGnostics Forschungscampus Jena e.V., Zentrum für Angewandte Forschung, Philosophenweg 7, 07743 Jena, Germany; Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany
| | - Jürgen Popp
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany; InfectoGnostics Forschungscampus Jena e.V., Zentrum für Angewandte Forschung, Philosophenweg 7, 07743 Jena, Germany; Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany
| |
Collapse
|
18
|
Proença MC, Nunes JFM, de Matos APA. Texture indicators for segmentation of polyomavirus particles in transmission electron microscopy images. MICROSCOPY AND MICROANALYSIS : THE OFFICIAL JOURNAL OF MICROSCOPY SOCIETY OF AMERICA, MICROBEAM ANALYSIS SOCIETY, MICROSCOPICAL SOCIETY OF CANADA 2013; 19:1170-1182. [PMID: 23773502 DOI: 10.1017/s1431927613001736] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/02/2023]
Abstract
A fully automatic approach to locate polyomavirus particles in transmission electron microscopy images is presented that can localize intact particles, many damaged capsids, and an acceptable percentage of superposed ones. Performance of the approach is quantified in 25 electron micrographs containing nearly 390 particles and compared with the interpretation of the micrographs by two independent electron microscopy experts. All parameterization is based on the particle expected dimensions. This approach uses indicators calculated from the local co-occurrence matrix of gray levels to assess the textured pattern typical of polyomavirus and prune the initial set of candidates. In more complicated backgrounds, about 2-10% of the elements survive. A restricted set of the accepted points is used to evaluate the typical average and variance and to reduce the set of survivors accordingly. These intermediate points are evaluated using (i) a statistical index concerning the radiometric distribution of a circular neighborhood around the centroid of each candidate and (ii) a structural index resuming the expected morphological characteristics of eight radial intensity profiles encompassing the area of the possible particle. This hierarchical approach attains 90% efficiency in the detection of entire virus particles, tolerating a certain lack of differentiation in the borders and a certain amount of shape alterations.
Collapse
Affiliation(s)
- Maria C Proença
- Laboratory of Optics, Lasers and Systems, Physics Department, Faculty of Sciences of the University of Lisbon, Edifício C8, Campo Grande, 1749-016 Lisboa, Portugal
| | | | | |
Collapse
|
19
|
Proenca MDCMS, Nunes JFM, de Matos APA. Automatic virus particle selection--the entropy approach. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2013; 22:1996-2003. [PMID: 23380855 DOI: 10.1109/tip.2013.2244216] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
This paper describes a fully automatic approach to locate icosahedral virus particles in transmission electron microscopy images. The initial detection of the particles takes place through automatic segmentation of the entropy-proportion image; this image is computed in particular regions of interest defined by two concentric structuring elements contained in a small overlapping window running over all the image. Morphological features help to select the candidates, as the threshold is kept low enough to avoid false negatives. The candidate points are subject to a credibility test based on features extracted from eight radial intensity profiles in each point from a texture image. A candidate is accepted if these features meet the set of acceptance conditions describing the typical intensity profiles of these kinds of particles. The set of points accepted is subjected to a last validation in a three-parameter space using a discrimination plan that is a function of the input image to separate possible outliers.
Collapse
|
20
|
Kylberg G, Uppström M, Sintorn IM. Virus Texture Analysis Using Local Binary Patterns and Radial Density Profiles. PROGRESS IN PATTERN RECOGNITION, IMAGE ANALYSIS, COMPUTER VISION, AND APPLICATIONS 2011. [DOI: 10.1007/978-3-642-25085-9_68] [Citation(s) in RCA: 20] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
|