1
|
Li Y, Zeng GH, Liang YJ, Yang HR, Zhu XL, Zhai YJ, Duan LX, Xu YY. Improving quantitative prediction of protein subcellular locations in fluorescence images through deep generative models. Comput Biol Med 2024; 179:108913. [PMID: 39047508 DOI: 10.1016/j.compbiomed.2024.108913] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2024] [Revised: 06/21/2024] [Accepted: 07/15/2024] [Indexed: 07/27/2024]
Abstract
Machine learning has been employed in recognizing protein localization at the subcellular level, which highly facilitates the protein function studies, especially for those multi-label proteins that localize in more than one organelle. However, existing works mostly study the qualitative classification of protein subcellular locations, ignoring fraction of one multi-label protein in different locations. Actually, about 50 % proteins are multi-label proteins, and the ignorance of quantitative information highly restricts the understanding of their spatial distribution and functional mechanism. One reason of the lack of quantitative study is the insufficiency of quantitative annotations. To address the data shortage problem, here we proposed a generative model, PLocGAN, which could generate cell images with conditional quantitative annotation of the fluorescence distribution. The model was a conditional generative adversarial network, in which the condition learning utilized partial label learning to overcome the lack of training labels and allowed training with only qualitative labels. Meanwhile, it used contrastive learning to enhance diversity of the generated images. We assessed the PLocGAN on four pixel-fused synthetic datasets and one real dataset, and demonstrated that the model could generate images with good fidelity and diversity, outperforming existing state-of-the-art generative methods. To verify the utility of PLocGAN in the quantitative prediction of protein subcellular locations, we replaced the training images with generated quantitative images and built prediction models, and found that they had a boosting effect on the quantitative estimation. This work demonstrates the effectiveness of deep generative models in bioimage analysis, and provides a new solution for quantitative subcellular proteomics.
Collapse
Affiliation(s)
- Yu Li
- School of Biomedical Engineering and Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Guo-Hua Zeng
- School of Biomedical Engineering and Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Yong-Jia Liang
- School of Biomedical Engineering and Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Hong-Rui Yang
- School of Biomedical Engineering and Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Xi-Liang Zhu
- School of Biomedical Engineering and Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Yu-Jia Zhai
- Cancer Center, Affiliated Hospital of Guangdong Medical University, Zhanjiang, 524000, China
| | - Li-Xia Duan
- Guangzhou Red Cross Hospital, Medical College, Jinan University, Guangzhou, 510220, China
| | - Ying-Ying Xu
- School of Biomedical Engineering and Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China.
| |
Collapse
|
2
|
Ma J, Chen H. Efficient Supervised Pretraining of Swin-Transformer for Virtual Staining of Microscopy Images. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:1388-1399. [PMID: 38010933 DOI: 10.1109/tmi.2023.3337253] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
Fluorescence staining is an important technique in life science for labeling cellular constituents. However, it also suffers from being time-consuming, having difficulty in simultaneous labeling, etc. Thus, virtual staining, which does not rely on chemical labeling, has been introduced. Recently, deep learning models such as transformers have been applied to virtual staining tasks. However, their performance relies on large-scale pretraining, hindering their development in the field. To reduce the reliance on large amounts of computation and data, we construct a Swin-transformer model and propose an efficient supervised pretraining method based on the masked autoencoder (MAE). Specifically, we adopt downsampling and grid sampling to mask 75% of pixels and reduce the number of tokens. The pretraining time of our method is only 1/16 compared with the original MAE. We also design a supervised proxy task to predict stained images with multiple styles instead of masked pixels. Additionally, most virtual staining approaches are based on private datasets and evaluated by different metrics, making a fair comparison difficult. Therefore, we develop a standard benchmark based on three public datasets and build a baseline for the convenience of future researchers. We conduct extensive experiments on three benchmark datasets, and the experimental results show the proposed method achieves the best performance both quantitatively and qualitatively. In addition, ablation studies are conducted, and experimental results illustrate the effectiveness of the proposed pretraining method. The benchmark and code are available at https://github.com/birkhoffkiki/CAS-Transformer.
Collapse
|
3
|
Jiang Y, Sha H, Liu S, Qin P, Zhang Y. AutoUnmix: an autoencoder-based spectral unmixing method for multi-color fluorescence microscopy imaging. BIOMEDICAL OPTICS EXPRESS 2023; 14:4814-4827. [PMID: 37791286 PMCID: PMC10545201 DOI: 10.1364/boe.498421] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 08/12/2023] [Accepted: 08/14/2023] [Indexed: 10/05/2023]
Abstract
Multiplexed fluorescence microscopy imaging is widely used in biomedical applications. However, simultaneous imaging of multiple fluorophores can result in spectral leaks and overlapping, which greatly degrades image quality and subsequent analysis. Existing popular spectral unmixing methods are mainly based on computational intensive linear models, and the performance is heavily dependent on the reference spectra, which may greatly preclude its further applications. In this paper, we propose a deep learning-based blindly spectral unmixing method, termed AutoUnmix, to imitate the physical spectral mixing process. A transfer learning framework is further devised to allow our AutoUnmix to adapt to a variety of imaging systems without retraining the network. Our proposed method has demonstrated real-time unmixing capabilities, surpassing existing methods by up to 100-fold in terms of unmixing speed. We further validate the reconstruction performance on both synthetic datasets and biological samples. The unmixing results of AutoUnmix achieve the highest SSIM of 0.99 in both three- and four-color imaging, with nearly up to 20% higher than other popular unmixing methods. For experiments where spectral profiles and morphology are akin to simulated data, our method realizes the quantitative performance demonstrated above. Due to the desirable property of data independency and superior blind unmixing performance, we believe AutoUnmix is a powerful tool for studying the interaction process of different organelles labeled by multiple fluorophores.
Collapse
Affiliation(s)
- Yuan Jiang
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, Guangdong 518055, China
| | - Hao Sha
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, Guangdong 518055, China
| | - Shuai Liu
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, Guangdong Province 518055, China
| | - Peiwu Qin
- Center of Precision Medicine and Healthcare, Tsinghua-Berkeley Shenzhen Institute, Guangdong Province, 518055, China
- Institute of Biopharmaceutical and Health Engineering, Tsinghua Shenzhen International Graduate School, Shenzhen, Guangdong Province, 518055, China
| | - Yongbing Zhang
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, Guangdong 518055, China
| |
Collapse
|
4
|
Yuan H, Yu H, Gui S, Ji S. Explainability in Graph Neural Networks: A Taxonomic Survey. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:5782-5799. [PMID: 36063508 DOI: 10.1109/tpami.2022.3204236] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/06/2023]
Abstract
Deep learning methods are achieving ever-increasing performance on many artificial intelligence tasks. A major limitation of deep models is that they are not amenable to interpretability. This limitation can be circumvented by developing post hoc techniques to explain predictions, giving rise to the area of explainability. Recently, explainability of deep models on images and texts has achieved significant progress. In the area of graph data, graph neural networks (GNNs) and their explainability are experiencing rapid developments. However, there is neither a unified treatment of GNN explainability methods, nor a standard benchmark and testbed for evaluations. In this survey, we provide a unified and taxonomic view of current GNN explainability methods. Our unified and taxonomic treatments of this subject shed lights on the commonalities and differences of existing methods and set the stage for further methodological developments. To facilitate evaluations, we provide a testbed for GNN explainability, including datasets, common algorithms and evaluation metrics. Furthermore, we conduct comprehensive experiments to compare and analyze the performance of many techniques. Altogether, this work provides a unified methodological treatment of GNN explainability and a standardized testbed for evaluations.
Collapse
|
5
|
Ding N, Zhang G, Zhang L, Shen Z, Yin L, Zhou S, Deng Y. Engineering an AI-based forward-reverse platform for the design of cross-ribosome binding sites of a transcription factor biosensor. Comput Struct Biotechnol J 2023; 21:2929-2939. [PMID: 38213883 PMCID: PMC10781712 DOI: 10.1016/j.csbj.2023.04.026] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2023] [Revised: 04/26/2023] [Accepted: 04/26/2023] [Indexed: 01/13/2024] Open
Abstract
A cross-ribosome binding site (cRBS) adjusts the dynamic range of transcription factor-based biosensors (TFBs) by controlling protein expression and folding. The rational design of a cRBS with desired TFB dynamic range remains an important issue in TFB forward and reverse engineering. Here, we report a novel artificial intelligence (AI)-based forward-reverse engineering platform for TFB dynamic range prediction and de novo cRBS design with selected TFB dynamic ranges. The platform demonstrated superior in processing unbalanced minority-class datasets and was guided by sequence characteristics from trained cRBSs. The platform identified correlations between cRBSs and dynamic ranges to mimic bidirectional design between these factors based on Wasserstein generative adversarial network (GAN) with a gradient penalty (GP) (WGAN-GP) and balancing GAN with GP (BAGAN-GP). For forward and reverse engineering, the predictive accuracy was up to 98% and 82%, respectively. Collectively, we generated an AI-based method for the rational design of TFBs with desired dynamic ranges.
Collapse
Affiliation(s)
- Nana Ding
- National Engineering Research Center for Cereal Fermentation and Food Biomanufacturing, Jiangnan University, 1800 Lihu Road, Wuxi, Jiangsu 214122, People’s Republic of China
- Jiangsu Provincial Research Center for Bioactive Product Processing Technology, Jiangnan University, People’s Republic of China
- Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, NO.1239 Siping Road, Shanghai 201210, People’s Republic of China
| | - Guangkun Zhang
- Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, NO.1239 Siping Road, Shanghai 201210, People’s Republic of China
| | - LinPei Zhang
- National Engineering Research Center for Cereal Fermentation and Food Biomanufacturing, Jiangnan University, 1800 Lihu Road, Wuxi, Jiangsu 214122, People’s Republic of China
- Jiangsu Provincial Research Center for Bioactive Product Processing Technology, Jiangnan University, People’s Republic of China
| | - Ziyun Shen
- National Engineering Research Center for Cereal Fermentation and Food Biomanufacturing, Jiangnan University, 1800 Lihu Road, Wuxi, Jiangsu 214122, People’s Republic of China
- Jiangsu Provincial Research Center for Bioactive Product Processing Technology, Jiangnan University, People’s Republic of China
| | - Lianghong Yin
- State Key Laboratory of Subtropical Silviculture, Zhejiang A&F University, Hangzhou 311300, People’s Republic of China
- Zhejiang Provincial Key Laboratory of Resources Protection and Innovation of Traditional Chinese Medicine, Zhejiang A&F University, Hangzhou 311300, People’s Republic of China
| | - Shenghu Zhou
- National Engineering Research Center for Cereal Fermentation and Food Biomanufacturing, Jiangnan University, 1800 Lihu Road, Wuxi, Jiangsu 214122, People’s Republic of China
- Jiangsu Provincial Research Center for Bioactive Product Processing Technology, Jiangnan University, People’s Republic of China
| | - Yu Deng
- National Engineering Research Center for Cereal Fermentation and Food Biomanufacturing, Jiangnan University, 1800 Lihu Road, Wuxi, Jiangsu 214122, People’s Republic of China
- Jiangsu Provincial Research Center for Bioactive Product Processing Technology, Jiangnan University, People’s Republic of China
| |
Collapse
|
6
|
Liu C, Wang D, Zhang H, Wu W, Sun W, Zhao T, Zheng N. Using Simulated Training Data of Voxel-Level Generative Models to Improve 3D Neuron Reconstruction. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:3624-3635. [PMID: 35834465 DOI: 10.1109/tmi.2022.3191011] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Reconstructing neuron morphologies from fluorescence microscope images plays a critical role in neuroscience studies. It relies on image segmentation to produce initial masks either for further processing or final results to represent neuronal morphologies. This has been a challenging step due to the variation and complexity of noisy intensity patterns in neuron images acquired from microscopes. Whereas progresses in deep learning have brought the goal of accurate segmentation much closer to reality, creating training data for producing powerful neural networks is often laborious. To overcome the difficulty of obtaining a vast number of annotated data, we propose a novel strategy of using two-stage generative models to simulate training data with voxel-level labels. Trained upon unlabeled data by optimizing a novel objective function of preserving predefined labels, the models are able to synthesize realistic 3D images with underlying voxel labels. We showed that these synthetic images could train segmentation networks to obtain even better performance than manually labeled data. To demonstrate an immediate impact of our work, we further showed that segmentation results produced by networks trained upon synthetic data could be used to improve existing neuron reconstruction methods.
Collapse
|
7
|
Khadangi A, Boudier T, Hanssen E, Rajagopal V. CardioVinci: building blocks for virtual cardiac cells using deep learning. Philos Trans R Soc Lond B Biol Sci 2022; 377:20210469. [PMID: 36189496 PMCID: PMC9527637 DOI: 10.1098/rstb.2021.0469] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Advances in electron microscopy (EM) such as electron tomography and focused ion-beam scanning electron microscopy provide unprecedented, three-dimensional views of cardiac ultrastructures within sample volumes ranging from hundreds of nanometres to hundreds of micrometres. The datasets from these samples are typically large, with file sizes ranging from gigabytes to terabytes and the number of image slices within the three-dimensional stack in the hundreds. A significant bottleneck with these large datasets is the time taken to extract and statistically analyse three-dimensional changes in cardiac ultrastructures. This is because of the inherently low contrast and the significant amount of structural detail that is present in EM images. These datasets often require manual annotation, which needs substantial person-hours and may result in only partial segmentation that makes quantitative analysis of the three-dimensional volumes infeasible. We present CardioVinci, a deep learning workflow to automatically segment and statistically quantify the morphologies and spatial assembly of mitochondria, myofibrils and Z-discs with minimal manual annotation. The workflow encodes a probabilistic model of the three-dimensional cardiomyocyte using a generative adversarial network. This generative model can be used to create new models of cardiomyocyte architecture that reflect variations in morphologies and cell architecture found in EM datasets. This article is part of the theme issue ‘The cardiomyocyte: new revelations on the interplay between architecture and function in growth, health, and disease’.
Collapse
Affiliation(s)
- Afshin Khadangi
- Department of Biomedical Engineering, Faculty of Engineering and Information Technology, University of Melbourne, Parkville, Australia
| | - Thomas Boudier
- Institut de Biologie Paris-Seine, Sorbonne Université Campus Pierre et Marie Curie, Paris, France
| | - Eric Hanssen
- Ian Holmes Imaging Center, Bio21, University of Melbourne, Parkville, Victoria, Australia
| | - Vijay Rajagopal
- Department of Biomedical Engineering, Faculty of Engineering and Information Technology, University of Melbourne, Parkville, Australia
| |
Collapse
|
8
|
Wei Z, Liu X, Yan R, Sun G, Yu W, Liu Q, Guo Q. Pixel-level multimodal fusion deep networks for predicting subcellular organelle localization from label-free live-cell imaging. Front Genet 2022; 13:1002327. [PMID: 36386823 PMCID: PMC9644055 DOI: 10.3389/fgene.2022.1002327] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2022] [Accepted: 09/26/2022] [Indexed: 01/25/2023] Open
Abstract
Complex intracellular organizations are commonly represented by dividing the metabolic process of cells into different organelles. Therefore, identifying sub-cellular organelle architecture is significant for understanding intracellular structural properties, specific functions, and biological processes in cells. However, the discrimination of these structures in the natural organizational environment and their functional consequences are not clear. In this article, we propose a new pixel-level multimodal fusion (PLMF) deep network which can be used to predict the location of cellular organelle using label-free cell optical microscopy images followed by deep-learning-based automated image denoising. It provides valuable insights that can be of tremendous help in improving the specificity of label-free cell optical microscopy by using the Transformer-Unet network to predict the ground truth imaging which corresponds to different sub-cellular organelle architectures. The new prediction method proposed in this article combines the advantages of a transformer's global prediction and CNN's local detail analytic ability of background features for label-free cell optical microscopy images, so as to improve the prediction accuracy. Our experimental results showed that the PLMF network can achieve over 0.91 Pearson's correlation coefficient (PCC) correlation between estimated and true fractions on lung cancer cell-imaging datasets. In addition, we applied the PLMF network method on the cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new way for the time-resolved study of subcellular components in different cells, especially for cancer cells.
Collapse
Affiliation(s)
- Zhihao Wei
- Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology, Beijing, China
| | - Xi Liu
- Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology, Beijing, China
| | - Ruiqing Yan
- Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology, Beijing, China
| | - Guocheng Sun
- Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology, Beijing, China,School of Mechanical Engineering & Hydrogen Energy Research Centre, Beijing Institute of Petrochemical Technology, Beijing, China
| | - Weiyong Yu
- Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology, Beijing, China
| | - Qiang Liu
- Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology, Beijing, China
| | - Qianjin Guo
- Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology, Beijing, China,School of Mechanical Engineering & Hydrogen Energy Research Centre, Beijing Institute of Petrochemical Technology, Beijing, China,*Correspondence: Qianjin Guo,
| |
Collapse
|
9
|
Multiple Parallel Fusion Network for Predicting Protein Subcellular Localization from Stimulated Raman Scattering (SRS) Microscopy Images in Living Cells. Int J Mol Sci 2022; 23:ijms231810827. [PMID: 36142736 PMCID: PMC9504098 DOI: 10.3390/ijms231810827] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2022] [Revised: 09/10/2022] [Accepted: 09/13/2022] [Indexed: 11/23/2022] Open
Abstract
Stimulated Raman Scattering Microscopy (SRS) is a powerful tool for label-free detailed recognition and investigation of the cellular and subcellular structures of living cells. Determining subcellular protein localization from the cell level of SRS images is one of the basic goals of cell biology, which can not only provide useful clues for their functions and biological processes but also help to determine the priority and select the appropriate target for drug development. However, the bottleneck in predicting subcellular protein locations of SRS cell imaging lies in modeling complicated relationships concealed beneath the original cell imaging data owing to the spectral overlap information from different protein molecules. In this work, a multiple parallel fusion network, MPFnetwork, is proposed to study the subcellular locations from SRS images. This model used a multiple parallel fusion model to construct feature representations and combined multiple nonlinear decomposing algorithms as the automated subcellular detection method. Our experimental results showed that the MPFnetwork could achieve over 0.93 dice correlation between estimated and true fractions on SRS lung cancer cell datasets. In addition, we applied the MPFnetwork method to cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new method for the time-resolved study of subcellular components in different cells, especially cancer cells.
Collapse
|
10
|
Contreras M, Bachman W, Long DS. Discrete protein metric (DPM): A new image similarity metric to calculate accuracy of deep learning-generated cell focal adhesion predictions. Micron 2022; 160:103302. [PMID: 35689876 PMCID: PMC10228147 DOI: 10.1016/j.micron.2022.103302] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2022] [Revised: 05/13/2022] [Accepted: 05/13/2022] [Indexed: 01/21/2023]
Abstract
Understanding cell behaviors can provide new knowledge on the development of different pathologies. Focal adhesion (FA) sites are important sub-cellular structures that are involved in these processes. To better facilitate the study of FA sites, deep learning (DL) can be used to predict FA site morphology based on limited microscopic datasets (e.g., cell membrane images). However, calculating the accuracy score of these predictions can be challenging due to the discrete/point pattern like nature of FA sites. In the present work, a new image similarity metric, discrete protein metric (DPM), was developed to calculate FA prediction accuracy. This metric measures differences in distribution (d), shape/size (s), and angle (a) of FA sites between predicted and ground truth microscopy images. Performance of the DPM was evaluated by comparing it to three other commonly used image similarity metrics: Pearson correlation coefficient (PCC), feature similarity index (FSIM), and Intersection over Union (IoU). A sensitivity analysis was performed by comparing changes in each metric value due to quantifiable changes in FA site location, number, aspect ratio, area, or orientation. Furthermore, accuracy score of DL-generated predictions was calculated using all four metrics to compare their ability to capture variation across samples. Results showed better sensitivity and range of variation for DPM compared to the other metrics tested. Most importantly, DPM had the ability to determine which FA predictions were quantitatively more accurate and consistent with qualitative assessments. The proposed DPM hence provides a method to validate DL-generated FA predictions and has the potential to be used for investigation of other sub-cellular protein aggregates relevant to cell biology.
Collapse
Affiliation(s)
- Miguel Contreras
- Mechanobiology and Biomedicine Lab, Department of Biomedical Engineering, Wichita State University, Wichita KS USA
| | - William Bachman
- Mechanobiology and Biomedicine Lab, Department of Biomedical Engineering, Wichita State University, Wichita KS USA
| | - David S Long
- Mechanobiology and Biomedicine Lab, Department of Biomedical Engineering, Wichita State University, Wichita KS USA.
| |
Collapse
|
11
|
Yuan H, Cai L, Hu X, Wang J, Ji S. Interpreting Image Classifiers by Generating Discrete Masks. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:2019-2030. [PMID: 33021938 DOI: 10.1109/tpami.2020.3028783] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Deep models are commonly treated as black-boxes and lack interpretability. Here, we propose a novel approach to interpret deep image classifiers by generating discrete masks. Our method follows the generative adversarial network formalism. The deep model to be interpreted is the discriminator while we train a generator to explain it. The generator is trained to capture discriminative image regions that should convey the same or similar meaning as the original image from the model's perspective. It produces a probability map from which a discrete mask can be sampled. Then the discriminator is used to measure the quality of the sampled mask and provide feedbacks for updating. Due to the sampling operations, the generator cannot be trained directly by back-propagation. We propose to update it using policy gradient. Furthermore, we propose to incorporate gradients as auxiliary information to reduce the search space and facilitate training. We conduct both quantitative and qualitative experiments on the ILSVRC dataset. Experimental results indicate that our method can provide reasonable explanations for predictions and outperform existing approaches. In addition, our method can pass the model randomization test, indicating that it is reasoning the attribution of network predictions.
Collapse
|
12
|
Zaritsky A, Jamieson AR, Welf ES, Nevarez A, Cillay J, Eskiocak U, Cantarel BL, Danuser G. Interpretable deep learning uncovers cellular properties in label-free live cell images that are predictive of highly metastatic melanoma. Cell Syst 2021; 12:733-747.e6. [PMID: 34077708 PMCID: PMC8353662 DOI: 10.1016/j.cels.2021.05.003] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2020] [Revised: 01/22/2021] [Accepted: 05/07/2021] [Indexed: 12/22/2022]
Abstract
Deep learning has emerged as the technique of choice for identifying hidden patterns in cell imaging data but is often criticized as "black box." Here, we employ a generative neural network in combination with supervised machine learning to classify patient-derived melanoma xenografts as "efficient" or "inefficient" metastatic, validate predictions regarding melanoma cell lines with unknown metastatic efficiency in mouse xenografts, and use the network to generate in silico cell images that amplify the critical predictive cell properties. These exaggerated images unveiled pseudopodial extensions and increased light scattering as hallmark properties of metastatic cells. We validated this interpretation using live cells spontaneously transitioning between states indicative of low and high metastatic efficiency. This study illustrates how the application of artificial intelligence can support the identification of cellular properties that are predictive of complex phenotypes and integrated cell functions but are too subtle to be identified in the raw imagery by a human expert. A record of this paper's transparent peer review process is included in the supplemental information. VIDEO ABSTRACT.
Collapse
Affiliation(s)
- Assaf Zaritsky
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX 75390, USA; Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev, Beer-Sheva 84105, Israel.
| | - Andrew R Jamieson
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX 75390, USA
| | - Erik S Welf
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX 75390, USA
| | - Andres Nevarez
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX 75390, USA; Section of Molecular Biology, Division of Biological Sciences, University of California San Diego, 9500 Gilman Drive, San Diego, La Jolla, CA 92093, USA
| | - Justin Cillay
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX 75390, USA
| | - Ugur Eskiocak
- Children's Research Institute and Department of Pediatrics, University of Texas Southwestern Medical Center, Dallas, TX 75390, USA
| | - Brandi L Cantarel
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX 75390, USA
| | - Gaudenz Danuser
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX 75390, USA.
| |
Collapse
|
13
|
Lee KCM, Guck J, Goda K, Tsia KK. Toward deep biophysical cytometry: prospects and challenges. Trends Biotechnol 2021; 39:1249-1262. [PMID: 33895013 DOI: 10.1016/j.tibtech.2021.03.006] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Revised: 03/15/2021] [Accepted: 03/15/2021] [Indexed: 12/13/2022]
Abstract
The biophysical properties of cells reflect their identities, underpin their homeostatic state in health, and define the pathogenesis of disease. Recent leapfrogging advances in biophysical cytometry now give access to this information, which is obscured in molecular assays, with a discriminative power that was once inconceivable. However, biophysical cytometry should go 'deeper' in terms of exploiting the information-rich cellular biophysical content, generating a molecular knowledge base of cellular biophysical properties, and standardizing the protocols for wider dissemination. Overcoming these barriers, which requires concurrent innovations in microfluidics, optical imaging, and computer vision, could unleash the enormous potential of biophysical cytometry not only for gaining a new mechanistic understanding of biological systems but also for identifying new cost-effective biomarkers of disease.
Collapse
Affiliation(s)
- Kelvin C M Lee
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Pokfulam Road, Hong Kong
| | - Jochen Guck
- Max Planck Institute for the Science of Light, and Max-Planck-Zentrum für Physik und Medizin, 91058 Erlangen, Germany; Department of Physics, Friedrich-Alexander Universität Erlangen-Nürnberg, 91058 Erlangen, Germany
| | - Keisuke Goda
- Department of Chemistry, The University of Tokyo, Tokyo 113-0033, Japan; Institute of Technological Sciences, Wuhan University, Hubei 430072, China; Department of Bioengineering, University of California, Los Angeles, California 90095, USA
| | - Kevin K Tsia
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Pokfulam Road, Hong Kong; Advanced Biomedical Instrumentation Centre, Hong Kong Science Park, Shatin, New Territories, Hong Kong.
| |
Collapse
|
14
|
A versatile deep learning architecture for classification and label-free prediction of hyperspectral images. NAT MACH INTELL 2021; 3:306-315. [PMID: 34676358 PMCID: PMC8528004 DOI: 10.1038/s42256-021-00309-y] [Citation(s) in RCA: 36] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
Hyperspectral imaging is a technique that provides rich chemical or compositional information not regularly available to traditional imaging modalities such as intensity imaging or color imaging based on the reflection, transmission, or emission of light. Analysis of hyperspectral imaging often relies on machine learning methods to extract information. Here, we present a new flexible architecture, the U-within-U-Net, that can perform classification, segmentation, and prediction of orthogonal imaging modalities on a variety of hyperspectral imaging techniques. Specifically, we demonstrate feature segmentation and classification on the Indian Pines hyperspectral dataset and simultaneous location prediction of multiple drugs in mass spectrometry imaging of rat liver tissue. We further demonstrate label-free fluorescence image prediction from hyperspectral stimulated Raman scattering microscopy images. The applicability of the U-within-U-Net architecture on diverse datasets with widely varying input and output dimensions and data sources suggest that it has great potential in advancing the use of hyperspectral imaging across many different application areas ranging from remote sensing, to medical imaging, to microscopy.
Collapse
|
15
|
|
16
|
Liu Y, Yuan H, Wang Z, Ji S. Global Pixel Transformers for Virtual Staining of Microscopy Images. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:2256-2266. [PMID: 31985413 DOI: 10.1109/tmi.2020.2968504] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Visualizing the details of different cellular structures is of great importance to elucidate cellular functions. However, it is challenging to obtain high quality images of different structures directly due to complex cellular environments. Fluorescence staining is a popular technique to label different structures but has several drawbacks. In particular, label staining is time consuming and may affect cell morphology, and simultaneous labels are inherently limited. This raises the need of building computational models to learn relationships between unlabeled microscopy images and labeled fluorescence images, and to infer fluorescence labels of other microscopy images excluding the physical staining process. We propose to develop a novel deep model for virtual staining of unlabeled microscopy images. We first propose a novel network layer, known as the global pixel transformer layer, that fuses global information from inputs effectively. The proposed global pixel transformer layer can generate outputs with arbitrary dimensions, and can be employed for all the regular, down-sampling, and up-sampling operators. We then incorporate our proposed global pixel transformer layers and dense blocks to build an U-Net like network. We believe such a design can promote feature reusing between layers. In addition, we propose a multi-scale input strategy to encourage networks to capture features at different scales. We conduct evaluations across various fluorescence image prediction tasks to demonstrate the effectiveness of our approach. Both quantitative and qualitative results show that our method outperforms the state-of-the-art approach significantly. It is also shown that our proposed global pixel transformer layer is useful to improve the fluorescence image prediction results.
Collapse
|
17
|
Belthangady C, Royer LA. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat Methods 2019; 16:1215-1225. [DOI: 10.1038/s41592-019-0458-z] [Citation(s) in RCA: 204] [Impact Index Per Article: 40.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2018] [Accepted: 05/22/2019] [Indexed: 02/06/2023]
|