1
|
Ritter C, Lee JY, Pham MT, Pabba MK, Cardoso MC, Bartenschlager R, Rohr K. Multi-detector fusion and Bayesian smoothing for tracking viral and chromatin structures. Med Image Anal 2024; 97:103227. [PMID: 38897031 DOI: 10.1016/j.media.2024.103227] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2022] [Revised: 08/15/2023] [Accepted: 05/27/2024] [Indexed: 06/21/2024]
Abstract
Automatic tracking of viral and intracellular structures displayed as spots with varying sizes in fluorescence microscopy images is an important task to quantify cellular processes. We propose a novel probabilistic tracking approach for multiple particle tracking based on multi-detector and multi-scale data fusion as well as Bayesian smoothing. The approach integrates results from multiple detectors using a novel intensity-based covariance intersection method which takes into account information about the image intensities, positions, and uncertainties. The method ensures a consistent estimate of multiple fused particle detections and does not require an optimization step. Our probabilistic tracking approach performs data fusion of detections from classical and deep learning methods as well as exploits single-scale and multi-scale detections. In addition, we use Bayesian smoothing to fuse information of predictions from both past and future time points. We evaluated our approach using image data of the Particle Tracking Challenge and achieved state-of-the-art results or outperformed previous methods. Our method was also assessed on challenging live cell fluorescence microscopy image data of viral and cellular proteins expressed in hepatitis C virus-infected cells and chromatin structures in non-infected cells, acquired at different spatial-temporal resolutions. We found that the proposed approach outperforms existing methods.
Collapse
Affiliation(s)
- C Ritter
- Biomedical Computer Vision Group, BioQuant, IPMB, Heidelberg University, Im Neuenheimer Feld 267, Heidelberg, Germany.
| | - J-Y Lee
- Department of Infectious Diseases, Molecular Virology, Heidelberg University, Im Neuenheimer Feld 344, Heidelberg, Germany; German Center for Infection Research (DZIF), Heidelberg Partner Site, Germany
| | - M-T Pham
- Department of Infectious Diseases, Molecular Virology, Heidelberg University, Im Neuenheimer Feld 344, Heidelberg, Germany; German Center for Infection Research (DZIF), Heidelberg Partner Site, Germany
| | - M K Pabba
- Department of Biology, Cell Biology and Epigenetics, Technical University of Darmstadt, Schnittspahnstraße 10, Darmstadt, Germany
| | - M C Cardoso
- Department of Biology, Cell Biology and Epigenetics, Technical University of Darmstadt, Schnittspahnstraße 10, Darmstadt, Germany
| | - R Bartenschlager
- Department of Infectious Diseases, Molecular Virology, Heidelberg University, Im Neuenheimer Feld 344, Heidelberg, Germany; German Center for Infection Research (DZIF), Heidelberg Partner Site, Germany
| | - K Rohr
- Biomedical Computer Vision Group, BioQuant, IPMB, Heidelberg University, Im Neuenheimer Feld 267, Heidelberg, Germany.
| |
Collapse
|
2
|
Roudot P, Legant WR, Zou Q, Dean KM, Isogai T, Welf ES, David AF, Gerlich DW, Fiolka R, Betzig E, Danuser G. u-track3D: Measuring, navigating, and validating dense particle trajectories in three dimensions. CELL REPORTS METHODS 2023; 3:100655. [PMID: 38042149 PMCID: PMC10783629 DOI: 10.1016/j.crmeth.2023.100655] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/12/2023] [Revised: 08/10/2023] [Accepted: 11/09/2023] [Indexed: 12/04/2023]
Abstract
We describe u-track3D, a software package that extends the versatile u-track framework established in 2D to address the specific challenges of 3D particle tracking. First, we present the performance of the new package in quantifying a variety of intracellular dynamics imaged by multiple 3D microcopy platforms and on the standard 3D test dataset of the particle tracking challenge. These analyses indicate that u-track3D presents a tracking solution that is competitive to both conventional and deep-learning-based approaches. We then present the concept of dynamic region of interest (dynROI), which allows an experimenter to interact with dynamic 3D processes in 2D views amenable to visual inspection. Third, we present an estimator of trackability that automatically defines a score for every trajectory, thereby overcoming the challenges of trajectory validation by visual inspection. With these combined strategies, u-track3D provides a complete framework for unbiased studies of molecular processes in complex volumetric sequences.
Collapse
Affiliation(s)
- Philippe Roudot
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA; Aix Marseille University, CNRS, Centrale Marseille, I2M, Turing Centre for Living Systems, Marseille, France.
| | - Wesley R Legant
- Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill, North Carolina State University, Chapel Hill, NC, USA; Department of Pharmacology, University of North Carolina, Chapel Hill, NC, USA
| | - Qiongjing Zou
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Kevin M Dean
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Tadamoto Isogai
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Erik S Welf
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Ana F David
- Institute of Molecular Biotechnology of the Austrian Academy of Sciences, Vienna BioCenter, Vienna, Austria
| | - Daniel W Gerlich
- Institute of Molecular Biotechnology of the Austrian Academy of Sciences, Vienna BioCenter, Vienna, Austria
| | - Reto Fiolka
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Eric Betzig
- Department of Molecular & Cell Biology, University of California, Berkeley, Berkeley, CA, USA
| | - Gaudenz Danuser
- Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA.
| |
Collapse
|
3
|
Geometric deep learning reveals the spatiotemporal features of microscopic motion. NAT MACH INTELL 2023. [DOI: 10.1038/s42256-022-00595-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
Abstract
AbstractThe characterization of dynamical processes in living systems provides important clues for their mechanistic interpretation and link to biological functions. Owing to recent advances in microscopy techniques, it is now possible to routinely record the motion of cells, organelles and individual molecules at multiple spatiotemporal scales in physiological conditions. However, the automated analysis of dynamics occurring in crowded and complex environments still lags behind the acquisition of microscopic image sequences. Here we present a framework based on geometric deep learning that achieves the accurate estimation of dynamical properties in various biologically relevant scenarios. This deep-learning approach relies on a graph neural network enhanced by attention-based components. By processing object features with geometric priors, the network is capable of performing multiple tasks, from linking coordinates into trajectories to inferring local and global dynamic properties. We demonstrate the flexibility and reliability of this approach by applying it to real and simulated data corresponding to a broad range of biological experiments.
Collapse
|
4
|
Del Giudice F, Barnes C. Rapid Temperature-Dependent Rheological Measurements of Non-Newtonian Solutions Using a Machine-Learning Aided Microfluidic Rheometer. Anal Chem 2022; 94:3617-3628. [PMID: 35167252 DOI: 10.1021/acs.analchem.1c05208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Biofluids such as synovial fluid, blood plasma, and saliva contain several proteins which impart non-Newtonian properties to the biofluids. The concentration of such protein macromolecules in biofluids is regarded as an important biomarker for the diagnosis of several health conditions, including cardiovascular disorders, joint quality, and Alzheimer's. Existing technologies for the measurements of macromolecules in biofluids are limited; they require a long turnaround time, or require complex protocols, thus calling for alternative, more suitable, methodologies aimed at such measurements. According to the well-established relations for polymer solutions, the concentration of macromolecules in solutions can also be derived via measurement of rheological properties such as shear-viscosity and the longest relaxation time. We here introduce a microfluidic rheometer for rapid simultaneous measurement of shear viscosity and longest relaxation time of non-Newtonian solutions at different temperatures. At variance with previous technologies, our microfluidic rheometer provides a very short turnaround time of around 2 min or less thanks to the implementation of a machine-learning algorithm. We validated our platform on several aqueous solutions of poly(ethylene oxide). We also performed measurements on hyaluronic acid solutions in the clinical range for joint grade assessment. We observed monotonic behavior with the concentration for both rheological properties, thus speculating on their use as potential rheo-markers, i.e., rheological biomarkers, across several disease states.
Collapse
Affiliation(s)
- Francesco Del Giudice
- Department of Chemical Engineering, Faculty of Science and Engineering, School of Engineering and Applied Science, Swansea University Fabian Way, Swansea, SA1 8EN, United Kingdom
| | - Claire Barnes
- Department of Biomedical Engineering, Faculty of Science and Engineering, School of Engineering and Applied Science, Swansea University Fabian Way, Swansea, SA1 8EN, United Kingdom
| |
Collapse
|
5
|
Cheng HJ, Hsu CH, Hung CL, Lin CY. A review for Cell and Particle Tracking on Microscopy Images using Algorithms and Deep Learning Technologies. Biomed J 2021; 45:465-471. [PMID: 34628059 PMCID: PMC9421944 DOI: 10.1016/j.bj.2021.10.001] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2021] [Revised: 09/30/2021] [Accepted: 10/01/2021] [Indexed: 01/06/2023] Open
Abstract
Time-lapse microscopy images generated by biological experiments have been widely used for observing target activities, such as the motion trajectories and survival states. Based on these observations, biologists can conclude experimental results or present new hypotheses for several biological applications, i.e. virus research or drug design. Many methods or tools have been proposed in the past to observe cell and particle activities, which are defined as single cell tracking and single particle tracking problems, by using algorithms and deep learning technologies. In this article, a review for these works is presented in order to summarize the past methods and research topics at first, then points out the problems raised by these works, and finally proposes future research directions. The contributions of this article will help researchers to understand past development trends and further propose innovative technologies.
Collapse
Affiliation(s)
- Hui-Jun Cheng
- Affiliated Cancer Hospital & Institute of Guangzhou Medical University, Guangzhou 510095, China; Department of Computer Science and Information Engineering, Providence University, Taichung 43301, Taiwan
| | - Ching-Hsien Hsu
- Department of Computer Science and Information Engineering, Asia University, Taichung 41354, Taiwan; Guangdong-Hong Kong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, School of Mathematics and Big Data, Foshan University, Foshan 528000, China; Department of Medical Research, China Medical University Hospital, China Medical University, Taiwan
| | - Che-Lun Hung
- Institute of Biomedical Informatics, National Yang Ming Chiao Tung University, Taipei 11221, Taiwan; Department of Computer Science and Communication Engineering, Providence University, Taichung 43301, Taiwan
| | - Chun-Yuan Lin
- Department of Computer Science and Information Engineering, Asia University, Taichung 41354, Taiwan; Department of Computer Science and Information Engineering, Chang Gung University, Taoyuan 33302, Taiwan.
| |
Collapse
|
6
|
Lagache T, Hanson A, Pérez-Ortega JE, Fairhall A, Yuste R. Tracking calcium dynamics from individual neurons in behaving animals. PLoS Comput Biol 2021; 17:e1009432. [PMID: 34624016 PMCID: PMC8528277 DOI: 10.1371/journal.pcbi.1009432] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2020] [Revised: 10/20/2021] [Accepted: 09/08/2021] [Indexed: 12/03/2022] Open
Abstract
Measuring the activity of neuronal populations with calcium imaging can capture emergent functional properties of neuronal circuits with single cell resolution. However, the motion of freely behaving animals, together with the intermittent detectability of calcium sensors, can hinder automatic monitoring of neuronal activity and their subsequent functional characterization. We report the development and open-source implementation of a multi-step cellular tracking algorithm (Elastic Motion Correction and Concatenation or EMC2) that compensates for the intermittent disappearance of moving neurons by integrating local deformation information from detectable neurons. We demonstrate the accuracy and versatility of our algorithm using calcium imaging data from two-photon volumetric microscopy in visual cortex of awake mice, and from confocal microscopy in behaving Hydra, which experiences major body deformation during its contractions. We quantify the performance of our algorithm using ground truth manual tracking of neurons, along with synthetic time-lapse sequences, covering a wide range of particle motions and detectability parameters. As a demonstration of the utility of the algorithm, we monitor for several days calcium activity of the same neurons in layer 2/3 of mouse visual cortex in vivo, finding significant turnover within the active neurons across days, with only few neurons that remained active across days. Also, combining automatic tracking of single neuron activity with statistical clustering, we characterize and map neuronal ensembles in behaving Hydra, finding three major non-overlapping ensembles of neurons (CB, RP1 and RP2) whose activity correlates with contractions and elongations. Our results show that the EMC2 algorithm can be used as a robust and versatile platform for neuronal tracking in behaving animals.
Collapse
Affiliation(s)
- Thibault Lagache
- Department of Biological Sciences, Columbia University, New York, New York, United States of America
- Marine Biological Laboratory, Woods Hole, Massachusetts, United States of America
| | - Alison Hanson
- Department of Biological Sciences, Columbia University, New York, New York, United States of America
- Marine Biological Laboratory, Woods Hole, Massachusetts, United States of America
- Department of Psychiatry, New York State Psychiatric Institute, Columbia University, New York, New York, United States of America
| | - Jesús E Pérez-Ortega
- Department of Biological Sciences, Columbia University, New York, New York, United States of America
| | - Adrienne Fairhall
- Department of Physiology and Biophysics, University of Washington, Seattle, Washington, United States of America
- UW Computational Neuroscience Center, University of Washington, Seattle, Washington, United States of America
| | - Rafael Yuste
- Department of Biological Sciences, Columbia University, New York, New York, United States of America
- Marine Biological Laboratory, Woods Hole, Massachusetts, United States of America
- Donostia International Physics Center, San Sebastian, Spain
| |
Collapse
|
7
|
Deep probabilistic tracking of particles in fluorescence microscopy images. Med Image Anal 2021; 72:102128. [PMID: 34229189 DOI: 10.1016/j.media.2021.102128] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2020] [Revised: 05/14/2021] [Accepted: 05/26/2021] [Indexed: 01/16/2023]
Abstract
Tracking of particles in temporal fluorescence microscopy image sequences is of fundamental importance to quantify dynamic processes of intracellular structures as well as virus structures. We introduce a probabilistic deep learning approach for fluorescent particle tracking, which is based on a recurrent neural network that mimics classical Bayesian filtering. Compared to previous deep learning methods for particle tracking, our approach takes into account uncertainty, both aleatoric and epistemic uncertainty. Thus, information about the reliability of the computed trajectories is determined. Manual tuning of tracking parameters is not necessary and prior knowledge about the noise statistics is not required. Short and long-term temporal dependencies of individual object dynamics are exploited for state prediction, and assigned detections are used to update the predicted states. For correspondence finding, we introduce a neural network which computes assignment probabilities jointly across multiple detections as well as determines the probabilities of missing detections. Training requires only simulated data and therefore tedious manual annotation of ground truth is not needed. We performed a quantitative performance evaluation based on synthetic and real 2D as well as 3D fluorescence microscopy images. We used image data of the Particle Tracking Challenge as well as real time-lapse fluorescence microscopy images displaying virus structures and chromatin structures. It turned out that our approach yields state-of-the-art results or improves the tracking results compared to previous methods.
Collapse
|
8
|
Pratapa A, Doron M, Caicedo JC. Image-based cell phenotyping with deep learning. Curr Opin Chem Biol 2021; 65:9-17. [PMID: 34023800 DOI: 10.1016/j.cbpa.2021.04.001] [Citation(s) in RCA: 44] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2021] [Accepted: 04/10/2021] [Indexed: 12/25/2022]
Abstract
A cell's phenotype is the culmination of several cellular processes through a complex network of molecular interactions that ultimately result in a unique morphological signature. Visual cell phenotyping is the characterization and quantification of these observable cellular traits in images. Recently, cellular phenotyping has undergone a massive overhaul in terms of scale, resolution, and throughput, which is attributable to advances across electronic, optical, and chemical technologies for imaging cells. Coupled with the rapid acceleration of deep learning-based computational tools, these advances have opened up new avenues for innovation across a wide variety of high-throughput cell biology applications. Here, we review applications wherein deep learning is powering the recognition, profiling, and prediction of visual phenotypes to answer important biological questions. As the complexity and scale of imaging assays increase, deep learning offers computational solutions to elucidate the details of previously unexplored cellular phenotypes.
Collapse
|
9
|
Meijering E. A bird's-eye view of deep learning in bioimage analysis. Comput Struct Biotechnol J 2020; 18:2312-2325. [PMID: 32994890 PMCID: PMC7494605 DOI: 10.1016/j.csbj.2020.08.003] [Citation(s) in RCA: 62] [Impact Index Per Article: 12.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2020] [Revised: 07/26/2020] [Accepted: 08/01/2020] [Indexed: 02/07/2023] Open
Abstract
Deep learning of artificial neural networks has become the de facto standard approach to solving data analysis problems in virtually all fields of science and engineering. Also in biology and medicine, deep learning technologies are fundamentally transforming how we acquire, process, analyze, and interpret data, with potentially far-reaching consequences for healthcare. In this mini-review, we take a bird's-eye view at the past, present, and future developments of deep learning, starting from science at large, to biomedical imaging, and bioimage analysis in particular.
Collapse
Affiliation(s)
- Erik Meijering
- School of Computer Science and Engineering & Graduate School of Biomedical Engineering, University of New South Wales, Sydney, Australia
| |
Collapse
|