1
|
Gouzou D, Taimori A, Haloubi T, Finlayson N, Wang Q, Hopgood JR, Vallejo M. Applications of machine learning in time-domain fluorescence lifetime imaging: a review. Methods Appl Fluoresc 2024; 12:022001. [PMID: 38055998 PMCID: PMC10851337 DOI: 10.1088/2050-6120/ad12f7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 09/25/2023] [Accepted: 12/06/2023] [Indexed: 12/08/2023]
Abstract
Many medical imaging modalities have benefited from recent advances in Machine Learning (ML), specifically in deep learning, such as neural networks. Computers can be trained to investigate and enhance medical imaging methods without using valuable human resources. In recent years, Fluorescence Lifetime Imaging (FLIm) has received increasing attention from the ML community. FLIm goes beyond conventional spectral imaging, providing additional lifetime information, and could lead to optical histopathology supporting real-time diagnostics. However, most current studies do not use the full potential of machine/deep learning models. As a developing image modality, FLIm data are not easily obtainable, which, coupled with an absence of standardisation, is pushing back the research to develop models which could advance automated diagnosis and help promote FLIm. In this paper, we describe recent developments that improve FLIm image quality, specifically time-domain systems, and we summarise sensing, signal-to-noise analysis and the advances in registration and low-level tracking. We review the two main applications of ML for FLIm: lifetime estimation and image analysis through classification and segmentation. We suggest a course of action to improve the quality of ML studies applied to FLIm. Our final goal is to promote FLIm and attract more ML practitioners to explore the potential of lifetime imaging.
Collapse
Affiliation(s)
- Dorian Gouzou
- Dorian Gouzou and Marta Vallejo are with Institute of Signals, Sensors and Systems, School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh, EH14 4AS, United Kingdom
| | - Ali Taimori
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Tarek Haloubi
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Neil Finlayson
- Neil Finlayson is with Institute for Integrated Micro and Nano Systems, School of Engineering, University ofEdinburgh, Edinburgh EH9 3FF, United Kingdom
| | - Qiang Wang
- Qiang Wang is with Centre for Inflammation Research, University of Edinburgh, Edinburgh, EH16 4TJ, United Kingdom
| | - James R Hopgood
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Marta Vallejo
- Dorian Gouzou and Marta Vallejo are with Institute of Signals, Sensors and Systems, School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh, EH14 4AS, United Kingdom
| |
Collapse
|
2
|
Prinke P, Haueisen J, Klee S, Rizqie MQ, Supriyanto E, König K, Breunig HG, Piątek Ł. Automatic segmentation of skin cells in multiphoton data using multi-stage merging. Sci Rep 2021; 11:14534. [PMID: 34267247 PMCID: PMC8282875 DOI: 10.1038/s41598-021-93682-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2021] [Accepted: 06/27/2021] [Indexed: 01/10/2023] Open
Abstract
We propose a novel automatic segmentation algorithm that separates the components of human skin cells from the rest of the tissue in fluorescence data of three-dimensional scans using non-invasive multiphoton tomography. The algorithm encompasses a multi-stage merging on preprocessed superpixel images to ensure independence from a single empirical global threshold. This leads to a high robustness of the segmentation considering the depth-dependent data characteristics, which include variable contrasts and cell sizes. The subsequent classification of cell cytoplasm and nuclei are based on a cell model described by a set of four features. Two novel features, a relationship between outer cell and inner nucleus (OCIN) and a stability index, were derived. The OCIN feature describes the topology of the model, while the stability index indicates segment quality in the multi-stage merging process. These two new features, combined with the local gradient magnitude and compactness, are used for the model-based fuzzy evaluation of the cell segments. We exemplify our approach on an image stack with 200 × 200 × 100 μm3, including the skin layers of the stratum spinosum and the stratum basale of a healthy volunteer. Our image processing pipeline contributes to the fully automated classification of human skin cells in multiphoton data and provides a basis for the detection of skin cancer using non-invasive optical biopsy.
Collapse
Affiliation(s)
- Philipp Prinke
- Institute for Biomedical Engineering and Informatics, Technische Universität Ilmenau, 98693, Ilmenau, Germany.
| | - Jens Haueisen
- Institute for Biomedical Engineering and Informatics, Technische Universität Ilmenau, 98693, Ilmenau, Germany
| | - Sascha Klee
- Institute for Biomedical Engineering and Informatics, Technische Universität Ilmenau, 98693, Ilmenau, Germany.,Division Biostatistics and Data Science, Department of General Health Studies, Karl Landsteiner University of Health Sciences, Dr. Karl-Dorrek-Straße 30, 3500, Krems, Austria
| | - Muhammad Qurhanul Rizqie
- Institute for Biomedical Engineering and Informatics, Technische Universität Ilmenau, 98693, Ilmenau, Germany.,Informatics Engineering Program, Universitas Sriwijaya, Palembang, South Sumatera, Indonesia
| | - Eko Supriyanto
- Institute for Biomedical Engineering and Informatics, Technische Universität Ilmenau, 98693, Ilmenau, Germany.,IJN-UTM Cardiovascular Engineering Centre, Universiti Teknologi Malaysia, Johor Bahru, Johor, Malaysia
| | - Karsten König
- Department of Biophotonics and Laser Technology, Saarland University, Campus A5.1, 66123, Saarbrücken, Germany.,JenLab GmbH, Johann-Hittorf-Straße 8, 12489, Berlin, Germany
| | - Hans Georg Breunig
- Department of Biophotonics and Laser Technology, Saarland University, Campus A5.1, 66123, Saarbrücken, Germany.,JenLab GmbH, Johann-Hittorf-Straße 8, 12489, Berlin, Germany
| | - Łukasz Piątek
- Institute for Biomedical Engineering and Informatics, Technische Universität Ilmenau, 98693, Ilmenau, Germany.,Department of Artificial Intelligence, University of Information Technology and Management, H. Sucharskiego 2 Str, 35-225, Rzeszów, Poland
| |
Collapse
|
3
|
Datta R, Heaster TM, Sharick JT, Gillette AA, Skala MC. Fluorescence lifetime imaging microscopy: fundamentals and advances in instrumentation, analysis, and applications. JOURNAL OF BIOMEDICAL OPTICS 2020; 25:1-43. [PMID: 32406215 PMCID: PMC7219965 DOI: 10.1117/1.jbo.25.7.071203] [Citation(s) in RCA: 292] [Impact Index Per Article: 73.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/06/2020] [Accepted: 04/24/2020] [Indexed: 05/18/2023]
Abstract
SIGNIFICANCE Fluorescence lifetime imaging microscopy (FLIM) is a powerful technique to distinguish the unique molecular environment of fluorophores. FLIM measures the time a fluorophore remains in an excited state before emitting a photon, and detects molecular variations of fluorophores that are not apparent with spectral techniques alone. FLIM is sensitive to multiple biomedical processes including disease progression and drug efficacy. AIM We provide an overview of FLIM principles, instrumentation, and analysis while highlighting the latest developments and biological applications. APPROACH This review covers FLIM principles and theory, including advantages over intensity-based fluorescence measurements. Fundamentals of FLIM instrumentation in time- and frequency-domains are summarized, along with recent developments. Image segmentation and analysis strategies that quantify spatial and molecular features of cellular heterogeneity are reviewed. Finally, representative applications are provided including high-resolution FLIM of cell- and organelle-level molecular changes, use of exogenous and endogenous fluorophores, and imaging protein-protein interactions with Förster resonance energy transfer (FRET). Advantages and limitations of FLIM are also discussed. CONCLUSIONS FLIM is advantageous for probing molecular environments of fluorophores to inform on fluorophore behavior that cannot be elucidated with intensity measurements alone. Development of FLIM technologies, analysis, and applications will further advance biological research and clinical assessments.
Collapse
Affiliation(s)
- Rupsa Datta
- Morgridge Institute for Research, Madison, Wisconsin, United States
| | - Tiffany M. Heaster
- Morgridge Institute for Research, Madison, Wisconsin, United States
- University of Wisconsin, Department of Biomedical Engineering, Madison, Wisconsin, United States
| | - Joe T. Sharick
- Morgridge Institute for Research, Madison, Wisconsin, United States
| | - Amani A. Gillette
- Morgridge Institute for Research, Madison, Wisconsin, United States
- University of Wisconsin, Department of Biomedical Engineering, Madison, Wisconsin, United States
| | - Melissa C. Skala
- Morgridge Institute for Research, Madison, Wisconsin, United States
- University of Wisconsin, Department of Biomedical Engineering, Madison, Wisconsin, United States
| |
Collapse
|
4
|
Banerjee SJ, Azharuddin M, Sen D, Savale S, Datta H, Dasgupta AK, Roy S. Using complex networks towards information retrieval and diagnostics in multidimensional imaging. Sci Rep 2015; 5:17271. [PMID: 26626047 PMCID: PMC4667282 DOI: 10.1038/srep17271] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2015] [Accepted: 10/28/2015] [Indexed: 11/24/2022] Open
Abstract
We present a fresh and broad yet simple approach towards information retrieval in general and diagnostics in particular by applying the theory of complex networks on multidimensional, dynamic images. We demonstrate a successful use of our method with the time series generated from high content thermal imaging videos of patients suffering from the aqueous deficient dry eye (ADDE) disease. Remarkably, network analyses of thermal imaging time series of contact lens users and patients upon whom Laser-Assisted in situ Keratomileusis (Lasik) surgery has been conducted, exhibit pronounced similarity with results obtained from ADDE patients. We also propose a general framework for the transformation of multidimensional images to networks for futuristic biometry. Our approach is general and scalable to other fluctuation-based devices where network parameters derived from fluctuations, act as effective discriminators and diagnostic markers.
Collapse
Affiliation(s)
| | - Mohammad Azharuddin
- Department of Biochemistry, University of Calcutta, 35 Ballygunge Circular Road, Kolkata 700 019, India
| | - Debanjan Sen
- Regional Institute of Ophthalmology, Calcutta Medical College and Hospital, Kolkata 700 073, India
| | - Smruti Savale
- Regional Institute of Ophthalmology, Calcutta Medical College and Hospital, Kolkata 700 073, India
| | - Himadri Datta
- Regional Institute of Ophthalmology, Calcutta Medical College and Hospital, Kolkata 700 073, India
| | - Anjan Kr Dasgupta
- Department of Biochemistry, University of Calcutta, 35 Ballygunge Circular Road, Kolkata 700 019, India
| | - Soumen Roy
- Bose Institute, 93/1 Acharya PC Roy Road, Kolkata 700 009, India
| |
Collapse
|
5
|
Pécot T, Bouthemy P, Boulanger J, Chessel A, Bardin S, Salamero J, Kervrann C. Background fluorescence estimation and vesicle segmentation in live cell imaging with conditional random fields. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2015; 24:667-80. [PMID: 25531952 DOI: 10.1109/tip.2014.2380178] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/09/2023]
Abstract
Image analysis applied to fluorescence live cell microscopy has become a key tool in molecular biology since it enables to characterize biological processes in space and time at the subcellular level. In fluorescence microscopy imaging, the moving tagged structures of interest, such as vesicles, appear as bright spots over a static or nonstatic background. In this paper, we consider the problem of vesicle segmentation and time-varying background estimation at the cellular scale. The main idea is to formulate the joint segmentation-estimation problem in the general conditional random field framework. Furthermore, segmentation of vesicles and background estimation are alternatively performed by energy minimization using a min cut-max flow algorithm. The proposed approach relies on a detection measure computed from intensity contrasts between neighboring blocks in fluorescence microscopy images. This approach permits analysis of either 2D + time or 3D + time data. We demonstrate the performance of the so-called C-CRAFT through an experimental comparison with the state-of-the-art methods in fluorescence video-microscopy. We also use this method to characterize the spatial and temporal distribution of Rab6 transport carriers at the cell periphery for two different specific adhesion geometries.
Collapse
|