1
|
Wang K, Song L, Wang C, Ren Z, Zhao G, Dou J, Di J, Barbastathis G, Zhou R, Zhao J, Lam EY. On the use of deep learning for phase recovery. LIGHT, SCIENCE & APPLICATIONS 2024; 13:4. [PMID: 38161203 PMCID: PMC10758000 DOI: 10.1038/s41377-023-01340-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Revised: 11/13/2023] [Accepted: 11/16/2023] [Indexed: 01/03/2024]
Abstract
Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.
Collapse
Affiliation(s)
- Kaiqiang Wang
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China.
- School of Physical Science and Technology, Northwestern Polytechnical University, Xi'an, China.
- Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China.
| | - Li Song
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China
| | - Chutian Wang
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China
| | - Zhenbo Ren
- School of Physical Science and Technology, Northwestern Polytechnical University, Xi'an, China
| | - Guangyuan Zhao
- Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China
| | - Jiazhen Dou
- School of Information Engineering, Guangdong University of Technology, Guangzhou, China
| | - Jianglei Di
- School of Information Engineering, Guangdong University of Technology, Guangzhou, China
| | - George Barbastathis
- Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Renjie Zhou
- Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China
| | - Jianlin Zhao
- School of Physical Science and Technology, Northwestern Polytechnical University, Xi'an, China.
| | - Edmund Y Lam
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China.
| |
Collapse
|
2
|
Nguyen TL, Pradeep S, Judson-Torres RL, Reed J, Teitell MA, Zangle TA. Quantitative Phase Imaging: Recent Advances and Expanding Potential in Biomedicine. ACS NANO 2022; 16:11516-11544. [PMID: 35916417 PMCID: PMC10112851 DOI: 10.1021/acsnano.1c11507] [Citation(s) in RCA: 40] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/22/2023]
Abstract
Quantitative phase imaging (QPI) is a label-free, wide-field microscopy approach with significant opportunities for biomedical applications. QPI uses the natural phase shift of light as it passes through a transparent object, such as a mammalian cell, to quantify biomass distribution and spatial and temporal changes in biomass. Reported in cell studies more than 60 years ago, ongoing advances in QPI hardware and software are leading to numerous applications in biology, with a dramatic expansion in utility over the past two decades. Today, investigations of cell size, morphology, behavior, cellular viscoelasticity, drug efficacy, biomass accumulation and turnover, and transport mechanics are supporting studies of development, physiology, neural activity, cancer, and additional physiological processes and diseases. Here, we review the field of QPI in biology starting with underlying principles, followed by a discussion of technical approaches currently available or being developed, and end with an examination of the breadth of applications in use or under development. We comment on strengths and shortcomings for the deployment of QPI in key biomedical contexts and conclude with emerging challenges and opportunities based on combining QPI with other methodologies that expand the scope and utility of QPI even further.
Collapse
|
3
|
Yang X, Zhao Q, Huang T, Hu Z, Bu T, He H, Hou A, Li M, Xiao Y, Ma H. Deep learning for denoising in a Mueller matrix microscope. BIOMEDICAL OPTICS EXPRESS 2022; 13:3535-3551. [PMID: 35781954 PMCID: PMC9208591 DOI: 10.1364/boe.457219] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Revised: 05/05/2022] [Accepted: 05/17/2022] [Indexed: 06/15/2023]
Abstract
The Mueller matrix microscope is a powerful tool for characterizing the microstructural features of a complex biological sample. Performance of a Mueller matrix microscope usually relies on two major specifications: measurement accuracy and acquisition time, which may conflict with each other but both contribute to the complexity and expenses of the apparatus. In this paper, we report a learning-based method to improve both specifications of a Mueller matrix microscope using a rotating polarizer and a rotating waveplate polarization state generator. Low noise data from long acquisition time are used as the ground truth. A modified U-Net structured network incorporating channel attention effectively reduces the noise in lower quality Mueller matrix images obtained with much shorter acquisition time. The experimental results show that using high quality Mueller matrix data as ground truth, such a learning-based method can achieve both high measurement accuracy and short acquisition time in polarization imaging.
Collapse
Affiliation(s)
- Xiongjie Yang
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
- Contributed equally
| | - Qianhao Zhao
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
- Contributed equally
| | - Tongyu Huang
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
- Department of Biomedical Engineering, Tsinghua University, Beijing 100084, China
| | - Zheng Hu
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
| | - Tongjun Bu
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
| | - Honghui He
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
| | - Anli Hou
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
- Department of Gynaecology, University of Chinese Academy of Sciences Shenzhen Hospital, Shenzhen 518106, China
| | - Migao Li
- Guangdong Liss Optical Instrument Co., Ltd., Guangzhou 510095, China
| | - Yucheng Xiao
- Beijing Normal University - Hong Kong Baptist University United International College, Zhuhai 519085, China
| | - Hui Ma
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
- Department of Biomedical Engineering, Tsinghua University, Beijing 100084, China
- Department of Physics, Tsinghua University, Beijing 100084, China
| |
Collapse
|
5
|
Automatic Colorectal Cancer Screening Using Deep Learning in Spatial Light Interference Microscopy Data. Cells 2022; 11:cells11040716. [PMID: 35203365 PMCID: PMC8870406 DOI: 10.3390/cells11040716] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Revised: 02/11/2022] [Accepted: 02/14/2022] [Indexed: 11/24/2022] Open
Abstract
The surgical pathology workflow currently adopted by clinics uses staining to reveal tissue architecture within thin sections. A trained pathologist then conducts a visual examination of these slices and, since the investigation is based on an empirical assessment, a certain amount of subjectivity is unavoidable. Furthermore, the reliance on external contrast agents such as hematoxylin and eosin (H&E), albeit being well-established methods, makes it difficult to standardize color balance, staining strength, and imaging conditions, hindering automated computational analysis. In response to these challenges, we applied spatial light interference microscopy (SLIM), a label-free method that generates contrast based on intrinsic tissue refractive index signatures. Thus, we reduce human bias and make imaging data comparable across instruments and clinics. We applied a mask R-CNN deep learning algorithm to the SLIM data to achieve an automated colorectal cancer screening procedure, i.e., classifying normal vs. cancerous specimens. Our results, obtained on a tissue microarray consisting of specimens from 132 patients, resulted in 91% accuracy for gland detection, 99.71% accuracy in gland-level classification, and 97% accuracy in core-level classification. A SLIM tissue scanner accompanied by an application-specific deep learning algorithm may become a valuable clinical tool, enabling faster and more accurate assessments by pathologists.
Collapse
|
6
|
Mudeng V, Kim M, Choe SW. Objective Numerical Evaluation of Diffuse, Optically Reconstructed Images Using Structural Similarity Index. BIOSENSORS 2021; 11:504. [PMID: 34940261 PMCID: PMC8699273 DOI: 10.3390/bios11120504] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/31/2021] [Revised: 12/06/2021] [Accepted: 12/06/2021] [Indexed: 05/13/2023]
Abstract
Diffuse optical tomography is emerging as a non-invasive optical modality used to evaluate tissue information by obtaining the optical properties' distribution. Two procedures are performed to produce reconstructed absorption and reduced scattering images, which provide structural information that can be used to locate inclusions within tissues with the assistance of a known light intensity around the boundary. These methods are referred to as a forward problem and an inverse solution. Once the reconstructed image is obtained, a subjective measurement is used as the conventional way to assess the image. Hence, in this study, we developed an algorithm designed to numerically assess reconstructed images to identify inclusions using the structural similarity (SSIM) index. We compared four SSIM algorithms with 168 simulated reconstructed images involving the same inclusion position with different contrast ratios and inclusion sizes. A multiscale, improved SSIM containing a sharpness parameter (MS-ISSIM-S) was proposed to represent the potential evaluation compared with the human visible perception. The results indicated that the proposed MS-ISSIM-S is suitable for human visual perception by demonstrating a reduction of similarity score related to various contrasts with a similar size of inclusion; thus, this metric is promising for the objective numerical assessment of diffuse, optically reconstructed images.
Collapse
Affiliation(s)
- Vicky Mudeng
- Department of Medical IT Convergence Engineering, Kumoh National Institute of Technology, Gumi 39253, Korea;
- Department of Electrical Engineering, Institut Teknologi Kalimantan, Balikpapan 76127, Indonesia
| | - Minseok Kim
- Department of Mechanical System Engineering, Kumoh National Institute of Technology, Gumi 39177, Korea
- Department of Aeronautics, Mechanical and Electronic Convergence Engineering, Kumoh National Institute of Technology, Gumi 39177, Korea
| | - Se-woon Choe
- Department of Medical IT Convergence Engineering, Kumoh National Institute of Technology, Gumi 39253, Korea;
- Department of IT Convergence Engineering, Kumoh National Institute of Technology, Gumi 39253, Korea
| |
Collapse
|
7
|
Li J, Garfinkel J, Zhang X, Wu D, Zhang Y, de Haan K, Wang H, Liu T, Bai B, Rivenson Y, Rubinstein G, Scumpia PO, Ozcan A. Biopsy-free in vivo virtual histology of skin using deep learning. LIGHT, SCIENCE & APPLICATIONS 2021; 10:233. [PMID: 34795202 PMCID: PMC8602311 DOI: 10.1038/s41377-021-00674-8] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Revised: 10/22/2021] [Accepted: 10/28/2021] [Indexed: 05/09/2023]
Abstract
An invasive biopsy followed by histological staining is the benchmark for pathological diagnosis of skin tumors. The process is cumbersome and time-consuming, often leading to unnecessary biopsies and scars. Emerging noninvasive optical technologies such as reflectance confocal microscopy (RCM) can provide label-free, cellular-level resolution, in vivo images of skin without performing a biopsy. Although RCM is a useful diagnostic tool, it requires specialized training because the acquired images are grayscale, lack nuclear features, and are difficult to correlate with tissue pathology. Here, we present a deep learning-based framework that uses a convolutional neural network to rapidly transform in vivo RCM images of unstained skin into virtually-stained hematoxylin and eosin-like images with microscopic resolution, enabling visualization of the epidermis, dermal-epidermal junction, and superficial dermis layers. The network was trained under an adversarial learning scheme, which takes ex vivo RCM images of excised unstained/label-free tissue as inputs and uses the microscopic images of the same tissue labeled with acetic acid nuclear contrast staining as the ground truth. We show that this trained neural network can be used to rapidly perform virtual histology of in vivo, label-free RCM images of normal skin structure, basal cell carcinoma, and melanocytic nevi with pigmented melanocytes, demonstrating similar histological features to traditional histology from the same excised tissue. This application of deep learning-based virtual staining to noninvasive imaging technologies may permit more rapid diagnoses of malignant skin neoplasms and reduce invasive skin biopsies.
Collapse
Affiliation(s)
- Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | | | - Xiaoran Zhang
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
| | - Di Wu
- Computer Science Department, University of California, Los Angeles, CA, 90095, USA
| | - Yijie Zhang
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Kevin de Haan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Hongda Wang
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Tairan Liu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Bijie Bai
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Yair Rivenson
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | | | - Philip O Scumpia
- Division of Dermatology, University of California, Los Angeles, CA, 90095, USA.
- Department of Dermatology, Veterans Affairs Greater Los Angeles Healthcare System, Los Angeles, CA, 90073, USA.
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
- Department of Surgery, University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|