1
|
Imani H, Islam MB, Junayed MS, Ahad MAR. Stereoscopic video deblurring transformer. Sci Rep 2024; 14:14342. [PMID: 38906905 PMCID: PMC11192757 DOI: 10.1038/s41598-024-63860-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Accepted: 06/03/2024] [Indexed: 06/23/2024] Open
Abstract
Stereoscopic cameras, such as those in mobile phones and various recent intelligent systems, are becoming increasingly common. Multiple variables can impact the stereo video quality, e.g., blur distortion due to camera/object movement. Monocular image/video deblurring is a mature research field, while there is limited research on stereoscopic content deblurring. This paper introduces a new Transformer-based stereo video deblurring framework with two crucial new parts: a self-attention layer and a feed-forward layer that realizes and aligns the correlation among various video frames. The traditional fully connected (FC) self-attention layer fails to utilize data locality effectively, as it depends on linear layers for calculating attention maps The Vision Transformer, on the other hand, also has this limitation, as it takes image patches as inputs to model global spatial information. 3D convolutional neural networks (3D CNNs) process successive frames to correct motion blur in the stereo video. Besides, our method uses other stereo-viewpoint information to assist deblurring. The parallax attention module (PAM) is significantly improved to combine the stereo and cross-view information for more deblurring. An extensive ablation study validates that our method efficiently deblurs the stereo videos based on the experiments on two publicly available stereo video datasets. Experimental results of our approach demonstrate state-of-the-art performance compared to the image and video deblurring techniques by a large margin.
Collapse
Affiliation(s)
- Hassan Imani
- Faculty of Engineering and Natural Sciences, Bahcesehir University, 34353, Istanbul, Turkey
| | - Md Baharul Islam
- Faculty of Engineering and Natural Sciences, Bahcesehir University, 34353, Istanbul, Turkey.
- Department of Computing and Software Engineering, Florida Gulf Coast University, Fort Myers, FL, 33965, USA.
| | - Masum Shah Junayed
- Faculty of Engineering and Natural Sciences, Bahcesehir University, 34353, Istanbul, Turkey
- Department of Computer Science and Engineering, University of Connecticut, Storrs, CT, 06269, USA
| | - Md Atiqur Rahman Ahad
- Department of Computer Science and Digital Technologies, University of East London, London, UK.
| |
Collapse
|
2
|
Li R, Ma J, Li D, Wu Y, Qian C, Zhang L, Chen H, Kottos T, Li EP. Non-Invasive Self-Adaptive Information States' Acquisition inside Dynamic Scattering Spaces. RESEARCH (WASHINGTON, D.C.) 2024; 7:0375. [PMID: 38826565 PMCID: PMC11140760 DOI: 10.34133/research.0375] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/03/2024] [Accepted: 04/12/2024] [Indexed: 06/04/2024]
Abstract
Pushing the information states' acquisition efficiency has been a long-held goal to reach the measurement precision limit inside scattering spaces. Recent studies have indicated that maximal information states can be attained through engineered modes; however, partial intrusion is generally required. While non-invasive designs have been substantially explored across diverse physical scenarios, the non-invasive acquisition of information states inside dynamic scattering spaces remains challenging due to the intractable non-unique mapping problem, particularly in the context of multi-target scenarios. Here, we establish the feasibility of non-invasive information states' acquisition experimentally for the first time by introducing a tandem-generated adversarial network framework inside dynamic scattering spaces. To illustrate the framework's efficacy, we demonstrate that efficient information states' acquisition for multi-target scenarios can achieve the Fisher information limit solely through the utilization of the external scattering matrix of the system. Our work provides insightful perspectives for precise measurements inside dynamic complex systems.
Collapse
Affiliation(s)
- Ruifeng Li
- Zhejiang University–University of Illinois at Urbana-Champaign Institute,
Zhejiang University, Haining 314400, China
- College of Information Science and Electronic Engineering,
Zhejiang University, Hangzhou 310027, China
| | - Jinyan Ma
- Zhejiang University–University of Illinois at Urbana-Champaign Institute,
Zhejiang University, Haining 314400, China
- College of Information Science and Electronic Engineering,
Zhejiang University, Hangzhou 310027, China
| | - Da Li
- Zhejiang University–University of Illinois at Urbana-Champaign Institute,
Zhejiang University, Haining 314400, China
- College of Information Science and Electronic Engineering,
Zhejiang University, Hangzhou 310027, China
| | - Yunlong Wu
- Zhejiang University–University of Illinois at Urbana-Champaign Institute,
Zhejiang University, Haining 314400, China
- College of Information Science and Electronic Engineering,
Zhejiang University, Hangzhou 310027, China
| | - Chao Qian
- Zhejiang University–University of Illinois at Urbana-Champaign Institute,
Zhejiang University, Haining 314400, China
- College of Information Science and Electronic Engineering,
Zhejiang University, Hangzhou 310027, China
| | - Ling Zhang
- Zhejiang University–University of Illinois at Urbana-Champaign Institute,
Zhejiang University, Haining 314400, China
- College of Information Science and Electronic Engineering,
Zhejiang University, Hangzhou 310027, China
| | - Hongsheng Chen
- Zhejiang University–University of Illinois at Urbana-Champaign Institute,
Zhejiang University, Haining 314400, China
- College of Information Science and Electronic Engineering,
Zhejiang University, Hangzhou 310027, China
| | - Tsampikos Kottos
- Wave Transport in Complex Systems Lab, Department of Physics,
Wesleyan University, Middletown, CT 06459, USA
| | - Er-Ping Li
- Zhejiang University–University of Illinois at Urbana-Champaign Institute,
Zhejiang University, Haining 314400, China
- College of Information Science and Electronic Engineering,
Zhejiang University, Hangzhou 310027, China
| |
Collapse
|
3
|
Gao X, Huang T, Tang P, Di J, Zhong L, Zhang W. Enhancing scanning electron microscopy imaging quality of weakly conductive samples through unsupervised learning. Sci Rep 2024; 14:6439. [PMID: 38499623 PMCID: PMC10948821 DOI: 10.1038/s41598-024-57056-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2024] [Accepted: 03/13/2024] [Indexed: 03/20/2024] Open
Abstract
Scanning electron microscopy (SEM) is a crucial tool for analyzing submicron-scale structures. However, the attainment of high-quality SEM images is contingent upon the high conductivity of the material due to constraints imposed by its imaging principles. For weakly conductive materials or structures induced by intrinsic properties or organic doping, the SEM imaging quality is significantly compromised, thereby impeding the accuracy of subsequent structure-related analyses. Moreover, the unavailability of paired high-low quality images in this context renders the supervised-based image processing methods ineffective in addressing this challenge. Here, an unsupervised method based on Cycle-consistent Generative Adversarial Network (CycleGAN) was proposed to enhance the quality of SEM images for weakly conductive samples. The unsupervised model can perform end-to-end learning using unpaired blurred and clear SEM images from weakly and well-conductive samples, respectively. To address the requirements of material structure analysis, an edge loss function was further introduced to recover finer details in the network-generated images. Various quantitative evaluations substantiate the efficacy of the proposed method in SEM image quality improvement with better performance than the traditional methods. Our framework broadens the application of artificial intelligence in materials analysis, holding significant implications in fields such as materials science and image restoration.
Collapse
Affiliation(s)
- Xin Gao
- Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education, Guangdong University of Technology, Guangzhou, 510006, China
| | - Tao Huang
- Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education, Guangdong University of Technology, Guangzhou, 510006, China
| | - Ping Tang
- Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education, Guangdong University of Technology, Guangzhou, 510006, China
| | - Jianglei Di
- Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education, Guangdong University of Technology, Guangzhou, 510006, China
| | - Liyun Zhong
- Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education, Guangdong University of Technology, Guangzhou, 510006, China
| | - Weina Zhang
- Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education, Guangdong University of Technology, Guangzhou, 510006, China.
| |
Collapse
|
4
|
Tadesse K, Mandracchia B, Yoon K, Han K, Jia S. Three-dimensional multifocal scanning microscopy for super-resolution cell and tissue imaging. OPTICS EXPRESS 2023; 31:38550-38559. [PMID: 38017958 DOI: 10.1364/oe.501100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Accepted: 10/24/2023] [Indexed: 11/30/2023]
Abstract
Recent advancements in image-scanning microscopy have significantly enriched super-resolution biological research, providing deeper insights into cellular structures and processes. However, current image-scanning techniques often require complex instrumentation and alignment, constraining their broader applicability in cell biological discovery and convenient, cost-effective integration into commonly used frameworks like epi-fluorescence microscopes. Here, we introduce three-dimensional multifocal scanning microscopy (3D-MSM) for super-resolution imaging of cells and tissue with substantially reduced instrumental complexity. This method harnesses the inherent 3D movement of specimens to achieve stationary, multi-focal excitation and super-resolution microscopy through a standard epi-fluorescence platform. We validated the system using a range of phantom, single-cell, and tissue specimens. The combined strengths of structured illumination, confocal detection, and epi-fluorescence setup result in two-fold resolution improvement in all three dimensions, effective optical sectioning, scalable volume acquisition, and compatibility with general imaging and sample protocols. We anticipate that 3D-MSM will pave a promising path for future super-resolution investigations in cell and tissue biology.
Collapse
|
5
|
Fanous MJ, Pillar N, Ozcan A. Digital staining facilitates biomedical microscopy. FRONTIERS IN BIOINFORMATICS 2023; 3:1243663. [PMID: 37564725 PMCID: PMC10411189 DOI: 10.3389/fbinf.2023.1243663] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Accepted: 07/17/2023] [Indexed: 08/12/2023] Open
Abstract
Traditional staining of biological specimens for microscopic imaging entails time-consuming, laborious, and costly procedures, in addition to producing inconsistent labeling and causing irreversible sample damage. In recent years, computational "virtual" staining using deep learning techniques has evolved into a robust and comprehensive application for streamlining the staining process without typical histochemical staining-related drawbacks. Such virtual staining techniques can also be combined with neural networks designed to correct various microscopy aberrations, such as out-of-focus or motion blur artifacts, and improve upon diffracted-limited resolution. Here, we highlight how such methods lead to a host of new opportunities that can significantly improve both sample preparation and imaging in biomedical microscopy.
Collapse
Affiliation(s)
- Michael John Fanous
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, United States
| | - Nir Pillar
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, United States
- Bioengineering Department, University of California, Los Angeles, CA, United States
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, United States
- Bioengineering Department, University of California, Los Angeles, CA, United States
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, United States
- Department of Surgery, David Geffen School of Medicine, University of California, Los Angeles, CA, United States
| |
Collapse
|
6
|
Wang Y, Lei J, Zheng J, Wang X, Cheng M, Liu M, Zhang J, Chen W, Hu X, Gu W, Guo S, Hu X, Gao Z, Liu D. Chromatic aberration correction based on cross-channel information alignment in microscopy. APPLIED OPTICS 2023; 62:3289-3298. [PMID: 37132829 DOI: 10.1364/ao.482013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/04/2023]
Abstract
A microscope usually consists of dozens of complex lenses and requires careful assembly, alignment, and testing before use. Chromatic aberration correction is a significant step in the design of microscopes. Reducing chromatic aberration by improving optical design will inevitably increase the overall weight and size of the microscope, leading to more cost in manufacturing and maintenance. Nevertheless, the improvement in hardware can only achieve limited correction. In this paper, we propose an algorithm based on cross-channel information alignment to shift some of the correction tasks from optical design to post-processing. Additionally, a quantitative framework is established to evaluate the performance of the chromatic aberration algorithm. Our algorithm outperforms the other state-of-the-art methods in both visual appearance and objective assessments. The results indicate that the proposed algorithm can effectively obtain higher-quality images without changing the hardware or engaging the optical parameters.
Collapse
|
7
|
Bai B, Yang X, Li Y, Zhang Y, Pillar N, Ozcan A. Deep learning-enabled virtual histological staining of biological samples. LIGHT, SCIENCE & APPLICATIONS 2023; 12:57. [PMID: 36864032 PMCID: PMC9981740 DOI: 10.1038/s41377-023-01104-7] [Citation(s) in RCA: 33] [Impact Index Per Article: 33.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/13/2022] [Revised: 02/10/2023] [Accepted: 02/14/2023] [Indexed: 06/18/2023]
Abstract
Histological staining is the gold standard for tissue examination in clinical pathology and life-science research, which visualizes the tissue and cellular structures using chromatic dyes or fluorescence labels to aid the microscopic assessment of tissue. However, the current histological staining workflow requires tedious sample preparation steps, specialized laboratory infrastructure, and trained histotechnologists, making it expensive, time-consuming, and not accessible in resource-limited settings. Deep learning techniques created new opportunities to revolutionize staining methods by digitally generating histological stains using trained neural networks, providing rapid, cost-effective, and accurate alternatives to standard chemical staining methods. These techniques, broadly referred to as virtual staining, were extensively explored by multiple research groups and demonstrated to be successful in generating various types of histological stains from label-free microscopic images of unstained samples; similar approaches were also used for transforming images of an already stained tissue sample into another type of stain, performing virtual stain-to-stain transformations. In this Review, we provide a comprehensive overview of the recent research advances in deep learning-enabled virtual histological staining techniques. The basic concepts and the typical workflow of virtual staining are introduced, followed by a discussion of representative works and their technical innovations. We also share our perspectives on the future of this emerging field, aiming to inspire readers from diverse scientific fields to further expand the scope of deep learning-enabled virtual histological staining techniques and their applications.
Collapse
Affiliation(s)
- Bijie Bai
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Xilin Yang
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Yuzhu Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Yijie Zhang
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Nir Pillar
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA.
| |
Collapse
|
8
|
Rivenson Y, Ozcan A. Deep learning accelerates whole slide imaging for next-generation digital pathology applications. LIGHT, SCIENCE & APPLICATIONS 2022; 11:300. [PMID: 36241615 PMCID: PMC9568604 DOI: 10.1038/s41377-022-00999-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Deep learning demonstrates the ability to significantly increase the scanning speed of whole slide imaging in histology. This transformative solution can be used to further accelerate the adoption of digital pathology.
Collapse
Affiliation(s)
- Yair Rivenson
- Pictor Labs, Inc., Los Angeles, USA.
- Electrical and Computer Engineering Department, University of California, Los Angeles, 90095, CA, USA.
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, 90095, CA, USA
- Bioengineering Department, University of California, Los Angeles, 90095, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, 90095, CA, USA
- Department of Surgery, David Geffen School of Medicine, University of California, Los Angeles, 90095, CA, USA
| |
Collapse
|