Neural network-based image reconstruction in swept-source optical coherence tomography using undersampled spectral data

Neural network-based processing and reconstruction of compromised biophotonic image data

In recent years, the integration of deep learning techniques with biophotonic setups has opened new horizons in bioimaging. A compelling trend in this field involves deliberately compromising certain measurement metrics to engineer better bioimaging tools in terms of e.g., cost, speed, and form-factor, followed by compensating for the resulting defects through the utilization of deep learning models trained on a large amount of ideal, superior or alternative data. This strategic approach has found increasing popularity due to its potential to enhance various aspects of biophotonic imaging. One of the primary motivations for employing this strategy is the pursuit of higher temporal resolution or increased imaging speed, critical for capturing fine dynamic biological processes. Additionally, this approach offers the prospect of simplifying hardware requirements and complexities, thereby making advanced imaging standards more accessible in terms of cost and/or size. This article provides an in-depth review of the diverse measurement aspects that researchers intentionally impair in their biophotonic setups, including the point spread function (PSF), signal-to-noise ratio (SNR), sampling density, and pixel resolution. By deliberately compromising these metrics, researchers aim to not only recuperate them through the application of deep learning networks, but also bolster in return other crucial parameters, such as the field of view (FOV), depth of field (DOF), and space-bandwidth product (SBP). Throughout this article, we discuss various biophotonic methods that have successfully employed this strategic approach. These techniques span a wide range of applications and showcase the versatility and effectiveness of deep learning in the context of compromised biophotonic data. Finally, by offering our perspectives on the exciting future possibilities of this rapidly evolving concept, we hope to motivate our readers from various disciplines to explore novel ways of balancing hardware compromises with compensation via artificial intelligence (AI).

Photon-level single-pixel 3D tomography with masked attention network

Tomography plays an important role in characterizing the three-dimensional structure of samples within specialized scenarios. In the paper, a masked attention network is presented to eliminate interference from different layers of the sample, substantially enhancing the resolution for photon-level single-pixel tomographic imaging. The simulation and experimental results have demonstrated that the axial resolution and lateral resolution of the imaging system can be improved by about 3 and 2 times respectively, with a sampling rate of 3.0 %. The scheme is expected to be seamlessly integrated into various tomography systems, which is conducive to promoting the tomographic imaging for biology, medicine, and materials science.

Machine Learning-Assisted Short-Wave InfraRed (SWIR) Techniques for Biomedical Applications: Towards Personalized Medicine

Personalized medicine transforms healthcare by adapting interventions to individuals' unique genetic, molecular, and clinical profiles. To maximize diagnostic and/or therapeutic efficacy, personalized medicine requires advanced imaging devices and sensors for accurate assessment and monitoring of individual patient conditions or responses to therapeutics. In the field of biomedical optics, short-wave infrared (SWIR) techniques offer an array of capabilities that hold promise to significantly enhance diagnostics, imaging, and therapeutic interventions. SWIR techniques provide in vivo information, which was previously inaccessible, by making use of its capacity to penetrate biological tissues with reduced attenuation and enable researchers and clinicians to delve deeper into anatomical structures, physiological processes, and molecular interactions. Combining SWIR techniques with machine learning (ML), which is a powerful tool for analyzing information, holds the potential to provide unprecedented accuracy for disease detection, precision in treatment guidance, and correlations of complex biological features, opening the way for the data-driven personalized medicine field. Despite numerous biomedical demonstrations that utilize cutting-edge SWIR techniques, the clinical potential of this approach has remained significantly underexplored. This paper demonstrates how the synergy between SWIR imaging and ML is reshaping biomedical research and clinical applications. As the paper showcases the growing significance of SWIR imaging techniques that are empowered by ML, it calls for continued collaboration between researchers, engineers, and clinicians to boost the translation of this technology into clinics, ultimately bridging the gap between cutting-edge technology and its potential for personalized medicine.

Frequency-aware optical coherence tomography image super-resolution via conditional generative adversarial neural network

Optical coherence tomography (OCT) has stimulated a wide range of medical image-based diagnosis and treatment in fields such as cardiology and ophthalmology. Such applications can be further facilitated by deep learning-based super-resolution technology, which improves the capability of resolving morphological structures. However, existing deep learning-based method only focuses on spatial distribution and disregards frequency fidelity in image reconstruction, leading to a frequency bias. To overcome this limitation, we propose a frequency-aware super-resolution framework that integrates three critical frequency-based modules (i.e., frequency transformation, frequency skip connection, and frequency alignment) and frequency-based loss function into a conditional generative adversarial network (cGAN). We conducted a large-scale quantitative study from an existing coronary OCT dataset to demonstrate the superiority of our proposed framework over existing deep learning frameworks. In addition, we confirmed the generalizability of our framework by applying it to fish corneal images and rat retinal images, demonstrating its capability to super-resolve morphological details in eye imaging.

Deep learning-based image enhancement in optical coherence tomography by exploiting interference fringe

Optical coherence tomography (OCT), an interferometric imaging technique, provides non-invasive, high-speed, high-sensitive volumetric biological imaging in vivo. However, systemic features inherent in the basic operating principle of OCT limit its imaging performance such as spatial resolution and signal-to-noise ratio. Here, we propose a deep learning-based OCT image enhancement framework that exploits raw interference fringes to achieve further enhancement from currently obtainable optimized images. The proposed framework for enhancing spatial resolution and reducing speckle noise in OCT images consists of two separate models: an A-scan-based network (NetA) and a B-scan-based network (NetB). NetA utilizes spectrograms obtained via short-time Fourier transform of raw interference fringes to enhance axial resolution of A-scans. NetB was introduced to enhance lateral resolution and reduce speckle noise in B-scan images. The individually trained networks were applied sequentially. We demonstrate the versatility and capability of the proposed framework by visually and quantitatively validating its robust performance. Comparative studies suggest that deep learning utilizing interference fringes can outperform the existing methods. Furthermore, we demonstrate the advantages of the proposed method by comparing our outcomes with multi-B-scan averaged images and contrast-adjusted images. We expect that the proposed framework will be a versatile technology that can improve functionality of OCT.

SiamHAS: Siamese Tracker with Hierarchical Attention Strategy for Aerial Tracking

For the Siamese network-based trackers utilizing modern deep feature extraction networks without taking full advantage of the different levels of features, tracking drift is prone to occur in aerial scenarios, such as target occlusion, scale variation, and low-resolution target tracking. Additionally, the accuracy is low in challenging scenarios of visual tracking, which is due to the imperfect utilization of features. To improve the performance of the existing Siamese tracker in the above-mentioned challenging scenes, we propose a Siamese tracker based on Transformer multi-level feature enhancement with a hierarchical attention strategy. The saliency of the extracted features is enhanced by the process of Transformer Multi-level Enhancement; the application of the hierarchical attention strategy makes the tracker adaptively notice the target region information and improve the tracking performance in challenging aerial scenarios. Meanwhile, we conducted extensive experiments and qualitative or quantitative discussions on UVA123, UAV20L, and OTB100 datasets. Finally, the experimental results show that our SiamHAS performs favorably against several state-of-the-art trackers in these challenging scenarios.

Deep learning empowered highly compressive SS-OCT via learnable spectral-spatial sub-sampling

With the rapid advances of light source technology, the A-line imaging rate of swept-source optical coherence tomography (SS-OCT) has experienced a great increase in the past three decades. The bandwidths of data acquisition, data transfer, and data storage, which can easily reach several hundred megabytes per second, have now been considered major bottlenecks for modern SS-OCT system design. To address these issues, various compression schemes have been previously proposed. However, most of the current methods focus on enhancing the capability of the reconstruction algorithm and can only provide a data compression ratio (DCR) up to 4 without impairing the image quality. In this Letter, we proposed a novel design paradigm, in which the sub-sampling pattern for interferogram acquisition is jointly optimized with the reconstruction algorithm in an end-to-end manner. To validate the idea, we retrospectively apply the proposed method on an ex vivo human coronary optical coherence tomography (OCT) dataset. The proposed method could reach a maximum DCR of ∼62.5 with peak signal-to-noise ratio (PSNR) of 24.2 dB, while a DCR of ∼27.78 could yield a visually pleasant image with a PSNR of ∼24.6 dB. We believe the proposed system could be a viable remedy for the ever-growing data issue in SS-OCT.

Multi-Scale Reconstruction of Undersampled Spectral-Spatial OCT Data for Coronary Imaging Using Deep Learning

Coronary artery disease (CAD) is a cardiovascular condition with high morbidity and mortality. Intravascular optical coherence tomography (IVOCT) has been considered as an optimal imagining system for the diagnosis and treatment of CAD. Constrained by Nyquist theorem, dense sampling in IVOCT attains high resolving power to delineate cellular structures/features. There is a trade-off between high spatial resolution and fast scanning rate for coronary imaging. In this paper, we propose a viable spectral-spatial acquisition method that down-scales the sampling process in both spectral and spatial domain while maintaining high quality in image reconstruction. The down-scaling schedule boosts data acquisition speed without any hardware modifications. Additionally, we propose a unified multi-scale reconstruction framework, namely Multiscale-Spectral-Spatial-Magnification Network (MSSMN), to resolve highly down-scaled (compressed) OCT images with flexible magnification factors. We incorporate the proposed methods into Spectral Domain OCT (SD-OCT) imaging of human coronary samples with clinical features such as stent and calcified lesions. Our experimental results demonstrate that spectral-spatial down-scaled data can be better reconstructed than data that are down-scaled solely in either spectral or spatial domain. Moreover, we observe better reconstruction performance using MSSMN than using existing reconstruction methods. Our acquisition method and multi-scale reconstruction framework, in combination, may allow faster SD-OCT inspection with high resolution during coronary intervention.

Noise Reduction of OCT Images Based on the External Patch Prior Guided Internal Clustering and Morphological Analysis

Optical coherence tomography (OCT) is widely used in biomedical imaging. However, noise severely affects diagnosing and identifying diseased tissues on OCT images. Here, a noise reduction method based on the external patch prior guided internal clustering and morphological analysis (E2PGICMA) is developed to remove the noise of OCT images. The external patch prior guided internal clustering algorithm is used to reduce speckle noise. The morphological analysis algorithm is employed to the background for contrast enhancement. OCT images of in vivo normal skin tissues were analyzed to remove noise using the proposed method. The estimated standard deviations of the noise were chosen as different values for evaluating the quantitative metrics. The visual quality improvement includes more textures and fine detail preservation. The denoising effects of different methods were compared. Then, quantitative and qualitative evaluations of this proposed method were conducted. The results demonstrated that the SNR, PSNR, and XCOR were higher than those of the other noise-reduction methods, reaching 15.05 dB, 27.48 dB, and 0.9959, respectively. Furthermore, the presented method’s noise reduction ratio (NRR) reached 0.8999. This proposed method can efficiently remove the background and speckle noise. Improving the proposed noise reduction method would outperform existing state-of-the-art OCT despeckling methods.

cGAN-assisted imaging through stationary scattering media

Analyzing images taken through scattering media is challenging, owing to speckle decorrelations from perturbations in the media. For in-line imaging modalities, which are appealing because they are compact, require no moving parts, and are robust, negating the effects of such scattering becomes particularly challenging. Here we explore the use of conditional generative adversarial networks (cGANs) to mitigate the effects of the additional scatterers in in-line geometries, including digital holographic microscopy. Using light scattering simulations and experiments on objects of interest with and without additional scatterers, we find that cGANs can be quickly trained with minuscule datasets and can also efficiently learn the one-to-one statistical mapping between the cross-domain input-output image pairs. Importantly, the output images are faithful enough to enable quantitative feature extraction. We also show that with rapid training using only 20 image pairs, it is possible to negate this undesired scattering to accurately localize diffraction-limited impulses with high spatial accuracy, therefore transforming a shift variant system to a linear shift invariant (LSI) system.

Rapid Vehicle Detection in Aerial Images under the Complex Background of Dense Urban Areas

Vehicle detection on aerial remote sensing images under the complex background of urban areas has always received great attention in the field of remote sensing; however, the view of remote sensing images usually covers a large area, and the size of the vehicle is small and the background is complex. Therefore, compared with object detection in the ground view images, vehicle detection in aerial images remains a challenging problem. In this paper, we propose a single-scale rapid convolutional neural network (SSRD-Net). In the proposed framework, we design a global relational (GR) block to enhance the fusion of local and global features; moreover, we adjust the image segmentation method to unify the vehicle size in the input image, thus simplifying the model structure and improving the detection speed. We further introduce an aerial remote sensing image dataset with rotating bounding boxes (RO-ARS), which has complex backgrounds such as snow, clouds, and fog scenes. We also design a data augmentation method to get more images with clouds and fog. Finally, we evaluate the performance of the proposed model on several datasets, and the experimental results show that the recall and precision are improved compared with existing methods.

Light People: Professor Aydogan Ozcan

In 2016, the news that Google's artificial intelligence (AI) robot AlphaGo, based on the principle of deep learning, won the victory over lee Sedol, the former world Go champion and the famous 9th Dan competitor of Korea, caused a sensation in both fields of AI and Go, which brought epoch-making significance to the development of deep learning. Deep learning is a complex machine learning algorithm that uses multiple layers of artificial neural networks to automatically analyze signals or data. At present, deep learning has penetrated into our daily life, such as the applications of face recognition and speech recognition. Scientists have also made many remarkable achievements based on deep learning. Professor Aydogan Ozcan from the University of California, Los Angeles (UCLA) led his team to research deep learning algorithms, which provided new ideas for the exploring of optical computational imaging and sensing technology, and introduced image generation and reconstruction methods which brought major technological innovations to the development of related fields. Optical designs and devices are moving from being physically driven to being data-driven. We are much honored to have Aydogan Ozcan, Fellow of the National Academy of Inventors and Chancellor's Professor of UCLA, to unscramble his latest scientific research results and foresight for the future development of related fields, and to share his journey of pursuing Optics, his indissoluble relationship with Light: Science & Applications (LSA), and his experience in talent cultivation.