Pixel-wise regression using U-Net and its application on pansharpening

AB-PLS-DA: Pansharpening tailored for scanning electron microscopy and energy-dispersive X-ray spectrometry multimodal fusion

Pansharpening constitutes a category of data fusion techniques designed to enhance the spatial resolution of multispectral (MS) images by integrating spatial details from a high-resolution panchromatic (PAN) image. This process combines the high-spectral data of MS images with the rich spatial information of the PAN image, resulting in a pansharpened output ideal for more effective image analysis, such as object detection and environmental monitoring. Traditionally developed for satellite data, our paper introduces a novel pansharpening approach customized for the fusion of Scanning Electron Microscopy (SEM) and Energy-Dispersive X-ray Spectrometry (EDS) data. The proposed method, grounded in Partial Least Squares regression with Discriminant Analysis (PLS-DA), significantly boosts the spatial resolution of EDS data while preserving spectral details. A key feature of this approach involves partitioning the PAN image into intensity bins and dynamically adapting this division in cases of overlapping compounds with similar average atomic numbers. We evaluate the method's effectiveness using in-house EDS images obtained from both even and uneven sample surfaces. Comparative analysis against existing benchmarks and state-of-the-art pansharpening techniques demonstrates superior performance in both spectral and spatial quality indicators for our method.

Categorization of collagen type I and II blend hydrogel using multipolarization SHG imaging with ResNet regression

Previously, the discrimination of collagen types I and II was successfully achieved using peptide pitch angle and anisotropic parameter methods. However, these methods require fitting polarization second harmonic generation (SHG) pixel-wise information into generic mathematical models, revealing inconsistencies in categorizing collagen type I and II blend hydrogels. In this study, a ResNet approach based on multipolarization SHG imaging is proposed for the categorization and regression of collagen type I and II blend hydrogels at 0%, 25%, 50%, 75%, and 100% type II, without the need for prior time-consuming model fitting. A ResNet model, pretrained on 18 progressive polarization SHG images at 10° intervals for each percentage, categorizes the five blended collagen hydrogels with a mean absolute error (MAE) of 0.021, while the model pretrained on nonpolarization images exhibited 0.083 MAE. Moreover, the pretrained models can also generally regress the blend hydrogels at 20%, 40%, 60%, and 80% type II. In conclusion, the multipolarization SHG image-based ResNet analysis demonstrates the potential for an automated approach using deep learning to extract valuable information from the collagen matrix.

Assessment of encoder-decoder-based segmentation models for thyroid ultrasound images

Encoder-decoder-based semantic segmentation models classify image pixels into the corresponding class, such as the ROI (region of interest) or background. In the present study, simple / dilated convolution / series / directed acyclic graph (DAG)-based encoder-decoder semantic segmentation models have been implemented, i.e., SegNet (VGG16), SegNet (VGG19), U-Net, mobileNetv2, ResNet18, ResNet50, Xception and Inception networks for the segment TTUS(Thyroid Tumor Ultrasound) images. Transfer learning has been used to train these segmentation networks using original and despeckled TTUS images. The performance of the networks has been calculated using mIoU and mDC metrics. Based on the exhaustive experiments, it has been observed that ResNet50-based segmentation model obtained the best results objectively with values 0.87 for mIoU, 0.94 for mDC, and also according to radiologist opinion on shape, margin, and echogenicity characteristics of segmented lesions. It is noted that the segmentation model, namely ResNet50, provides better segmentation based on objective and subjective assessment. It may be used in the healthcare system to identify thyroid nodules accurately in real time.

HMFT: Hyperspectral and Multispectral Image Fusion Super-Resolution Method Based on Efficient Transformer and Spatial-Spectral Attention Mechanism

Due to the imaging mechanism of hyperspectral images, the spatial resolution of the resulting images is low. An effective method to solve this problem is to fuse the low-resolution hyperspectral image (LR-HSI) with the high-resolution multispectral image (HR-MSI) to generate the high-resolution hyperspectral image (HR-HSI). Currently, the state-of-the-art fusion approach is based on convolutional neural networks (CNN), and few have attempted to use Transformer, which shows impressive performance on advanced vision tasks. In this paper, a simple and efficient hybrid architecture network based on Transformer is proposed to solve the hyperspectral image fusion super-resolution problem. We use the clever combination of convolution and Transformer as the backbone network to fully extract spatial-spectral information by taking advantage of the local and global concerns of both. In order to pay more attention to the information features such as high-frequency information conducive to HR-HSI reconstruction and explore the correlation between spectra, the convolutional attention mechanism is used to further refine the extracted features in spatial and spectral dimensions, respectively. In addition, considering that the resolution of HSI is usually large, we use the feature split module (FSM) to replace the self-attention computation method of the native Transformer to reduce the computational complexity and storage scale of the model and greatly improve the efficiency of model training. Many experiments show that the proposed network architecture achieves the best qualitative and quantitative performance compared with the latest HSI super-resolution methods.

Hyperspectral Image Super-Resolution via Deep Spatiospectral Attention Convolutional Neural Networks

Hyperspectral images (HSIs) are of crucial importance in order to better understand features from a large number of spectral channels. Restricted by its inner imaging mechanism, the spatial resolution is often limited for HSIs. To alleviate this issue, in this work, we propose a simple and efficient architecture of deep convolutional neural networks to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to a new architecture based on: 1) the use of the LR-HSI at the HR-MSI's scale to get an output with satisfied spectral preservation and 2) the application of the attention and pixelShuffle modules to extract information, aiming to output high-quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves the best performance (both qualitatively and quantitatively) compared with recent state-of-the-art HSI super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as a better network generalization ability, a limited computational burden, and the robustness with respect to the number of training samples. Please find the source code and pretrained models from https://liangjiandeng.github.io/Projects_Res/HSRnet_2021tnnls.html.

U-Net based Mapping from Digital Images to Laser Doppler Imaging for Burn Assessment

The incidence of burn injuries is higher in low-and middle-income countries, and particularly in remote areas where the access to specialized burn assessment, care and recovery is limited. Given the high costs associated with one of the most used techniques to evaluate the severity of a burn, namely laser Doppler imaging (LDI), an alternative approach could be beneficial for remote locations. This study proposes a novel approach to estimate the LDI from digital images of a burn. The approach is a pixel-wise regression model based on convolutional neural networks. To minimize the dependency on the conditions in which the images are taken, the effect of two image normalization techniques is also studied. Results indicate that the model performs satisfactorily on average, presenting low mean absolute and squared errors and high structural similarity index. While no significant differences are found when changing the normalization of the images, the performance is affected by their quality. This suggests that changes in the intensity of the images do not alter the relevant information about the wound, whereas changes in brightness, contrast and sharpness do.

Unsupervised super-resolution reconstruction of hyperspectral histology images for whole-slide imaging

SIGNIFICANCE

Hyperspectral imaging (HSI) provides rich spectral information for improved histopathological cancer detection. However, acquiring high-resolution HSI data for whole-slide imaging (WSI) can be time-consuming and requires a huge amount of storage space.

AIM

WSI using a color camera can be achieved with fast speed, high image resolution, and excellent image quality due to the established techniques. We aim to develop an RGB-guided unsupervised hyperspectral super-resolution reconstruction method that is hypothesized to improve image quality while maintaining the spectral characteristics.

APPROACH

High-resolution hyperspectral images of 32 histologic slides were obtained via automated WSI. High-resolution RGB histology images were registered to the hyperspectral images for RGB guidance. An unsupervised super-resolution network was trained to take the downsampled low-resolution hyperspectral patches (LR-HSI) and high-resolution RGB patches (HR-RGB) as inputs to reconstruct high-resolution hyperspectral patches (HR-HSI). Then, an Inception-based network was trained with the HR-RGB, original HR-HSI, and generated HR-HSI, respectively, for whole-slide histopathological cancer detection.

RESULTS

Our super-resolution reconstruction network generated high-resolution hyperspectral images with well-maintained spectral characteristics and improved image quality. Image classification using the original hyperspectral data outperformed RGB because of the extra spectral information. The generated hyperspectral image patches further improved the results.

CONCLUSIONS

The proposed method potentially reduces image acquisition time, saves storage space without compromising image quality, and improves the image classification performance.

Deep Learning-Enabled Technologies for Bioimage Analysis

Deep learning (DL) is a subfield of machine learning (ML), which has recently demonstrated its potency to significantly improve the quantification and classification workflows in biomedical and clinical applications. Among the end applications profoundly benefitting from DL, cellular morphology quantification is one of the pioneers. Here, we first briefly explain fundamental concepts in DL and then we review some of the emerging DL-enabled applications in cell morphology quantification in the fields of embryology, point-of-care ovulation testing, as a predictive tool for fetal heart pregnancy, cancer diagnostics via classification of cancer histology images, autosomal polycystic kidney disease, and chronic kidney diseases.

Unsupervised Super Resolution Network for Hyperspectral Histologic Imaging

Hyperspectral imaging (HSI) has many advantages in microscopic applications, including high sensitivity and specificity for cancer detection on histological slides. However, acquiring hyperspectral images of a whole slide with a high image resolution and a high image quality can take a long scanning time and require a very large data storage. One potential solution is to acquire and save low-resolution hyperspectral images and reconstruct the high-resolution ones only when needed. The purpose of this study is to develop a simple yet effective unsupervised super resolution network for hyperspectral histologic imaging with the guidance of RGB digital histology images. High-resolution hyperspectral images of hemoxylin & eosin (H&E) stained slides were obtained at 10× magnification and down-sampled 2×, 4×, and 5× to generate low-resolution hyperspectral data. High-resolution digital histologic RGB images of the same field of view (FOV) were cropped and registered to the corresponding high-resolution hyperspectral images. A neural network based on a modified U-Net architecture, which takes the low-resolution hyperspectral images and high-resolution RGB images as inputs, was trained with unsupervised methods to output high-resolution hyperspectral data. The generated high-resolution hyperspectral images have similar spectral signatures and improved image contrast than the original high-resolution hyperspectral images, which indicates that the super resolution network with RGB guidance can improve the image quality. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will potentially promote the use of hyperspectral imaging technology in digital pathology and many other clinical applications.

Detecting spiral wave tips using deep learning

The chaotic spatio-temporal electrical activity during life-threatening cardiac arrhythmias like ventricular fibrillation is governed by the dynamics of vortex-like spiral or scroll waves. The organizing centers of these waves are called wave tips (2D) or filaments (3D) and they play a key role in understanding and controlling the complex and chaotic electrical dynamics. Therefore, in many experimental and numerical setups it is required to detect the tips of the observed spiral waves. Most of the currently used methods significantly suffer from the influence of noise and are often adjusted to a specific situation (e.g. a specific numerical cardiac cell model). In this study, we use a specific type of deep neural networks (UNet), for detecting spiral wave tips and show that this approach is robust against the influence of intermediate noise levels. Furthermore, we demonstrate that if the UNet is trained with a pool of numerical cell models, spiral wave tips in unknown cell models can also be detected reliably, suggesting that the UNet can in some sense learn the concept of spiral wave tips in a general way, and thus could also be used in experimental situations in the future (ex-vivo, cell-culture or optogenetic experiments).

An Improved Version of the Generalized Laplacian Pyramid Algorithm for Pansharpening

The spatial resolution of multispectral data can be synthetically improved by exploiting the spatial content of a companion panchromatic image. This process, named pansharpening, is widely employed by data providers to augment the quality of images made available for many applications. The huge demand requires the utilization of efficient fusion algorithms that do not require specific training phases, but rather exploit physical considerations to combine the available data. For this reason, classical model-based approaches are still widely used in practice. We created and assessed a method for improving a widespread approach, based on the generalized Laplacian pyramid decomposition, by combining two different cost-effective upgrades: the estimation of the detail-extraction filter from data and the utilization of an improved injection scheme based on multilinear regression. The proposed method was compared with several existing efficient pansharpening algorithms, employing the most credited performance evaluation protocols. The capability of achieving optimal results in very different scenarios was demonstrated by employing data acquired by the IKONOS and WorldView-3 satellites.

Lesion probability mapping in MS patients using a regression network on MR fingerprinting

BACKGROUND

To develop a regression neural network for the reconstruction of lesion probability maps on Magnetic Resonance Fingerprinting using echo-planar imaging (MRF-EPI) in addition to [Formula: see text], [Formula: see text], NAWM, and GM- probability maps.

METHODS

We performed MRF-EPI measurements in 42 patients with multiple sclerosis and 6 healthy volunteers along two sites. A U-net was trained to reconstruct the denoised and distortion corrected [Formula: see text] and [Formula: see text] maps, and to additionally generate NAWM-, GM-, and WM lesion probability maps.

RESULTS

WM lesions were predicted with a dice coefficient of [Formula: see text] and a lesion detection rate of [Formula: see text] for a threshold of 33%. The network jointly enabled accurate [Formula: see text] and [Formula: see text] times with relative deviations of 5.2% and 5.1% and average dice coefficients of [Formula: see text] and [Formula: see text] for NAWM and GM after binarizing with a threshold of 80%.

CONCLUSION

DL is a promising tool for the prediction of lesion probability maps in a fraction of time. These might be of clinical interest for the WM lesion analysis in MS patients.

Deep-Learning-Driven Full-Waveform Inversion for Ultrasound Breast Imaging

Ultrasound breast imaging is a promising alternative to conventional mammography because it does not expose women to harmful ionising radiation and it can successfully image dense breast tissue. However, conventional ultrasound imaging only provides morphological information with limited diagnostic value. Ultrasound computed tomography (USCT) uses energy in both transmission and reflection when imaging the breast to provide more diagnostically relevant quantitative tissue properties, but it is often based on time-of-flight tomography or similar ray approximations of the wave equation, resulting in reconstructed images with low resolution. Full-waveform inversion (FWI) is based on a more accurate approximation of wave-propagation phenomena and can consequently produce very high resolution images using frequencies below 1 megahertz. These low frequencies, however, are not available in most USCT acquisition systems, as they use transducers with central frequencies well above those required in FWI. To circumvent this problem, we designed, trained, and implemented a two-dimensional convolutional neural network to artificially generate missing low frequencies in USCT data. Our results show that FWI reconstructions using experiment data after the application of the proposed method successfully converged, showing good agreement with X-ray CT and reflection ultrasound-tomography images.

Deep-Learning-Driven Full-Waveform Inversion for Ultrasound Breast Imaging

Ultrasound breast imaging is a promising alternative to conventional mammography because it does not expose women to harmful ionising radiation and it can successfully image dense breast tissue. However, conventional ultrasound imaging only provides morphological information with limited diagnostic value. Ultrasound computed tomography (USCT) uses energy in both transmission and reflection when imaging the breast to provide more diagnostically relevant quantitative tissue properties, but it is often based on time-of-flight tomography or similar ray approximations of the wave equation, resulting in reconstructed images with low resolution. Full-waveform inversion (FWI) is based on a more accurate approximation of wave-propagation phenomena and can consequently produce very high resolution images using frequencies below 1 megahertz. These low frequencies, however, are not available in most USCT acquisition systems, as they use transducers with central frequencies well above those required in FWI. To circumvent this problem, we designed, trained, and implemented a two-dimensional convolutional neural network to artificially generate missing low frequencies in USCT data. Our results show that FWI reconstructions using experiment data after the application of the proposed method successfully converged, showing good agreement with X-ray CT and reflection ultrasound-tomography images.

A Comparison of Machine Learning Approaches to Improve Free Topography Data for Flood Modelling

Given the high financial and institutional cost of collecting and processing accurate topography data, many large-scale flood hazard assessments continue to rely instead on freely-available global Digital Elevation Models, despite the significant vertical biases known to affect them. To predict (and thereby reduce) these biases, we apply a fully-convolutional neural network (FCN), a form of artificial neural network originally developed for image segmentation which is capable of learning from multi-variate spatial patterns at different scales. We assess its potential by training such a model on a wide variety of remote-sensed input data (primarily multi-spectral imagery), using high-resolution, LiDAR-derived Digital Terrain Models published by the New Zealand government as the reference topography data. In parallel, two more widely used machine learning models are also trained, in order to provide benchmarks against which the novel FCN may be assessed. We find that the FCN outperforms the other models (reducing root mean square error in the testing dataset by 71%), likely due to its ability to learn from spatial patterns at multiple scales, rather than only a pixel-by-pixel basis. Significantly for flood hazard modelling applications, corrections were found to be especially effective along rivers and their floodplains. However, our results also suggest that models are likely to be biased towards the land cover and relief conditions most prevalent in their training data, with further work required to assess the importance of limiting training data inputs to those most representative of the intended application area(s).

Fringe projection profilometry by conducting deep learning from its digital twin

High-accuracy and high-speed three-dimensional (3D) fringe projection profilometry (FPP) has been widely applied in many fields. Recently, researchers discovered that deep learning can significantly improve fringe analysis. However, deep learning requires numerous objects to be scanned for training data. In this paper, we propose to build the digital twin of an FPP system and perform virtual scanning using computer graphics, which can significantly save cost and labor. The proposed method extracts 3D geometry directly from a single-shot fringe image, and real-world experiments have demonstrated the success of the virtually trained model. Our virtual scanning method can automatically generate 7,200 fringe images and 800 corresponding 3D scenes within 1.5 hours.

Structural Similarity Loss for Learning to Fuse Multi-Focus Images

Convolutional neural networks have recently been used for multi-focus image fusion. However, some existing methods have resorted to adding Gaussian blur to focused images, to simulate defocus, thereby generating data (with ground-truth) for supervised learning. Moreover, they classify pixels as 'focused' or 'defocused', and use the classified results to construct the fusion weight maps. This then necessitates a series of post-processing steps. In this paper, we present an end-to-end learning approach for directly predicting the fully focused output image from multi-focus input image pairs. The suggested approach uses a CNN architecture trained to perform fusion, without the need for ground truth fused images. The CNN exploits the image structural similarity (SSIM) to calculate the loss, a metric that is widely accepted for fused image quality evaluation. What is more, we also use the standard deviation of a local window of the image to automatically estimate the importance of the source images in the final fused image when designing the loss function. Our network can accept images of variable sizes and hence, we are able to utilize real benchmark datasets, instead of simulated ones, to train our network. The model is a feed-forward, fully convolutional neural network that can process images of variable sizes during test time. Extensive evaluation on benchmark datasets show that our method outperforms, or is comparable with, existing state-of-the-art techniques on both objective and subjective benchmarks.

Convolutional neural network based proton stopping-power-ratio estimation with dual-energy CT: a feasibility study

Dual-energy computed tomography (DECT) has shown a great potential for lowering range uncertainties, which is necessary for truly leveraging the Bragg peak in proton therapy. However, analytical stopping-power-ratio (SPR) estimation methods have limitations in resolving the influence from the beam-hardening artifact, i.e. CT number variation of the same object scanned under different imaging conditions, such as different patient size and location in the field-of-view (FOV). We present a convolutional neural network (CNN)-based framework to estimate proton SPR that accounts for patient geometry variation and addresses CT number variation. The proposed framework was tested on both prostate and head-and-neck (HN) patient datasets. Simulated CT images were used in order to have a well-defined ground-truth SPR for evaluation. Two training scenarios were evaluated: training with patient CT images (ideal scenario) and training with computational phantoms (realistic scenario). For the training in ideal scenario, computational phantoms were created based on 120 kVp patient CT images using a custom-defined density and material translation curve. Then, 80 kVp and 150 kVp Sn DECT image pairs were obtained using ray-tracing simulation, and their corresponding SPR was calculated from the known density and elemental compositions. For the training in realistic scenario, computational phantoms were created based on the geometry of calibration phantoms. For both scenarios, evaluation was performed on the phantoms created from patient CT images. Compared to a conventional parametric model, U-net trained with computational phantoms (realistic scenario) reduced the SPR estimation uncertainty (95th percentile) of the prostate patient from 1.10% to 0.71%, and HN patient from 2.11% to 1.20%. With the U-net trained with patient images (ideal scenario) uncertainty values were 0.32% and 0.42% for prostate and HN patients, respectively. These results suggest that CNN has great potential to improve the accuracy of SPR estimation in proton therapy by incorporating individual patient geometry information.

Mapping Post-Earthquake Landslide Susceptibility: A U-Net Like Approach

A serious earthquake could trigger thousands of landslides and produce some slopes more sensitive to slide in future. Landslides could threaten human’s lives and properties, and thus mapping the post-earthquake landslide susceptibility is very valuable for a rapid response to landslide disasters in terms of relief resource allocation and posterior earthquake reconstruction. Previous researchers have proposed many methods to map landslide susceptibility but seldom considered the spatial structure information of the factors that influence a slide. In this study, we first developed a U-net like model suitable for mapping post-earthquake landslide susceptibility. The post-earthquake high spatial airborne images were used for producing a landslide inventory. Pre-earthquake Landsat TM (Thematic Mapper) images and the influencing factors such as digital elevation model (DEM), slope, aspect, multi-scale topographic position index (mTPI), lithology, fault, road network, streams network, and macroseismic intensity (MI) were prepared as the input layers of the model. Application of the model to the heavy-hit area of the destructive 2008 Wenchuan earthquake resulted in a high validation accuracy (precision 0.77, recall 0.90, F1 score 0.83, and AUC 0.90). The performance of this U-net like model was also compared with those of traditional logistic regression (LR) and support vector machine (SVM) models on both the model area and independent testing area with the former being stronger than the two traditional models. The U-net like model introduced in this paper provides us the inspiration that balancing the environmental influence of a pixel itself and its surrounding pixels to perform a better landslide susceptibility mapping (LSM) task is useful and feasible when using remote sensing and GIS technology.

A Deep Learning-Based Robust Change Detection Approach for Very High Resolution Remotely Sensed Images with Multiple Features

Very high-resolution remote sensing change detection has always been an important research issue due to the registration error, robustness of the method, and monitoring accuracy, etc. This paper proposes a robust and more accurate approach of change detection (CD), and it is applied on a smaller experimental area, and then extended to a wider range. A feature space, including object features, Visual Geometry Group (VGG) depth features, and texture features, is constructed. The difference image is obtained by considering the contextual information in a radius scalable circular. This is to overcome the registration error caused by the rotation and shift of the instantaneous field of view and also to improve the reliability and robustness of the CD. To enhance the robustness of the U-Net model, the training dataset is constructed manually via various operations, such as blurring the image, increasing noise, and rotating the image. After this, the trained model is used to predict the experimental areas, which achieved 92.3% accuracy. The proposed method is compared with Support Vector Machine (SVM) and Siamese Network, and the check error rate dropped to 7.86%, while the Kappa increased to 0.8254. The results revealed that our method outperforms SVM and Siamese Network.

SD-UNet: Stripping Down U-Net for Segmentation of Biomedical Images on Platforms with Low Computational Budgets

During image segmentation tasks in computer vision, achieving high accuracy performance while requiring fewer computations and faster inference is a big challenge. This is especially important in medical imaging tasks but one metric is usually compromised for the other. To address this problem, this paper presents an extremely fast, small and computationally effective deep neural network called Stripped-Down UNet (SD-UNet), designed for the segmentation of biomedical data on devices with limited computational resources. By making use of depthwise separable convolutions in the entire network, we design a lightweight deep convolutional neural network architecture inspired by the widely adapted U-Net model. In order to recover the expected performance degradation in the process, we introduce a weight standardization algorithm with the group normalization method. We demonstrate that SD-UNet has three major advantages including: (i) smaller model size (23x smaller than U-Net); (ii) 8x fewer parameters; and (iii) faster inference time with a computational complexity lower than 8M floating point operations (FLOPs). Experiments on the benchmark dataset of the Internatioanl Symposium on Biomedical Imaging (ISBI) challenge for segmentation of neuronal structures in electron microscopic (EM) stacks and the Medical Segmentation Decathlon (MSD) challenge brain tumor segmentation (BRATs) dataset show that the proposed model achieves comparable and sometimes better results compared to the current state-of-the-art.

Survey of Deep-Learning Approaches for Remote Sensing Observation Enhancement

Deep Learning, and Deep Neural Networks in particular, have established themselves as the new norm in signal and data processing, achieving state-of-the-art performance in image, audio, and natural language understanding. In remote sensing, a large body of research has been devoted to the application of deep learning for typical supervised learning tasks such as classification. Less yet equally important effort has also been allocated to addressing the challenges associated with the enhancement of low-quality observations from remote sensing platforms. Addressing such channels is of paramount importance, both in itself, since high-altitude imaging, environmental conditions, and imaging systems trade-offs lead to low-quality observation, as well as to facilitate subsequent analysis, such as classification and detection. In this paper, we provide a comprehensive review of deep-learning methods for the enhancement of remote sensing observations, focusing on critical tasks including single and multi-band super-resolution, denoising, restoration, pan-sharpening, and fusion, among others. In addition to the detailed analysis and comparison of recently presented approaches, different research avenues which could be explored in the future are also discussed.

Phase Extraction from Single Interferogram Including Closed-Fringe Using Deep Learning

In an optical measurement system using an interferometer, a phase extracting technique from interferogram is the key issue. When the object is varying in time, the Fourier-transform method is commonly used since this method can extract a phase image from a single interferogram. However, there is a limitation, that an interferogram including closed-fringes cannot be applied. The closed-fringes appear when intervals of the background fringes are long. In some experimental setups, which need to change the alignments of optical components such as a 3-D optical tomographic system, the interval of the fringes cannot be controlled. To extract the phase from the interferogram including the closed-fringes we propose the use of deep learning. A large amount of the pairs of the interferograms and phase-shift images are prepared, and the trained network, the input for which is an interferogram and the output a corresponding phase-shift image, is obtained using supervised learning. From comparisons of the extracted phase, we can demonstrate that the accuracy of the trained network is superior to that of the Fourier-transform method. Furthermore, the trained network can be applicable to the interferogram including the closed-fringes, which is impossible with the Fourier transform method.