A comprehensive review of deep learning-based single image super-resolution

A Novel Simulation Method for 3D Digital-Image Correlation: Combining Virtual Stereo Vision and Image Super-Resolution Reconstruction

3D digital-image correlation (3D-DIC) is a non-contact optical technique for full-field shape, displacement, and deformation measurement. Given the high experimental hardware costs associated with 3D-DIC, the development of high-fidelity 3D-DIC simulations holds significant value. However, existing research on 3D-DIC simulation was mainly carried out through the generation of random speckle images. This study innovatively proposes a complete 3D-DIC simulation method involving optical simulation and mechanical simulation and integrating 3D-DIC, virtual stereo vision, and image super-resolution reconstruction technology. Virtual stereo vision can reduce hardware costs and eliminate camera-synchronization errors. Image super-resolution reconstruction can compensate for the decrease in precision caused by image-resolution loss. An array of software tools such as ANSYS SPEOS 2024R1, ZEMAX 2024R1, MECHANICAL 2024R1, and MULTIDIC v1.1.0 are used to implement this simulation. Measurement systems based on stereo vision and virtual stereo vision were built and tested for use in 3D-DIC. The results of the simulation experiment show that when the synchronization error of the basic stereo-vision system (BSS) is within 10-3 time steps, the reconstruction error is within 0.005 mm and the accuracy of the virtual stereo-vision system is between the BSS's synchronization error of 10-7 and 10-6 time steps. In addition, after image super-resolution reconstruction technology is applied, the reconstruction error will be reduced to within 0.002 mm. The simulation method proposed in this study can provide a novel research path for existing researchers in the field while also offering the opportunity for researchers without access to costly hardware to participate in related research.

PlantSR: Super-Resolution Improves Object Detection in Plant Images

Recent advancements in computer vision, especially deep learning models, have shown considerable promise in tasks related to plant image object detection. However, the efficiency of these deep learning models heavily relies on input image quality, with low-resolution images significantly hindering model performance. Therefore, reconstructing high-quality images through specific techniques will help extract features from plant images, thus improving model performance. In this study, we explored the value of super-resolution technology for improving object detection model performance on plant images. Firstly, we built a comprehensive dataset comprising 1030 high-resolution plant images, named the PlantSR dataset. Subsequently, we developed a super-resolution model using the PlantSR dataset and benchmarked it against several state-of-the-art models designed for general image super-resolution tasks. Our proposed model demonstrated superior performance on the PlantSR dataset, indicating its efficacy in enhancing the super-resolution of plant images. Furthermore, we explored the effect of super-resolution on two specific object detection tasks: apple counting and soybean seed counting. By incorporating super-resolution as a pre-processing step, we observed a significant reduction in mean absolute error. Specifically, with the YOLOv7 model employed for apple counting, the mean absolute error decreased from 13.085 to 5.71. Similarly, with the P2PNet-Soy model utilized for soybean seed counting, the mean absolute error decreased from 19.159 to 15.085. These findings underscore the substantial potential of super-resolution technology in improving the performance of object detection models for accurately detecting and counting specific plants from images. The source codes and associated datasets related to this study are available at Github.

Bio-inspired foveal super-resolution method for multi-focal-length images based on local gradient constraints

Most existing super-resolution (SR) imaging systems, inspired by the bionic compound eye, utilize image registration and reconstruction algorithms to overcome the angular resolution limitations of individual imaging systems. This article introduces a multi-aperture multi-focal-length imaging system and a multi-focal-length image super-resolution algorithm, mimicking the foveal imaging of the human eye. Experimental results demonstrate that with the proposed imaging system and an SR imaging algorithm inspired by the human visual system, the proposed method can enhance the spatial resolution of the foveal region by up to 4 × compared to the original acquired image. These findings validate the effectiveness of the proposed imaging system and computational imaging algorithm in enhancing image texture and spatial resolution.

Generalizing the Enhanced-Deep-Super-Resolution Neural Network to Brain MR Images: A Retrospective Study on the Cam-CAN Dataset

The Enhanced-Deep-Super-Resolution (EDSR) model is a state-of-the-art convolutional neural network suitable for improving image spatial resolution. It was previously trained with general-purpose pictures and then, in this work, tested on biomedical magnetic resonance (MR) images, comparing the network outcomes with traditional up-sampling techniques. We explored possible changes in the model response when different MR sequences were analyzed. T1w and T2w MR brain images of 70 human healthy subjects (F:M, 40:30) from the Cambridge Centre for Ageing and Neuroscience (Cam-CAN) repository were down-sampled and then up-sampled using EDSR model and BiCubic (BC) interpolation. Several reference metrics were used to quantitatively assess the performance of up-sampling operations (RMSE, pSNR, SSIM, and HFEN). Two-dimensional and three-dimensional reconstructions were evaluated. Different brain tissues were analyzed individually. The EDSR model was superior to BC interpolation on the selected metrics, both for two- and three- dimensional reconstructions. The reference metrics showed higher quality of EDSR over BC reconstructions for all the analyzed images, with a significant difference of all the criteria in T1w images and of the perception-based SSIM and HFEN in T2w images. The analysis per tissue highlights differences in EDSR performance related to the gray-level values, showing a relative lack of outperformance in reconstructing hyperintense areas. The EDSR model, trained on general-purpose images, better reconstructs MR T1w and T2w images than BC, without any retraining or fine-tuning. These results highlight the excellent generalization ability of the network and lead to possible applications on other MR measurements.

Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks

Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.

Twinned Residual Auto-Encoder (TRAE)-A new DL architecture for denoising super-resolution and task-aware feature learning from COVID-19 CT images

The detection of the COronaVIrus Disease 2019 (COVID-19) from Computed Tomography (CT) scans has become a very important task in modern medical diagnosis. Unfortunately, typical resolutions of state-of-the-art CT scans are still not adequate for reliable and accurate automatic detection of COVID-19 disease. Motivated by this consideration, in this paper, we propose a novel architecture that jointly affords the Single-Image Super-Resolution (SISR) and the reliable classification problems from Low Resolution (LR) and noisy CT scans. Specifically, the proposed architecture is based on a couple of Twinned Residual Auto-Encoders (TRAE), which exploits the feature vectors and the SR images recovered by a Master AE for performing transfer learning and then improves the training of a "twinned" Follower AE. In addition, we also develop a Task-Aware (TA) version of the basic TRAE architecture, namely the TA-TRAE, which further utilizes the set of feature vectors generated by the Follower AE for the joint training of an additional auxiliary classifier, so to perform automated medical diagnosis on the basis of the available LR input images without human support. Experimental results and comparisons with a number of state-of-the-art CNN/GAN/CycleGAN benchmark SISR architectures, performed by considering × 2 , × 4 , and × 8 super-resolution (i.e., upscaling) factors, support the effectiveness of the proposed TRAE/TA-TRAE architectures. In particular, the detection accuracy attained by the proposed architectures outperforms the corresponding ones of the implemented CNN, GAN and CycleGAN baselines up to 9.0%, 6.5%, and 6.0% at upscaling factors as high as × 8 .

A fine-tuned YOLOv5 deep learning approach for real-time house number detection

Detection of small objects in natural scene images is a complicated problem due to the blur and depth found in the images. Detecting house numbers from the natural scene images in real-time is a computer vision problem. On the other hand, convolutional neural network (CNN) based deep learning methods have been widely used in object detection in recent years. In this study, firstly, a classical CNN-based approach is used to detect house numbers with locations from natural images in real-time. Faster R-CNN, MobileNet, YOLOv4, YOLOv5 and YOLOv7, among the commonly used CNN models, models were applied. However, satisfactory results could not be obtained due to the small size and variable depth of the door plate objects. A new approach using the fine-tuning technique is proposed to improve the performance of CNN-based deep learning models. Experimental evaluations were made on real data from Kayseri province. Classic Faster R-CNN, MobileNet, YOLOv4, YOLOv5 and YOLOv7 methods yield f1 scores of 0.763, 0.677, 0.880, 0.943 and 0.842, respectively. The proposed fine-tuned Faster R-CNN, MobileNet, YOLOv4, YOLOv5, and YOLOv7 approaches achieved f1 scores of 0.845, 0.775, 0.932, 0.972 and 0.889, respectively. Thanks to the proposed fine-tuned approach, the f1 score of all models has increased. Regarding the run time of the methods, classic Faster R-CNN detects 0.603 seconds, while fine-tuned Faster R-CNN detects 0.633 seconds. Classic MobileNet detects 0.046 seconds, while fine-tuned MobileNet detects 0.048 seconds. Classic YOLOv4 and fine-tuned YOLOv4 detect 0.235 and 0.240 seconds, respectively. Classic YOLOv5 and fine-tuned YOLOv5 detect 0.015 seconds, and classic YOLOv7 and fine-tuned YOLOv7 detect objects in 0.009 seconds. While the YOLOv7 model was the fastest running model with an average running time of 0.009 seconds, the proposed fine-tuned YOLOv5 approach achieved the highest performance with an f1 score of 0.972.

TranSMS: Transformers for Super-Resolution Calibration in Magnetic Particle Imaging

Magnetic particle imaging (MPI) offers exceptional contrast for magnetic nanoparticles (MNP) at high spatio-temporal resolution. A common procedure in MPI starts with a calibration scan to measure the system matrix (SM), which is then used to set up an inverse problem to reconstruct images of the MNP distribution during subsequent scans. This calibration enables the reconstruction to sensitively account for various system imperfections. Yet time-consuming SM measurements have to be repeated under notable changes in system properties. Here, we introduce a novel deep learning approach for accelerated MPI calibration based on Transformers for SM super-resolution (TranSMS). Low-resolution SM measurements are performed using large MNP samples for improved signal-to-noise ratio efficiency, and the high-resolution SM is super-resolved via model-based deep learning. TranSMS leverages a vision transformer module to capture contextual relationships in low-resolution input images, a dense convolutional module for localizing high-resolution image features, and a data-consistency module to ensure measurement fidelity. Demonstrations on simulated and experimental data indicate that TranSMS significantly improves SM recovery and MPI reconstruction for up to 64-fold acceleration in two-dimensional imaging.

How much BiGAN and CycleGAN-learned hidden features are effective for COVID-19 detection from CT images? A comparative study

Bidirectional generative adversarial networks (BiGANs) and cycle generative adversarial networks (CycleGANs) are two emerging machine learning models that, up to now, have been used as generative models, i.e., to generate output data sampled from a target probability distribution. However, these models are also equipped with encoding modules, which, after weakly supervised training, could be, in principle, exploited for the extraction of hidden features from the input data. At the present time, how these extracted features could be effectively exploited for classification tasks is still an unexplored field. Hence, motivated by this consideration, in this paper, we develop and numerically test the performance of a novel inference engine that relies on the exploitation of BiGAN and CycleGAN-learned hidden features for the detection of COVID-19 disease from other lung diseases in computer tomography (CT) scans. In this respect, the main contributions of the paper are twofold. First, we develop a kernel density estimation (KDE)-based inference method, which, in the training phase, leverages the hidden features extracted by BiGANs and CycleGANs for estimating the (a priori unknown) probability density function (PDF) of the CT scans of COVID-19 patients and, then, in the inference phase, uses it as a target COVID-PDF for the detection of COVID diseases. As a second major contribution, we numerically evaluate and compare the classification accuracies of the implemented BiGAN and CycleGAN models against the ones of some state-of-the-art methods, which rely on the unsupervised training of convolutional autoencoders (CAEs) for attaining feature extraction. The performance comparisons are carried out by considering a spectrum of different training loss functions and distance metrics. The obtained classification accuracies of the proposed CycleGAN-based (resp., BiGAN-based) models outperform the corresponding ones of the considered benchmark CAE-based models of about 16% (resp., 14%).

Reconstructing high fidelity digital rock images using deep convolutional neural networks

Imaging methods have broad applications in geosciences. Scanning electron microscopy (SEM) and micro-CT scanning have been applied for studying various geological problems. Despite significant advances in imaging capabilities, and image processing algorithms, acquiring high-quality data from images is still challenging and time-consuming. Obtaining a 3D representative volume for a tight rock sample takes days to weeks. Image artifacts such as noise further complicate the use of imaging methods for the determination of rock properties. In this study, we present applications of several convolutional neural networks (CNN) for rapid image denoising, deblurring and super-resolving digital rock images. Such an approach enables rapid imaging of larger samples, which in turn improves the statistical relevance of the subsequent analysis. We demonstrate the application of several CNNs for image restoration applicable to scientific imaging. The results show that images can be denoised without a priori knowledge of the noise with great confidence. Furthermore, we show how attaching several CNNs in an end-to-end fashion can improve the final quality of reconstruction. Our experiments with SEM and CT scan images of several rock types show image denoising, deblurring and super-resolution can be performed simultaneously.