Superresolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging

Self-supervised multicontrast super-resolution for diffusion-weighted prostate MRI

PURPOSE

This study addresses the challenge of low resolution and signal-to-noise ratio (SNR) in diffusion-weighted images (DWI), which are pivotal for cancer detection. Traditional methods increase SNR at high b-values through multiple acquisitions, but this results in diminished image resolution due to motion-induced variations. Our research aims to enhance spatial resolution by exploiting the global structure within multicontrast DWI scans and millimetric motion between acquisitions.

METHODS

We introduce a novel approach employing a "Perturbation Network" to learn subvoxel-size motions between scans, trained jointly with an implicit neural representation (INR) network. INR encodes the DWI as a continuous volumetric function, treating voxel intensities of low-resolution acquisitions as discrete samples. By evaluating this function with a finer grid, our model predicts higher-resolution signal intensities for intermediate voxel locations. The Perturbation Network's motion-correction efficacy was validated through experiments on biological phantoms and in vivo prostate scans.

RESULTS

Quantitative analyses revealed significantly higher structural similarity measures of super-resolution images to ground truth high-resolution images compared to high-order interpolation (p< $$ < $$ 0.005). In blind qualitative experiments,96 . 1 % $$ 96.1\% $$ of super-resolution images were assessed to have superior diagnostic quality compared to interpolated images.

CONCLUSION

High-resolution details in DWI can be obtained without the need for high-resolution training data. One notable advantage of the proposed method is that it does not require a super-resolution training set. This is important in clinical practice because the proposed method can easily be adapted to images with different scanner settings or body parts, whereas the supervised methods do not offer such an option.

Super-resolution techniques for biomedical applications and challenges

Super-resolution (SR) techniques have revolutionized the field of biomedical applications by detailing the structures at resolutions beyond the limits of imaging or measuring tools. These techniques have been applied in various biomedical applications, including microscopy, magnetic resonance imaging (MRI), computed tomography (CT), X-ray, electroencephalogram (EEG), ultrasound, etc. SR methods are categorized into two main types: traditional non-learning-based methods and modern learning-based approaches. In both applications, SR methodologies have been effectively utilized on biomedical images, enhancing the visualization of complex biological structures. Additionally, these methods have been employed on biomedical data, leading to improvements in computational precision and efficiency for biomedical simulations. The use of SR techniques has resulted in more detailed and accurate analyses in diagnostics and research, essential for early disease detection and treatment planning. However, challenges such as computational demands, data interpretation complexities, and the lack of unified high-quality data persist. The article emphasizes these issues, underscoring the need for ongoing development in SR technologies to further improve biomedical research and patient care outcomes.

Neighborhood evaluator for efficient super-resolution reconstruction of 2D medical images

BACKGROUND

Deep learning-based super-resolution (SR) algorithms aim to reconstruct low-resolution (LR) images into high-fidelity high-resolution (HR) images by learning the low- and high-frequency information. Experts' diagnostic requirements are fulfilled in medical application scenarios through the high-quality reconstruction of LR digital medical images.

PURPOSE

Medical image SR algorithms should satisfy the requirements of arbitrary resolution and high efficiency in applications. However, there is currently no relevant study available. Several SR research on natural images have accomplished the reconstruction of resolutions without limitations. However, these methodologies provide challenges in meeting medical applications due to the large scale of the model, which significantly limits efficiency. Hence, we suggest a highly effective method for reconstructing medical images at any desired resolution.

METHODS

Statistical features of medical images exhibit greater continuity in the region of neighboring pixels than natural images. Hence, the process of reconstructing medical images is comparatively less challenging. Utilizing this property, we develop a neighborhood evaluator to represent the continuity of the neighborhood while controlling the network's depth.

RESULTS

The suggested method has superior performance across seven scales of reconstruction, as evidenced by experiments conducted on panoramic radiographs and two external public datasets. Furthermore, the proposed network significantly decreases the parameter count by over 20× and the computational workload by over 10× compared to prior researches. On large-scale reconstruction, the inference speed can be enhanced by over 5×.

CONCLUSION

The novel proposed SR strategy for medical images performs efficient reconstruction at arbitrary resolution, marking a significant breakthrough in the field. The given scheme facilitates the implementation of SR in mobile medical platforms.

Rapid microscopic 3D-diffusion tensor imaging fiber-tracking of mouse brain in vivo by super resolution reconstruction: validation on MAP6-KO mouse model

OBJECT

Exploring mouse brains by rapid 3D-Diffusion Tensor Imaging (3D-DTI) of high spatial resolution (HSR) is challenging in vivo. Here we use the super resolution reconstruction (SRR) postprocessing method to demonstrate its performance on Microtubule-Associated-Protein6 Knock-Out (MAP6-KO) mice.

MATERIALS AND METHODS

Two spin-echo DTI were acquired (9.4T, CryoProbe RF-coil): (i)-multislice 2D-DTI, (echo-planar integrating reversed-gradient) acquired in vivo in the three orthogonal orientations (360 μm slice-thickness, 120 × 120 μm in-plane resolution, 56 min scan duration); used in SRR software to reconstruct SRR 3D-DTI with HSR in slice-plane (120 × 120 × 120 µm) and (ii)-microscopic 3D-DTI (µ-3D-DTI), (100 × 100 × 100 µm; 8 h 6 min) on fixed-brains ex vivo, that were removed after paramagnetic contrast-agent injection to accelerate scan acquisition using short repetition-times without NMR-signal sensitivity loss.

RESULTS

White-matter defects, quantified from both 3D-DTI fiber-tracking were found very similar. Indeed, as expected the fornix and cerebral-peduncle volume losses were - 39% and - 35% in vivo (SRR 3D-DTI) versus - 34% and - 32% ex vivo (µ-3D-DTI), respectively (p<0.001). This finding is robust since the µ-3D-DTI feasibility on MAP6-KO ex vivo was already validated by fluorescent-microscopy of cleared brains.

DISCUSSION

First performance of the SRR to generate rapid HSR 3D-DTI of mouse brains in vivo is demonstrated. The method is suitable in neurosciences for longitudinal studies to identify molecular and genetic abnormalities in mouse models that are of growing developments.

ESRGAN-DP: Enhanced super-resolution generative adversarial network with adaptive dual perceptual loss

The proposal of perceptual loss solves the over-smoothing problem of images caused by pixel-wise loss and improves the visual quality of the images, but it also inevitably produces a large number of artifacts and distortions in images. The reason for this phenomenon is that the perceptual features only rely on a single pre-trained visual geometry group (VGG) network, which results in the features of the image being unable to be fully extracted, thus limiting the reasoning ability of the model. To fundamentally reduce the generation of artifacts and distortions, this paper proposes the Dual Perceptual Loss (DP Loss). First, we improve the perceptual feature extraction method so that it no longer only extracts single-type VGG features. In addition, a residual network (ResNet) feature that has a complementary relationship with the VGG feature can also be extracted. Then, we propose a dynamic weighting method to eliminate the magnitude difference between perceptual losses. Finally, to obtain the excellent effect of image reconstruction, enhanced super-resolution generative adversarial network (ESRGAN) with strong learning capability is used in this paper to adapt the complexity of DP Loss. The abundant experimental studies and evaluations are conducted on benchmark datasets. Results are encouraging and better than those previously reported on these datasets. The code is available at https://github.com/Sunny6-6-6/ESRGAN-DP.

An Arbitrary Scale Super-Resolution Approach for 3D MR Images via Implicit Neural Representation

High Resolution (HR) medical images provide rich anatomical structure details to facilitate early and accurate diagnosis. In magnetic resonance imaging (MRI), restricted by hardware capacity, scan time, and patient cooperation ability, isotropic 3-dimensional (3D) HR image acquisition typically requests long scan time and, results in small spatial coverage and low signal-to-noise ratio (SNR). Recent studies showed that, with deep convolutional neural networks, isotropic HR MR images could be recovered from low-resolution (LR) input via single image super-resolution (SISR) algorithms. However, most existing SISR methods tend to approach scale-specific projection between LR and HR images, thus these methods can only deal with fixed up-sampling rates. In this paper, we propose ArSSR, an Arbitrary Scale Super-Resolution approach for recovering 3D HR MR images. In the ArSSR model, the LR image and the HR image are represented using the same implicit neural voxel function with different sampling rates. Due to the continuity of the learned implicit function, a single ArSSR model is able to achieve arbitrary and infinite up-sampling rate reconstructions of HR images from any input LR image. Then the SR task is converted to approach the implicit voxel function via deep neural networks from a set of paired HR and LR training examples. The ArSSR model consists of an encoder network and a decoder network. Specifically, the convolutional encoder network is to extract feature maps from the LR input images and the fully-connected decoder network is to approximate the implicit voxel function. Experimental results on three datasets show that the ArSSR model can achieve state-of-the-art SR performance for 3D HR MR image reconstruction while using a single trained model to achieve arbitrary up-sampling scales.

Super-resolution acquisition and reconstruction for cone-beam SPECT with low-resolution detector

BACKGROUND AND OBJECTIVE

Single-photon emission computed tomography (SPECT) imaging, which provides information that reflects the human body's metabolic processes, has unique application value in disease diagnosis and efficacy evaluation. The imaging resolution of SPECT can be improved by exploiting high-performance detector hardware, but this exploitation generates high research and development costs. In addition, the inherent hardware structure of SPECT requires the use of a collimator, which limits the resolution in SPECT. The objective of this study is to propose a novel super-resolution (SR) reconstruction algorithm with two acquisition methods for cone-beam SPECT with low-resolution (LR) detector.

METHODS

A SR algorithm with two acquisition methods is proposed for cone-beam SPECT imaging in the projection domain. At each sampling angle, multi LR projections can be obtained by regularly moving the LR detector. For the two proposed acquisition methods, we develop a new SR reconstruction algorithm. Using our SR algorithm, a SR projection with the corresponding sampling angle can be obtained from multi LR projections via multi-iterations, and then, the SR SPECT image can be reconstructed. The peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), signal-to-noise ratio (SNR) and contrast recovery coefficient (CRC) are used to evaluate the final reconstruction quality.

RESULTS

The simulation results obtained under clean and noisy conditions verify the effectiveness of our SR algorithm. Three different phantoms are verified separately. 16 LR projections are obtained at each sampling angle, each with 32 × 32 bins. The high-resolution (HR) projection has 128 × 128 bins. The reconstruction result of the SR algorithm obtains an evaluation value that is almost the same as that of the HR reconstruction result. Our results indicate that the resolution of the resulting SPECT image is almost four times higher.

CONCLUSIONS

The authors develop a SR reconstruction algorithm with two acquisition methods for the cone-beam SPECT system. The simulation results obtained in clean and noisy environments prove that the SR algorithm has potential value, but it needs to be further tested on real equipment.

Realizing the Effective Detection of Tumor in Magnetic Resonance Imaging using Cluster-Sparse Assisted Super-Resolution

Recently, significant research has been done in Super-Resolution (SR) methods for augmenting the spatial resolution of the Magnetic Resonance (MR) images, which aids the physician in improved disease diagnoses. Single SR methods have drawbacks; they fail to capture self-similarity in non-local patches and are not robust to noise. To exploit the non-local self-similarity and intrinsic sparsity in MR images, this paper proposes the use of Cluster-Sparse Assisted Super-Resolution. This SR method effectively captures similarity in non-locally positioned patches by training on clusters of patches using a self-adaptive dictionary. This method of training also leads to better edge and texture detection. Experiments show that using Cluster-Sparse Assisted Super-Resolution for brain MR images results in enhanced detection of lesions leading to better diagnosis.

Reduction of Compression Artifacts Using a Densely Cascading Image Restoration Network

Since high quality realistic media are widely used in various computer vision applications, image compression is one of the essential technologies to enable real-time applications. Image compression generally causes undesired compression artifacts, such as blocking artifacts and ringing effects. In this study, we propose a densely cascading image restoration network (DCRN), which consists of an input layer, a densely cascading feature extractor, a channel attention block, and an output layer. The densely cascading feature extractor has three densely cascading (DC) blocks, and each DC block contains two convolutional layers, five dense layers, and a bottleneck layer. To optimize the proposed network architectures, we investigated the trade-off between quality enhancement and network complexity. Experimental results revealed that the proposed DCRN can achieve a better peak signal-to-noise ratio and structural similarity index measure for compressed joint photographic experts group (JPEG) images compared to the previous methods.

Multi-scale Xception based depthwise separable convolution for single image super-resolution

The main target of Single image super-resolution is to recover high-quality or high-resolution image from degraded version of low-quality or low-resolution image. Recently, deep learning-based approaches have achieved significant performance in image super-resolution tasks. However, existing approaches related with image super-resolution fail to use the features information of low-resolution images as well as do not recover the hierarchical features for the final reconstruction purpose. In this research work, we have proposed a new architecture inspired by ResNet and Xception networks, which enable a significant drop in the number of network parameters and improve the processing speed to obtain the SR results. We are compared our proposed algorithm with existing state-of-the-art algorithms and confirmed the great ability to construct HR images with fine, rich, and sharp texture details as well as edges. The experimental results validate that our proposed approach has robust performance compared to other popular techniques related to accuracy, speed, and visual quality.

Evaluation of super-resolution on 50 pancreatic cancer patients with real-time cine MRI from 0.35T MRgRT

MR-guided radiotherapy (MRgRT) systems provide excellent soft tissue imaging immediately prior to and in real time during radiation delivery for cancer treatment. However, 2D cine MRI often has limited spatial resolution due to high temporal resolution. This work applies a super resolution machine learning framework to 3.5 mm pixel edge length, low resolution (LR), sagittal 2D cine MRI images acquired on a MRgRT system to generate 0.9 mm pixel edge length, super resolution (SR), images originally acquired at 4 frames per second (FPS). LR images were collected from 50 pancreatic cancer patients treated on a ViewRay MR-LINAC. SR images were evaluated using three methods. 1) The first method utilized intrinsic image quality metrics for evaluation. 2) The second used relative metrics including edge detection and structural similarity index (SSIM). 3) Finally, automatically generated tumor contours were created on both low resolution and super resolution images to evaluate target delineation and compared with DICE and SSIM. Intrinsic image quality metrics all had statistically significant improvements for SR images versus LR images, with mean (±1 SD) BRISQUE scores of 29.65 ± 2.98 and 42.48 ± 0.98 for SR and LR, respectively. SR images showed good agreement with LR images in SSIM evaluation, indicating there was not significant distortion of the images. Comparison of LR and SR images with paired high resolution (HR) 3D images showed that SR images had a mean (±1 SD) SSIM value of 0.633 ± 0.063 and LR a value of 0.587 ± 0.067 (p ≪ 0.05). Contours generated on SR images were also more robust to noise addition than those generated on LR images. This study shows that super resolution with a machine learning framework can generate high spatial resolution images from 4fps low spatial resolution cine MRI acquired on the ViewRay MR-LINAC while maintaining tumor contour quality and without significant acquisition or post processing delay.

Super Resolution of Magnetic Resonance Images

In this work, novel denoising and super resolution (SR) approaches for magnetic resonance (MR) images are addressed, and are integrated in a unified framework, which do not require example low resolution (LR)/high resolution (HR)/cross-modality/noise-free images and prior information of noise–noise variance. The proposed method categorizes the patches as either smooth or textured and then denoises them by deploying different denoising strategies for efficient denoising. The denoising algorithm is integrated into the SR approach, which uses a gradient profile-based constraint in a sparse representation-based framework to improve the resolution of MR images with reduced smearing of image details. This constraint regularizes the estimation of HR images such that the estimated HR image has gradient profiles similar to the gradient profiles of the original HR image. For this, the gradient profile sharpness (GPS) values of an unknown HR image are estimated using an approximated piece-wise linear relation among GPS values of LR and upsampled LR images. The experiments are performed on three different publicly available datasets. The proposed SR approach outperforms the existing unsupervised SR approach addressed for real MR images that exploits low rank and total variation (LRTV) regularization, by an average peak signal to noise ratio (PSNR) of 0.73 dB and 0.38 dB for upsampling factors 2 and 3, respectively. For the super resolution of noisy real MR images (degraded with 2% noise), the proposed approach outperforms the LRTV approach by an average PSNR of 0.54 dB and 0.46 dB for upsampling factors 2 and 3, respectively. The qualitative analysis is shown for real MR images from healthy subjects and subjects with Alzheimer’s disease and structural deformity, i.e., cavernoma.

Enhanced Single Image Super Resolution Method Using Lightweight Multi-Scale Channel Dense Network

Super resolution (SR) enables to generate a high-resolution (HR) image from one or more low-resolution (LR) images. Since a variety of CNN models have been recently studied in the areas of computer vision, these approaches have been combined with SR in order to provide higher image restoration. In this paper, we propose a lightweight CNN-based SR method, named multi-scale channel dense network (MCDN). In order to design the proposed network, we extracted the training images from the DIVerse 2K (DIV2K) dataset and investigated the trade-off between the SR accuracy and the network complexity. The experimental results show that the proposed method can significantly reduce the network complexity, such as the number of network parameters and total memory capacity, while maintaining slightly better or similar perceptual quality compared to the previous methods.

Variational multi-task MRI reconstruction: Joint reconstruction, registration and super-resolution

Motion degradation is a central problem in Magnetic Resonance Imaging (MRI). This work addresses the problem of how to obtain higher quality, super-resolved motion-free reconstructions from highly undersampled MRI data. In this work, we present for the first time a variational multi-task framework that allows joining three relevant tasks in MRI: reconstruction, registration and super-resolution. Our framework takes a set of multiple undersampled MR acquisitions corrupted by motion into a novel multi-task optimisation model, which is composed of an L² fidelity term that allows sharing representation between tasks, super-resolution foundations and hyperelastic deformations to model biological tissue behaviors. We demonstrate that this combination yields significant improvements over sequential models and other bi-task methods. Our results exhibit fine details and compensate for motion producing sharp and highly textured images compared to state of the art methods while keeping low CPU time. Our improvements are appraised on both clinical assessment and statistical analysis.

High-fidelity, accelerated whole-brain submillimeter in vivo diffusion MRI using gSlider-spherical ridgelets (gSlider-SR)

PURPOSE

To develop an accelerated, robust, and accurate diffusion MRI acquisition and reconstruction technique for submillimeter whole human brain in vivo scan on a clinical scanner.

METHODS

We extend the ultra-high resolution diffusion MRI acquisition technique, gSlider, by allowing undersampling in q-space and radiofrequency (RF)-encoding space, thereby dramatically reducing the total acquisition time of conventional gSlider. The novel method, termed gSlider-SR, compensates for the lack of acquired information by exploiting redundancy in the dMRI data using a basis of spherical ridgelets (SR), while simultaneously enhancing the signal-to-noise ratio. Using Monte Carlo simulation with realistic noise levels and several acquisitions of in vivo human brain dMRI data (acquired on a Siemens Prisma 3T scanner), we demonstrate the efficacy of our method using several quantitative metrics.

RESULTS

For high-resolution dMRI data with realistic noise levels (synthetically added), we show that gSlider-SR can reconstruct high-quality dMRI data at different acceleration factors preserving both signal and angular information. With in vivo data, we demonstrate that gSlider-SR can accurately reconstruct 860 μm diffusion MRI data (64 diffusion directions at b = 2000 s / mm 2 ), at comparable quality as that obtained with conventional gSlider with four averages, thereby providing an eight-fold reduction in scan time (from 1 hour 20 to 10 minutes).

CONCLUSIONS

gSlider-SR enables whole-brain high angular resolution dMRI at a submillimeter spatial resolution with a dramatically reduced acquisition time, making it feasible to use the proposed scheme on existing clinical scanners.

Connectome mapping with edge density imaging differentiates pediatric mild traumatic brain injury from typically developing controls: proof of concept

BACKGROUND

Although acute neurologic impairment might be transient, other long-term effects can be observed with mild traumatic brain injury. However, when pediatric patients with mild traumatic brain injury present for medical care, conventional imaging with CT and MR imaging often does not reveal abnormalities.

OBJECTIVE

To determine whether edge density imaging can separate pediatric mild traumatic brain injury from typically developing controls.

MATERIALS AND METHODS

Subjects were recruited as part of the "Therapeutic Resources for Attention Improvement using Neuroimaging in Traumatic Brain Injury" (TRAIN-TBI) study. We included 24 adolescents (χ=14.1 years of age, σ=1.6 years, range 10-16 years), 14 with mild traumatic brain injury (TBI) and 10 typically developing controls. Neurocognitive assessments included the pediatric version of the California Verbal Learning Test (CVLT) and the Attention Network Task (ANT). Diffusion MR imaging was acquired on a 3-tesla (T) scanner. Edge density images were computed utilizing fiber tractography. Principal component analysis (PCA) and support vector machines (SVM) were used in an exploratory analysis to separate mild TBI and control groups. The diagnostic accuracy of edge density imaging, neurocognitive tests, and fractional anisotropy (FA) from diffusion tensor imaging (DTI) was computed with two-sample t-tests and receiver operating characteristic (ROC) metrics.

RESULTS

Support vector machine-principal component analysis of edge density imaging maps identified three white matter regions distinguishing pediatric mild TBI from controls. The bilateral tapetum, sagittal stratum, and callosal splenium identified mild TBI subjects with sensitivity of 79% and specificity of 100%. Accuracy from the area under the ROC curve (AUC) was 94%. Neurocognitive testing provided an AUC of 61% (CVLT) and 71% (ANT). Fractional anisotropy yielded an AUC of 48%.

CONCLUSION

In this proof-of-concept study, we show that edge density imaging is a new form of connectome mapping that provides better diagnostic delineation between pediatric mild TBI and healthy controls than DTI or neurocognitive assessments of memory or attention.

A Direct Drive Parallel Plane Piezoelectric Needle Positioning Robot for MRI Guided Intraspinal Injection

Recent developments in the field of cellular therapeutics have indicated the potential of stem cell injections directly to the spinal cord. Injections require either open surgery or a Magnetic Resonance Imaging (MRI) guided injection. Needle positioning during MRI imaging is a significant hurdle to direct spinal injection, as the small target region and interlaminar space require high positioning accuracy.

OBJECTIVE

To improve both the procedure time and positioning accuracy, an MRI guided robotic needle positioning system is developed.

METHODS

The robot uses linear piezoelectric motors to directly drive a parallel plane positioning mechanism. Feedback is provided through MRI during the orientation procedure. Both accuracy and repeatability of the robot are characterized.

RESULTS

This system is found to be capable of repeatability below 51 μm. Needle endpoint error is limited by imaging modality, but is validated to 156 μm.

CONCLUSION

The reported robot and MRI image feedback system is capable of repeatable and accurate needle guide positioning.

SIGNIFICANCE

This high accuracy will result in a significant improvement to the workflow of spinal injection procedures.

Deep Learning Single-Frame and Multiframe Super-Resolution for Cardiac MRI

Background Cardiac MRI is limited by long acquisition times, yet faster acquisition of smaller-matrix images reduces spatial detail. Deep learning (DL) might enable both faster acquisition and higher spatial detail via super-resolution. Purpose To explore the feasibility of using DL to enhance spatial detail from small-matrix MRI acquisitions and evaluate its performance against that of conventional image upscaling methods. Materials and Methods Short-axis cine cardiac MRI examinations performed between January 2012 and December 2018 at one institution were retrospectively collected for algorithm development and testing. Convolutional neural networks (CNNs), a form of DL, were trained to perform super resolution in image space by using synthetically generated low-resolution data. There were 70%, 20%, and 10% of examinations allocated to training, validation, and test sets, respectively. CNNs were compared against bicubic interpolation and Fourier-based zero padding by calculating the structural similarity index (SSIM) between high-resolution ground truth and each upscaling method. Means and standard deviations of the SSIM were reported, and statistical significance was determined by using the Wilcoxon signed-rank test. For evaluation of clinical performance, left ventricular volumes were measured, and statistical significance was determined by using the paired Student t test. Results For CNN training and retrospective analysis, 400 MRI scans from 367 patients (mean age, 48 years ± 18; 214 men) were included. All CNNs outperformed zero padding and bicubic interpolation at upsampling factors from two to 64 (P < .001). CNNs outperformed zero padding on more than 99.2% of slices (9828 of 9907). In addition, 10 patients (mean age, 51 years ± 22; seven men) were prospectively recruited for super-resolution MRI. Super-resolved low-resolution images yielded left ventricular volumes comparable to those from full-resolution images (P > .05), and super-resolved full-resolution images appeared to further enhance anatomic detail. Conclusion Deep learning outperformed conventional upscaling methods and recovered high-frequency spatial information. Although training was performed only on short-axis cardiac MRI examinations, the proposed strategy appeared to improve quality in other imaging planes. © RSNA, 2020 Online supplemental material is available for this article.

Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry

The deep convolutional neural network (DCNN) has recently been applied to the highly challenging and ill-posed problem of single image super-resolution (SISR), which aims to predict high-resolution (HR) images from their corresponding low-resolution (LR) images. In many remote sensing (RS) applications, spatial resolution of the aerial or satellite imagery has a great impact on the accuracy and reliability of information extracted from the images. In this study, the potential of a DCNN-based SISR model, called enhanced super-resolution generative adversarial network (ESRGAN), to predict the spatial information degraded or lost in a hyper-spatial resolution unmanned aircraft system (UAS) RGB image set is investigated. ESRGAN model is trained over a limited number of original HR (50 out of 450 total images) and virtually-generated LR UAS images by downsampling the original HR images using a bicubic kernel with a factor × 4 . Quantitative and qualitative assessments of super-resolved images using standard image quality measures (IQMs) confirm that the DCNN-based SISR approach can be successfully applied on LR UAS imagery for spatial resolution enhancement. The performance of DCNN-based SISR approach for the UAS image set closely approximates performances reported on standard SISR image sets with mean peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) index values of around 28 dB and 0.85 dB, respectively. Furthermore, by exploiting the rigorous Structure-from-Motion (SfM) photogrammetry procedure, an accurate task-based IQM for evaluating the quality of the super-resolved images is carried out. Results verify that the interior and exterior imaging geometry, which are extremely important for extracting highly accurate spatial information from UAS imagery in photogrammetric applications, can be accurately retrieved from a super-resolved image set. The number of corresponding keypoints and dense points generated from the SfM photogrammetry process are about 6 and 17 times more than those extracted from the corresponding LR image set, respectively.

Deep Inception-Residual Laplacian Pyramid Networks for Accurate Single-Image Super-Resolution

With exploiting contextual information over large image regions in an efficient way, the deep convolutional neural network has shown an impressive performance for single-image super-resolution (SR). In this paper, we propose a new deep convolutional network by cascading multiple well-designed inception-residual blocks within the deep Laplacian pyramid framework to progressively restore the missing high-frequency details in the low-resolution images. By optimizing our network structure, the trainable depth of our proposed network gains a significant improvement, which in turn improves super-resolving accuracy. However, the saturation and degradation of training accuracy remains a critical problem. With regard to this, we propose an effective two-stage training strategy, in which we first use the images downsampled from the ground-truth high-resolution (HR) images to pretrain the inception-residual blocks on each pyramid level with an extremely high learning rate enabled by gradient clipping, and then the original ground-truth HR images are used to fine-tune all the pretrained inception-residual blocks for obtaining our final SR models. Furthermore, we present a new loss function operating in both image space and local rank space to optimize our network for exploiting the contextual information among different output components. Extensive experiments on benchmark data sets validate that the proposed method outperforms the existing state-of-the-art SR methods in terms of the objective evaluation as well as the visual quality.

Improved compressed sensing and super-resolution of cardiac diffusion MRI with structure-guided total variation

Purpose

Structure‐guided total variation is a recently introduced prior that allows reconstruction of images using knowledge of the location and orientation of edges in a reference image. In this work, we demonstrate the advantages of a variant of structure‐guided total variation known as directional total variation (DTV), over traditional total variation (TV), in the context of compressed‐sensing reconstruction and super‐resolution.

Methods

We compared TV and DTV in retrospectively undersampled ex vivo diffusion tensor imaging and diffusion spectrum imaging data from healthy, sham, and hypertrophic rat hearts.

Results

In compressed sensing at an undersampling factor of 8, the RMS error of mean diffusivity and fractional anisotropy relative to the fully sampled ground truth were 44% and 20% lower in DTV compared with TV. In super‐resolution, these values were 29% and 14%, respectively. Similarly, we observed improvements in helix angle, transverse angle, sheetlet elevation, and sheetlet azimuth. The RMS error of the diffusion kurtosis in the undersampled data relative to the ground truth was uniformly lower (22% on average) with DTV compared to TV.

Conclusion

Acquiring one fully sampled non‐diffusion‐weighted image and 10 diffusion‐weighted images at 8× undersampling would result in an 80% net reduction in data needed. We demonstrate efficacy of the DTV algorithm over TV in reducing data sampling requirements, which can be translated into higher apparent resolution and potentially shorter scan times. This method would be equally applicable in diffusion MRI applications outside the heart.

Channel Splitting Network for Single MR Image Super-Resolution

High resolution magnetic resonance (MR) imaging is desirable in many clinical applications due to its contribution to more accurate subsequent analyses and early clinical diagnoses. Single image super-resolution (SISR) is an effective and cost efficient alternative technique to improve the spatial resolution of MR images. In the past few years, SISR methods based on deep learning techniques, especially convolutional neural networks (CNNs), have achieved the state-of-the-art performance on natural images. However, the information is gradually weakened and training becomes increasingly difficult as the network deepens. The problem is more serious for medical images because lacking high quality and effective training samples makes deep models prone to underfitting or overfitting. Nevertheless, many current models treat the hierarchical features on different channels equivalently, which is not helpful for the models to deal with the hierarchical features discriminatively and targetedly. To this end, we present a novel channel splitting network (CSN) to ease the representational burden of deep models. The proposed CSN model divides the hierarchical features into two branches, i.e., residual branch and dense branch, with different information transmissions. The residual branch is able to promote feature reuse, while the dense branch is beneficial to the exploration of new features. Besides, we also adopt the merge-and-run mapping to facilitate information integration between different branches. The extensive experiments on various MR images, including proton density (PD), T1, and T2 images, show that the proposed CSN model achieves superior performance over other state-of-the-art SISR methods.

Changes of fractional anisotropy and RGMa in crossed cerebellar diaschisis induced by middle cerebral artery occlusion

Crossed cerebellar diaschisis (CCD) is the phenomenon of hypoperfusion and hypometabolism of the contralateral cerebellar hemisphere caused by dysfunction of the associated supratentorial region. The aim of the present study was to analyze the changes in fractional anisotropy (FA) in CCD induced by middle cerebral artery occlusion (MCAO) using magnetic resonance-diffusion tensor imaging (MR-DTI). Furthermore, the role of repulsive guidance molecule a (RGMa) in CCD was assessed by measuring RGMa expression using histochemical analysis. In the present study, the cerebellar hemisphere was serially scanned with T2-weighted, serial diffusion-weighted and diffusion tensor (DT) imaging using a 3.0T GE Signa HDxt Scanner to analyze the changes in FA over 72 h. Subsequently, immunohistochemistry analyses of the corresponding cerebellar hemisphere sections were performed to assess the expression of RGMa. Results indicated that FA of both sides of the cerebellar hemisphere, particularly that of the contralateral cerebellar hemisphere (right side) derived from DTI, was reduced during the 72-h time period following MCAO, and the decrease was maximal and statistically significant at 12 h (P<0.05). Immunohistochemistry analysis revealed a significant increase in the expression of RGMa protein in the affected region of the contralateral cerebellar hemisphere (right side) at 24 h following MCAO injury (P<0.05). Furthermore, the expression of RGMa and FA was negatively correlated in MCAO (P<0.05). The results suggest that MR-DTI is an important assessment to evaluate changes of FA in CCD induced by MCAO. Furthermore, the present results suggest that RGMa, which was negatively correlated with FA in MCAO rats, may serve an important role in CCD.

Multi-Scale Inception Based Super-Resolution Using Deep Learning Approach

Single image super-resolution (SISR) aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) image. In order to address the SISR problem, recently, deep convolutional neural networks (CNNs) have achieved remarkable progress in terms of accuracy and efficiency. In this paper, an innovative technique, namely a multi-scale inception-based super-resolution (SR) using deep learning approach, or MSISRD, was proposed for fast and accurate reconstruction of SISR. The proposed network employs the deconvolution layer to upsample the LR image to the desired HR image. The proposed method is in contrast to existing approaches that use the interpolation techniques to upscale the LR image. Primarily, interpolation techniques are not designed for this purpose, which results in the creation of undesired noise in the model. Moreover, the existing methods mainly focus on the shallow network or stacking multiple layers in the model with the aim of creating a deeper network architecture. The technique based on the aforementioned design creates the vanishing gradients problem during the training and increases the computational cost of the model. Our proposed method does not use any hand-designed pre-processing steps, such as the bicubic interpolation technique. Furthermore, an asymmetric convolution block is employed to reduce the number of parameters, in addition to the inception block adopted from GoogLeNet, to reconstruct the multiscale information. Experimental results demonstrate that the proposed model exhibits an enhanced performance compared to twelve state-of-the-art methods in terms of the average peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) with a reduced number of parameters for the scale factor of 2 × , 4 × , and 8 × .

Multifold Acceleration of Diffusion MRI via Deep Learning Reconstruction from Slice-Undersampled Data

Diffusion MRI (dMRI), while powerful for characterization of tissue microstructure, suffers from long acquisition time. In this paper, we present a method for effective diffusion MRI reconstruction from slice-undersampled data. Instead of full diffusion-weighted (DW) image volumes, only a subsample of equally-spaced slices need to be acquired. We show that complementary information from DW volumes corresponding to different diffusion wavevectors can be harnessed using graph convolutional neural networks for reconstruction of the full DW volumes. The experimental results indicate a high acceleration factor of up to 5 can be achieved with minimal information loss.

Generative models: an upcoming innovation in musculoskeletal radiology? A preliminary test in spine imaging

Background

Deep learning is a ground-breaking technology that is revolutionising many research and industrial fields. Generative models are recently gaining interest. Here, we investigate their potential, namely conditional generative adversarial networks, in the field of magnetic resonance imaging (MRI) of the spine, by performing clinically relevant benchmark cases.

Methods

First, the enhancement of the resolution of T2-weighted (T2W) images (super-resolution) was tested. Then, automated image-to-image translation was tested in the following tasks: (1) from T1-weighted to T2W images of the lumbar spine and (2) vice versa; (3) from T2W to short time inversion-recovery (STIR) images; (4) from T2W to turbo inversion recovery magnitude (TIRM) images; (5) from sagittal standing x-ray projections to T2W images. Clinical and quantitative assessments of the outputs by means of image quality metrics were performed. The training of the models was performed on MRI and x-ray images from 989 patients.

Results

The performance of the models was generally positive and promising, but with several limitations. The number of disc protrusions or herniations showed good concordance (κ = 0.691) between native and super-resolution images. Moderate-to-excellent concordance was found when translating T2W to STIR and TIRM images (κ ≥ 0.842 regarding disc degeneration), while the agreement was poor when translating x-ray to T2W images.

Conclusions

Conditional generative adversarial networks are able to generate perceptually convincing synthetic images of the spine in super-resolution and image-to-image translation tasks. Taking into account the limitations of the study, deep learning-based generative methods showed the potential to be an upcoming innovation in musculoskeletal radiology.

Multi-Branch Deep Residual Network for Single Image Super-Resolution

Recently, algorithms based on the deep neural networks and residual networks have been applied for super-resolution and exhibited excellent performance. In this paper, a multi-branch deep residual network for single image super-resolution (MRSR) is proposed. In the network, we adopt a multi-branch network framework and further optimize the structure of residual network. By using residual blocks and filters reasonably, the model size is greatly expanded while the stable training is also guaranteed. Besides, a perceptual evaluation function, which contains three parts of loss, is proposed. The experiment results show that the evaluation function provides great support for the quality of reconstruction and the competitive performance. The proposed method mainly uses three steps of feature extraction, mapping, and reconstruction to complete the super-resolution reconstruction and shows superior performance than other state-of-the-art super-resolution methods on benchmark datasets.

Fusing multi-scale information in convolution network for MR image super-resolution reconstruction

BACKGROUND

Magnetic resonance (MR) images are usually limited by low spatial resolution, which leads to errors in post-processing procedures. Recently, learning-based super-resolution methods, such as sparse coding and super-resolution convolution neural network, have achieved promising reconstruction results in scene images. However, these methods remain insufficient for recovering detailed information from low-resolution MR images due to the limited size of training dataset.

METHODS

To investigate the different edge responses using different convolution kernel sizes, this study employs a multi-scale fusion convolution network (MFCN) to perform super-resolution for MRI images. Unlike traditional convolution networks that simply stack several convolution layers, the proposed network is stacked by multi-scale fusion units (MFUs). Each MFU consists of a main path and some sub-paths and finally fuses all paths within the fusion layer.

RESULTS

We discussed our experimental network parameters setting using simulated data to achieve trade-offs between the reconstruction performance and computational efficiency. We also conducted super-resolution reconstruction experiments using real datasets of MR brain images and demonstrated that the proposed MFCN has achieved a remarkable improvement in recovering detailed information from MR images and outperforms state-of-the-art methods.

CONCLUSIONS

We have proposed a multi-scale fusion convolution network based on MFUs which extracts different scales features to restore the detail information. The structure of the MFU is helpful for extracting multi-scale information and making full-use of prior knowledge from a few training samples to enhance the spatial resolution.

Image Super-resolution with Parametric Sparse Model Learning

Recovering a high-resolution (HR) image from its low-resolution (LR) version is an ill-posed inverse problem. Learning accurate prior of HR images is of great importance to solve this inverse problem. Existing super-resolution (SR) methods either learn a non-parametric image prior from training data (a large set of LR/HR patch pairs) or estimate a parametric prior from the LR image analytically. Both methods have their limitations: the former lacks flexibility when dealing with different SR settings; while the latter often fails to adapt to spatially varying image structures. In this paper, we propose to take a hybrid approach toward image SR by combining those two lines of ideas - that is, a parametric sparse prior of HR images is learned from the training set as well as the input LR image. By exploiting the strengths of both worlds, we can more accurately recover the sparse codes and therefore HR image patches than conventional sparse coding approaches. Experimental results show that the proposed hybrid SR method significantly outperforms existing model-based SR methods and is highly competitive to current state-of-the-art learning-based SR methods in terms of both subjective and objective image qualities.

Non-local diffusion-weighted image super-resolution using collaborative joint information

Due to the clinical durable scanning time and other physical constraints, the spatial resolution of diffusion-weighted magnetic resonance imaging (DWI) is highly limited. Using a post-processing method to improve the resolution of DWI holds the potential to improve the investigation of smaller white-matter structures and to reduce partial volume effects. In the present study, a novel non-local mean super-resolution method was proposed to increase the spatial resolution of DWI datasets. Based on a non-local strategy, joint information from the adjacent scanning directions was taken advantage of through the implementation of a novel weighting scheme. Besides this, an efficient rotationally invariant similarity measure was introduced for further improvement of high-resolution image reconstruction and computational efficiency. Quantitative and qualitative comparisons in synthetic and real DWI datasets demonstrated that the proposed method significantly enhanced the resolution of DWI, and is thus beneficial in improving the estimation accuracy for diffusion tensor imaging as well as high-angular resolution diffusion imaging.

Super-resolution intracranial quiescent interval slice-selective magnetic resonance angiography

PURPOSE

To evaluate the combination of nonenhanced quiescent-interval slice-selective (QISS) magnetic resonance angiography (MRA) with super-resolution reconstruction for portraying the intracranial arteries.

METHODS

The intracranial arteries of seven volunteers were imaged at 3T using QISS MRA acquired with a flow-compensated fast low-angle shot (FLASH) readout and thin overlapping slices. The impacts of super-resolution reconstruction and various acquisition parameters on the delineation of intracranial arteries were quantified using four metrics: arterial-to-background contrast-to-noise ratio (CNR), arterial-to-background contrast, arterial sharpness, and arterial full-width-at-half-maximum (FWHM). Three-dimensional time-of-flight (TOF) MRA was also acquired.

RESULTS

For similar voxel sizes, QISS MRA displayed the intracranial arteries with an arterial-to-background contrast that exceeded 3D TOF MRA by 59-84%, depending on the k-space sampling trajectory (P < 0.001). Super-resolution reconstruction improved CNR, contrast, and sharpness, while reducing arterial FWHM (P < 0.001). Cardiac triggering provided minimal benefits, while Cartesian sampling provided higher CNR than radial sampling for multishot QISS (P < 0.05). Scan time for a complete intracranial MRA was <90 s using an ungated single-shot QISS acquisition.

CONCLUSION

Thin, overlapping-slice QISS leveraging super-resolution reconstruction is a flexible approach for intracranial MRA that provides competitive image quality to standard-of-care 3D TOF, with the potential for reduced sensitivity to in-plane flow saturation and motion artifacts. Magn Reson Med 79:683-691, 2018. © 2017 International Society for Magnetic Resonance in Medicine.

A New Sparse Representation Framework for Reconstruction of an Isotropic High Spatial Resolution MR Volume From Orthogonal Anisotropic Resolution Scans

In magnetic resonance (MR), hardware limitations, scan time constraints, and patient movement often result in the acquisition of anisotropic 3-D MR images with limited spatial resolution in the out-of-plane views. Our goal is to construct an isotropic high-resolution (HR) 3-D MR image through upsampling and fusion of orthogonal anisotropic input scans. We propose a multiframe super-resolution (SR) reconstruction technique based on sparse representation of MR images. Our proposed algorithm exploits the correspondence between the HR slices and the low-resolution (LR) sections of the orthogonal input scans as well as the self-similarity of each input scan to train pairs of overcomplete dictionaries that are used in a sparse-land local model to upsample the input scans. The upsampled images are then combined using wavelet fusion and error backprojection to reconstruct an image. Features are learned from the data and no extra training set is needed. Qualitative and quantitative analyses were conducted to evaluate the proposed algorithm using simulated and clinical MR scans. Experimental results show that the proposed algorithm achieves promising results in terms of peak signal-to-noise ratio, structural similarity image index, intensity profiles, and visualization of small structures obscured in the LR imaging process due to partial volume effects. Our novel SR algorithm outperforms the nonlocal means (NLM) method using self-similarity, NLM method using self-similarity and image prior, self-training dictionary learning-based SR method, averaging of upsampled scans, and the wavelet fusion method. Our SR algorithm can reduce through-plane partial volume artifact by combining multiple orthogonal MR scans, and thus can potentially improve medical image analysis, research, and clinical diagnosis.

7T-guided super-resolution of 3T MRI

PURPOSE

High-resolution MR images can depict rich details of brain anatomical structures and show subtle changes in longitudinal data. 7T MRI scanners can acquire MR images with higher resolution and better tissue contrast than the routine 3T MRI scanners. However, 7T MRI scanners are currently more expensive and less available in clinical and research centers. To this end, we propose a method to generate super-resolution 3T MRI that resembles 7T MRI, which is called as 7T-like MR image in this paper.

METHODS

First, we propose a mapping from 3T MRI to 7T MRI space, using regression random forest. The mapped 3T MR images serve as intermediate results with similar appearance as 7T MR images. Second, we predict the final higher resolution 7T-like MR images based on sparse representation, using paired local dictionaries for both the mapped 3T MR images and 7T MR images.

RESULTS

Based on 15 subjects with both 3T and 7T MR images, the predicted 7T-like MR images by our method can best match the ground-truth 7T MR images, compared to other methods. Meanwhile, the experiment on brain tissue segmentation shows that our 7T-like MR images lead to the highest accuracy in the segmentation of WM, GM, and CSF brain tissues, compared to segmentations of 3T MR images as well as the reconstructed 7T-like MR images by other methods.

CONCLUSIONS

We propose a novel method for prediction of high-resolution 7T-like MR images from low-resolution 3T MR images. Our predicted 7T-like MR images demonstrate better spatial resolution compared to 3T MR images, as well as prediction results by other comparison methods. Such high-quality 7T-like MR images could better facilitate disease diagnosis and intervention.

Multi-contrast brain magnetic resonance image super-resolution using the local weight similarity

Background

Low-resolution images may be acquired in magnetic resonance imaging (MRI) due to limited data acquisition time or other physical constraints, and their resolutions can be improved with super-resolution methods. Since MRI can offer images of an object with different contrasts, e.g., T1-weighted or T2-weighted, the shared information between inter-contrast images can be used to benefit super-resolution.

Methods

In this study, an MRI image super-resolution approach to enhance in-plane resolution is proposed by exploring the statistical information estimated from another contrast MRI image that shares similar anatomical structures. We assume some edge structures are shown both in T1-weighted and T2-weighted MRI brain images acquired of the same subject, and the proposed approach aims to recover such kind of structures to generate a high-resolution image from its low-resolution counterpart.

Results

The statistical information produces a local weight of image that are found to be nearly invariant to the image contrast and thus this weight can be used to transfer the shared information from one contrast to another. We analyze this property with comprehensive mathematics as well as numerical experiments.

Conclusion

Experimental results demonstrate that the image quality of low-resolution images can be remarkably improved with the proposed method if this weight is borrowed from a high resolution image with another contrast.

Graphical Abstract

Multi-contrast MRI Image Super-resolution with Contrast-invariant Regression Weights

Electronic supplementary material

The online version of this article (doi:10.1186/s12880-016-0176-2) contains supplementary material, which is available to authorized users.

Diffusion-Weighted Images Superresolution Using High-Order SVD

The spatial resolution of diffusion-weighted imaging (DWI) is limited by several physical and clinical considerations, such as practical scanning times. Interpolation methods, which are widely used to enhance resolution, often result in blurred edges. Advanced superresolution scanning acquires images with specific protocols and long acquisition times. In this paper, we propose a novel single image superresolution (SR) method which introduces high-order SVD (HOSVD) to regularize the patch-based SR framework on DWI datasets. The proposed method was implemented on an adaptive basis which ensured a more accurate reconstruction of high-resolution DWI datasets. Meanwhile, the intrinsic dimensional decreasing property of HOSVD is also beneficial for reducing the computational burden. Experimental results from both synthetic and real DWI datasets demonstrate that the proposed method enhances the details in reconstructed high-resolution DWI datasets and outperforms conventional techniques such as interpolation methods and nonlocal upsampling.

A joint compressed-sensing and super-resolution approach for very high-resolution diffusion imaging

Diffusion MRI (dMRI) can provide invaluable information about the structure of different tissue types in the brain. Standard dMRI acquisitions facilitate a proper analysis (e.g. tracing) of medium-to-large white matter bundles. However, smaller fiber bundles connecting very small cortical or sub-cortical regions cannot be traced accurately in images with large voxel sizes. Yet, the ability to trace such fiber bundles is critical for several applications such as deep brain stimulation and neurosurgery. In this work, we propose a novel acquisition and reconstruction scheme for obtaining high spatial resolution dMRI images using multiple low resolution (LR) images, which is effective in reducing acquisition time while improving the signal-to-noise ratio (SNR). The proposed method called compressed-sensing super resolution reconstruction (CS-SRR), uses multiple overlapping thick-slice dMRI volumes that are under-sampled in q-space to reconstruct diffusion signal with complex orientations. The proposed method combines the twin concepts of compressed sensing and super-resolution to model the diffusion signal (at a given b-value) in a basis of spherical ridgelets with total-variation (TV) regularization to account for signal correlation in neighboring voxels. A computationally efficient algorithm based on the alternating direction method of multipliers (ADMM) is introduced for solving the CS-SRR problem. The performance of the proposed method is quantitatively evaluated on several in-vivo human data sets including a true SRR scenario. Our experimental results demonstrate that the proposed method can be used for reconstructing sub-millimeter super resolution dMRI data with very good data fidelity in clinically feasible acquisition time.

Investigation of resolution effects using a specialized diffusion tensor phantom

PURPOSE

The clinical potential of the diffusion imaging-based analysis of fine brain structures such as fornix or cingulum is high due to the central role of these structures in psychiatric diseases. However, the quantification of diffusion parameters in fine structures is especially prone to partial volume effects (PVEs).

METHODS

In this study, a phantom for the investigation of PVEs and their influence on diffusion parameters in fine structures of different diameter is presented. The phantom is produced by winding wet polyester fibers onto a spindle. The resulting fiber strands have well defined square cross-sections of 1-25 mm(2) and provide a homogeneous and high fractional anisotropy (FA ≈ 0.9).

RESULTS

Several PVEs are demonstrated and analyzed. It is shown that inferred results such as the fiber geometry and diffusion parameters strongly depend on the relative position of the structure of interest to the voxel-grid. Several implications of PVEs on post-processing methods such as Tract-based Spatial Statistics and fiber tractography are demonstrated.

CONCLUSION

These results show that the handling of PVEs in common post-processing tasks can be problematic, and that the presented phantom provides a valuable tool for the improvement and evaluation of these effects.