Real-time single-pixel video imaging with Fourier domain regularization

Adaptive real-time single-pixel imaging

For most imaging systems, there is a trade-off between spatial resolution, temporal resolution, and signal-to-noise ratio. Such a trade-off is particularly severe in single-pixel imaging systems, given the limited throughput of the only one available pixel. Here we report a real-time single-pixel imaging method that can adaptively balance the spatial resolution, temporal resolution, and signal-to-noise ratio of the imaging system according to the changes in the target scene. When scene changes are detected, the dynamic imaging mode will be activated. The temporal resolution will be given high priority and real-time single-pixel imaging will be conducted at a video frame rate (30 frames/s) to visualize the object motion. When no scene changes are detected, the static imaging mode will be activated. The spatial resolution and the signal-to-noise ratio will be progressively built up to resolve fine structures and to improve image quality. The proposed method not only adds practicability to single-pixel imaging, but also generates a new, to the best of our knowledge, insight in data redundancy reduction and information capacity improvement for other computational imaging schemes.

High-resolution iterative reconstruction at extremely low sampling rate for Fourier single-pixel imaging via diffusion model

The trade-off between imaging efficiency and imaging quality has always been encountered by Fourier single-pixel imaging (FSPI). To achieve high-resolution imaging, the increase in the number of measurements is necessitated, resulting in a reduction of imaging efficiency. Here, a novel high-quality reconstruction method for FSPI imaging via diffusion model was proposed. A score-based diffusion model is designed to learn prior information of the data distribution. The real-sampled low-frequency Fourier spectrum of the target is employed as a consistency term to iteratively constrain the model in conjunction with the learned prior information, achieving high-resolution reconstruction at extremely low sampling rates. The performance of the proposed method is evaluated by simulations and experiments. The results show that the proposed method has achieved superior quality compared with the traditional FSPI method and the U-Net method. Especially at the extremely low sampling rate (e.g., 1%), an approximately 241% improvement in edge intensity-based score was achieved by the proposed method for the coin experiment, compared with the traditional FSPI method. The method has the potential to achieve high-resolution imaging without compromising imaging speed, which will further expanding the application scope of FSPI in practical scenarios.

Spatially encoded hyperspectral compressive microscope for ultrabroadband VIS/NIR hyperspectral imaging

Hyperspectral imaging (HSI) has become a valuable tool in sample characterization in various scientific fields. While many approaches have been tested, specific applications and technology usually lead to only a narrow part of the spectrum being studied. We demonstrate the use of a broadband HSI setup based on compressed sensing capable of capturing data in visible (VIS), near-infrared (NIR), and short-wave infrared (SWIR) spectral regions. Using a tested design, we developed a dual configuration and tested its performance on a set of samples demonstrating spatial resolution and spectral reconstruction. Samples showing a potential use of the setup in optical defect detection are also tested. The setup showcases a dual single-pixel camera configuration capable of combining various detectors with a shared spatial modulation, further improving data efficiency and providing an affordable instrument from broadband spectral studies.

Spatial temporal Fourier single-pixel imaging

Generally, the imaging quality of Fourier single-pixel imaging (FSI) will severely degrade while achieving high-speed imaging at a low sampling rate (SR). To tackle this problem, a new, to the best of our knowledge, imaging technique is proposed: firstly, the Hessian-based norm constraint is introduced to deal with the staircase effect caused by the low SR and total variation regularization; secondly, based on the local similarity prior of consecutive frames in the time dimension, we designed the temporal local image low-rank constraint for the FSI, and combined the spatiotemporal random sampling method, the redundancy image information of consecutive frames can be utilized sufficiently; finally, by introducing additional variables to decompose the optimization problem into multiple sub-problems and analytically solving each one, a closed-form algorithm is derived for efficient image reconstruction. Experimental results show that the proposed method improves imaging quality significantly compared with state-of-the-art methods.

Single-pixel imaging with high spectral and spatial resolution

It has long been a challenge to obtain high spectral and spatial resolution simultaneously for the field of measurement and detection. Here we present a measurement system based on single-pixel imaging with compressive sensing that can realize excellent spectral and spatial resolution at the same time, as well as data compression. Our method can achieve high spectral and spatial resolution, which is different from the mutually restrictive relationship between the two in traditional imaging. In our experiments, 301 spectral channels are obtained in the band of 420-780 nm with a spectral resolution of 1.2 nm and a spatial resolution of 1.11 mrad. A sampling rate of 12.5% for a 64×64p i x e l image is obtained by using compressive sensing, which also reduces the measurement time; thus, high spectral and spatial resolution are realized simultaneously, even at a low sampling rate.

Compressed ultrahigh-speed single-pixel imaging by swept aggregate patterns

Single-pixel imaging (SPI) has emerged as a powerful technique that uses coded wide-field illumination with sampling by a single-point detector. Most SPI systems are limited by the refresh rates of digital micromirror devices (DMDs) and time-consuming iterations in compressed-sensing (CS)-based reconstruction. Recent efforts in overcoming the speed limit in SPI, such as the use of fast-moving mechanical masks, suffer from low reconfigurability and/or reduced accuracy. To address these challenges, we develop SPI accelerated via swept aggregate patterns (SPI-ASAP) that combines a DMD with laser scanning hardware to achieve pattern projection rates of up to 14.1 MHz and tunable frame sizes of up to 101×103 pixels. Meanwhile, leveraging the structural properties of S-cyclic matrices, a lightweight CS reconstruction algorithm, fully compatible with parallel computing, is developed for real-time video streaming at 100 frames per second (fps). SPI-ASAP allows reconfigurable imaging in both transmission and reflection modes, dynamic imaging under strong ambient light, and offline ultrahigh-speed imaging at speeds of up to 12,000 fps.

Experimental Study of Ghost Imaging in Underwater Environment

Underwater imaging technique is a crucial tool for humans to develop, utilize, and protect the ocean. We comprehensively compare the imaging performance of twenty-four ghost imaging (GI) methods in the underwater environment. The GI methods are divided into two types according to the illumination patterns, the random and orthogonal patterns. Three-group simulations were designed to show the imaging performance of the twenty-four GI methods. Moreover, an experimental system was built, and three-group experiments were implemented. The numerical and experimental results demonstrate that the orthogonal pattern-based compressed sensing GI methods have strong antinoise capability and can restore clear images for underwater objects with a low measurement number. The investigation results are helpful for the practical applications of the underwater GI.

Single pixel imaging at high pixel resolutions

The usually reported pixel resolution of single pixel imaging (SPI) varies between 32 × 32 and 256 × 256 pixels falling far below imaging standards with classical methods. Low resolution results from the trade-off between the acceptable compression ratio, the limited DMD modulation frequency, and reasonable reconstruction time, and has not improved significantly during the decade of intensive research on SPI. In this paper we show that image measurement at the full resolution of the DMD, which lasts only a fraction of a second, is possible for sparse images or in a situation when the field of view is limited but is a priori unknown. We propose the sampling and reconstruction strategies that enable us to reconstruct sparse images at the resolution of 1024 × 768 within the time of 0.3s. Non-sparse images are reconstructed with less details. The compression ratio is on the order of 0.4% which corresponds to an acquisition frequency of 7Hz. Sampling is differential, binary, and non-adaptive, and includes information on multiple partitioning of the image which later allows us to determine the actual field of view. Reconstruction is based on the differential Fourier domain regularized inversion (D-FDRI). The proposed SPI framework is an alternative to both adaptive SPI, which is challenging to implement in real time, and to classical compressive sensing image recovery methods, which are very slow at high resolutions.

Multi-Object Positioning and Imaging Based on Single-Pixel Imaging Using Binary Patterns

Single-pixel imaging (SPI) is a new type of imaging technology that uses a non-scanning single-pixel detector to image objects and has important application prospects and value in many fields. Most of the modulators currently used in SPI systems are digital micromirror device (DMD) modulators, which use a higher frequency for binary modulation than other alternatives. When modulating grayscale information, the modulation frequency is significantly reduced. This paper conducts research on multiple discrete objects in a scene and proposes using binary patterns to locate and image these objects. Compared with the existing methods of using gray patterns to locate and image multiple objects, the method proposed in this paper is more suitable for DMD-type SPI systems and has wider applicability and greater prospects. The principle of the proposed method is introduced, and the effectiveness of the method is experimentally verified. The experimental results show that, compared to traditional SPI methods, the number of patterns required by the proposed method is reduced by more than 85%.

Fast and high-quality single-pixel imaging

The imaging quality of the conventional single-pixel-imaging (SPI) technique seriously degrades at a low sampling rate. To tackle this problem, we propose an efficient sampling method and a high-quality real-time image reconstruction strategy: first, different from the conventional simple circular path sampling strategy or variable density random sampling technique, the proposed method samples the Fourier spectrum using the spectrum distribution of the image, that is, sampling the significant spectrum coefficients first, which will help to improve the image quality at a relevantly low sampling rate; second, to handle the long image reconstruction time caused by the iterative algorithm, the sparsity of the image and the alternating direction optimization strategy are combined to ameliorate the reconstruction process in the image gradient space. Compared with the state-of-the-art techniques, the proposed method significantly improves the imaging quality and achieves real-time reconstruction on the time scale of milliseconds.

Single-pixel camera based on a spinning mask

Single-pixel imaging (SPI) has been intensively studied in recent years for its capacity to obtain 2D images using a non-pixelated detector. However, the traditional modulation modality using an iteratively refreshed spatial light modulator has significantly restricted its imaging speed, which is a primary barrier to its widespread application. In this work, we propose and demonstrate a new, to the best of our knowledge, SPI scheme using a spinning mask for modulation. An annular binary mask is designed and spun to perform fast spatial modulation, neglecting the iterative modulation modality that limits SPI speed. A multi-spectral SPI system at 100 frames per second is demonstrated, covering a wide range of spectra, from ultraviolet to short-wave infrared light. We believe that this elegant and low-cost scheme will enable SPI to pave its way for practical application.

Differential real-time single-pixel imaging with Fourier domain regularization: applications to VIS-IR imaging and polarization imaging

The speed and quality of single-pixel imaging (SPI) are fundamentally limited by image modulation frequency and by the levels of optical noise and compression noise. In an approach to come close to these limits, we introduce a SPI technique, which is inherently differential, and comprises a novel way of measuring the zeroth spatial frequency of images and makes use of varied thresholding of sampling patterns. With the proposed sampling, the entropy of the detection signal is increased in comparison to standard SPI protocols. Image reconstruction is obtained with a single matrix-vector product so the cost of the reconstruction method scales proportionally with the number of measured samples. A differential operator is included in the reconstruction and following the method is based on finding the generalized inversion of the modified measurement matrix with regularization in the Fourier domain. We demonstrate 256 × 256 SPI at up to 17 Hz at visible and near-infrared wavelength ranges using 2 polarization or spectral channels. A low bit-resolution data acquisition device with alternating-current-coupling can be used in the measurement indicating that the proposed method combines improved noise robustness with a differential removal of the direct current component of the signal.

Phase and amplitude reconstruction in single-pixel transmission microscopy: a comparison of Hadamard, cosine, and noiselet bases

Hadamard, cosine, and noiselet bases are implemented into a digital holographic microscope based on single-pixel imaging with the capability to retrieve images of complex objects. The object is illuminated with coherent light modulated with different patterns deployed in a digital micromirror device, and the resulting fields are captured by single-pixel detection. For amplitude images, the experimental results of the three bases are evaluated with the peak SNR criteria. It is shown that the cosine basis recovers amplitude distributions with the best quality. Regarding phase images, the recovered ones compare well with those obtained with a CMOS camera.

Single pixel imaging at megahertz switching rates via cyclic Hadamard masks

Optical imaging is commonly performed with either a camera and wide-field illumination or with a single detector and a scanning collimated beam; unfortunately, these options do not exist at all wavelengths. Single-pixel imaging offers an alternative that can be performed with a single detector and wide-field illumination, potentially enabling imaging applications in which the detection and illumination technologies are immature. However, single-pixel imaging currently suffers from low imaging rates owing to its reliance on configurable spatial light modulators, generally limited to 22 kHz rates. We develop an approach for rapid single-pixel imaging which relies on cyclic patterns coded onto a spinning mask and demonstrate it for in vivo imaging of C. elegans worms. Spatial modulation rates of up to 2.4 MHz, imaging rates of up to 72 fps, and image-reconstruction times of down to 1.5 ms are reported, enabling real-time visualization of dynamic objects.

Noise Suppression in Compressive Single-Pixel Imaging

Compressive single-pixel imaging (CSPI) is a novel imaging scheme that retrieves images with nonpixelated detection. It has been studied intensively for its minimum requirement of detector resolution and capacity to reconstruct image with underdetermined acquisition. In practice, CSPI is inevitably involved with noise. It is thus essential to understand how noise affects its imaging process, and more importantly, to develop effective strategies for noise compression. In this work, two ypes of noise classified as multiplicative and additive noises are discussed. A normalized compressive reconstruction scheme is firstly proposed to counteract multiplicative noise. For additive noise, two types of compressive algorithms are studied. We find that pseudo-inverse operation could render worse reconstructions with more samplings in compressive sensing. This problem is then solved by introducing zero-mean inverse measurement matrix. Both experiment and simulation results show that our proposed algorithms significantly surpass traditional methods. Our study is believed to be helpful in not only CSPI but also other denoising works when compressive sensing is applied.

Hadamard transform-based hyperspectral imaging using a single-pixel detector

In this paper, a single-pixel hyperspectral imager is developed based on the Hadamard transformation. The imager's design, fabrication, signal processing method, and experimental results are discussed. The single-pixel hyperspectral imager works in pushbroom mode and employs both spatial encoding and spectral encoding to acquire the hyperspectral data cube. Hadamard encoding patterns, which are known for their multiplexing advantage to achieve high signal-to-noise ratio (SNR), are used in both encoding schemes. A digital micromirror device (DMD) from Texas Instruments (TI) is used for slow spatial encoding and a resonant scanning mirror in combination with a fixed Hadamard mask is used for fast spectral encoding. In addition, the pushbroom operation can be achieved internally by spatially shifting the location of the Hadamard encoded slit on the DMD, thus the imager is able to acquire 3D data cubes without the need to scan it across the object. Although our experimental results demonstrate the hyperspectral data cubes of various objects over a 450 nm ∼ 750 nm visible spectral range, the proposed imager can be easily configured to be used at other wavelengths due to the single-pixel detection mechanism used.

Real-time terahertz imaging with a single-pixel detector

Terahertz (THz) radiation is poised to have an essential role in many imaging applications, from industrial inspections to medical diagnosis. However, commercialization is prevented by impractical and expensive THz instrumentation. Single-pixel cameras have emerged as alternatives to multi-pixel cameras due to reduced costs and superior durability. Here, by optimizing the modulation geometry and post-processing algorithms, we demonstrate the acquisition of a THz-video (32 × 32 pixels at 6 frames-per-second), shown in real-time, using a single-pixel fiber-coupled photoconductive THz detector. A laser diode with a digital micromirror device shining visible light onto silicon acts as the spatial THz modulator. We mathematically account for the temporal response of the system, reduce noise with a lock-in free carrier-wave modulation and realize quick, noise-robust image undersampling. Since our modifications do not impose intricate manufacturing, require long post-processing, nor sacrifice the time-resolving capabilities of THz-spectrometers, their greatest asset, this work has the potential to serve as a foundation for all future single-pixel THz imaging systems.

Terahertz imaging is promising in many applications, but still relies on complex equipment. Here, the authors develop a simplified solution that enables terahertz real-time imaging using a single-pixel detector and rapid reconstruction methods.

Deringing and denoising in extremely under-sampled Fourier single pixel imaging

Undersampling in Fourier single pixel imaging (FSI) is often employed to reduce imaging time for real-time applications. However, the undersampled reconstruction contains ringing artifacts (Gibbs phenomenon) that occur because the high-frequency target information is not recorded. Furthermore, by employing 3-step FSI strategy (reduced measurements with low noise suppression) with a low-grade sensor (i.e., photodiode), this ringing is coupled with noise to produce unwanted artifacts, lowering image quality. To improve the imaging quality of real-time FSI, a fast image reconstruction framework based on deep convolutional autoencoder network (DCAN) is proposed. The network through context learning over FSI artifacts is capable of deringing, denoising, and recovering details in 256 × 256 images. The promising experimental results show that the proposed deep-learning-based FSI outperforms conventional FSI in terms of image quality even at very low sampling rates (1-4%).

Single-Pixel MEMS Imaging Systems

Single-pixel imaging technology is an attractive technology considering the increasing demand of imagers that can operate in wavelengths where traditional cameras have limited efficiency. Meanwhile, the miniaturization of imaging systems is also desired to build affordable and portable devices for field applications. Therefore, single-pixel imaging systems based on microelectromechanical systems (MEMS) is an effective solution to develop truly miniaturized imagers, owing to their ability to integrate multiple functionalities within a small device. MEMS-based single-pixel imaging systems have mainly been explored in two research directions, namely the encoding-based approach and the scanning-based approach. The scanning method utilizes a variety of MEMS scanners to scan the target scenery and has potential applications in the biological imaging field. The encoding-based system typically employs MEMS modulators and a single-pixel detector to encode the light intensities of the scenery, and the images are constructed by harvesting the power of computational technology. This has the capability to capture non-visible images and 3D images. Thus, this review discusses the two approaches in detail, and their applications are also reviewed to evaluate the efficiency and advantages in various fields.

Instant ghost imaging: algorithm and on-chip implementation

Ghost imaging (GI) is an imaging technique that uses the correlation between two light beams to reconstruct the image of an object. Conventional GI algorithms require large memory space to store the measured data and perform complicated offline calculations, limiting practical applications of GI. Here we develop an instant ghost imaging (IGI) technique with a differential algorithm and an implemented high-speed on-chip IGI hardware system. This algorithm uses the signal between consecutive temporal measurements to reduce the memory requirements without degradation of image quality compared with conventional GI algorithms. The on-chip IGI system can immediately reconstruct the image once the measurement finishes; there is no need to rely on post-processing or offline reconstruction. This system can be developed into a realtime imaging system. These features make IGI a faster, cheaper, and more compact alternative to a conventional GI system and make it viable for practical applications of GI.

Image watermarking and fusion based on Fourier single-pixel imaging with weighed light source

In previous single-pixel imaging systems, the light source was generally idle with respect to time. Here, we propose a novel image fusion and visible watermarking scheme based on Fourier single-pixel imaging (FSPI) with a multiplexed time-varying (TV) signal, which is generated by the watermark pattern hidden in the light source. We call this scheme TV-FSPI. With TV-FSPI, we can realize high-quality visible image watermarking, encrypted image watermarking and full-color visible image watermarking. We also discuss the extension to invisible watermarking based on TV-FSPI. Furthermore, we don't have to recode illumination patterns, because TV-FSPI can be extended to existing mainstream illumination patterns, such as random illumination mode and Hadamard illumination mode. Thus TV-FSPI has the potential to be used in single-pixel broadcasting system and multi-spectral single-pixel imaging system.

Secured regions of interest (SROIs) in single-pixel imaging

Single-pixel imaging, which is also known as computational ghost imaging, can reconstruct an entire image using one non-spatially resolved detector. However, it often requires a large amount of sampling, severely limiting its application. In this paper, we discuss the implementation of secured regions of interest (SROIs) in single-pixel imaging and illustrate its application using two experiments. Under a limited number of sampling times, we improved the resolution and recovered spectral information of interest in the ROI. Meanwhile, this scheme has high information security with high encryption and has great potential for single-pixel video and compressive multi-spectral single-pixel imaging.

Single-pixel imaging with sampling distributed over simplex vertices

We propose a method of reduction of experimental noise in single-pixel imaging by expressing the subsets of sampling patterns as linear combinations of vertices of a multidimensional regular simplex. This method also may be directly extended to complementary sampling. The modified measurement matrix contains nonnegative elements with patterns that may be directly displayed on intensity spatial light modulators. The measurement becomes theoretically independent of the ambient illumination, and in practice becomes more robust to the varying conditions of the experiment. We show how the optimal dimension of the simplex depends on the level of measurement noise. We present experimental results of single-pixel imaging using binarized sampling and real-time reconstruction with the Fourier domain regularized inversion method.

Single-Pixel Imaging and Its Application in Three-Dimensional Reconstruction: A Brief Review

Whereas modern digital cameras use a pixelated detector array to capture images, single-pixel imaging reconstructs images by sampling a scene with a series of masks and associating the knowledge of these masks with the corresponding intensity measured with a single-pixel detector. Though not performing as well as digital cameras in conventional visible imaging, single-pixel imaging has been demonstrated to be advantageous in unconventional applications, such as multi-wavelength imaging, terahertz imaging, X-ray imaging, and three-dimensional imaging. The developments and working principles of single-pixel imaging are reviewed, a mathematical interpretation is given, and the key elements are analyzed. The research works of three-dimensional single-pixel imaging and their potential applications are further reviewed and discussed.

Robust Entangled-Photon Ghost Imaging with Compressive Sensing

This work experimentally demonstrates that the imaging quality of quantum ghost imaging (GI) with entangled photons can be significantly improved by properly handling the errors caused by the imperfection of optical devices. We also consider compressive GI to reduce the number of measurements and thereby the data acquisition time. The image reconstruction is formulated as a sparse total least square problem which is solved with an iterative algorithm. Our experiments show that, compared with existing methods, the new method can achieve a significant performance gain in terms of mean square error and peak signal–noise ratio.

Design and realization of a wide field of view infrared scanning system with an integrated micro-electromechanical system mirror

We present a wide field of view (FOV) infrared scanning system, designed for single-pixel near-infrared thermal imaging. The scanning system consisted of a two-axis micro-electromechanical system (MEMS) mirror that was incorporated within the lens. The optical system consisted of two groups of lenses and a silicon avalanche photodiode. The system was designed for both the production of thermal images and also to utilize the techniques of radiation thermometry to measure the absolute temperature of targets from 500°C to 1100°C. Our system has the potential for real-time image acquisition, with improved data acquisition electronics. The FOV of our scanning system was ±30° when fully utilizing the MEMS mirror's scanning angle of ±5°. The pixel FOV (calculated from the distance to target size ratio) was 100:1. The image quality was analyzed, including the modulation transfer function, spot diagrams, ray fan plots, lateral chromatic aberrations, distortion, relative illumination, and size-of-source effect. The instrument was fabricated in our laboratory, and one of the thermal images, which was taken with the new lens, is presented as an example of the instrument optical performance.