Single-shot compressed optical-streaking ultra-high-speed photography

Discrete Illumination-Based Compressed Ultrafast Photography for High-Fidelity Dynamic Imaging

Compressed ultrafast photography (CUP) can capture irreversible or difficult-to-repeat dynamic scenes at the imaging speed of more than one billion frames per second, which is obtained by compressive sensing-based image reconstruction from a compressed 2D image through the discretization of detector pixels. However, an excessively high data compression ratio in CUP severely degrades the image reconstruction quality, thereby restricting its ability to observe ultrafast dynamic scenes with complex spatial structures. To address this issue, a discrete illumination-based CUP (DI-CUP) with high fidelity is reported. In DI-CUP, the dynamic scenes are loaded into an ultrashort laser pulse train with controllable sub-pulse number and time interval, thus the data compression ratio, as well as the overlap between adjacent frames, is greatly decreased and flexibly controlled through the discretization of dynamic scenes based on laser pulse train illumination, and high-fidelity image reconstruction can be realized within the same observation time window. Furthermore, the superior performance of DI-CUP is verified by observing femtosecond laser-induced ablation dynamics and plasma channel evolution, which are hardly resolved in the spatial structures using conventional CUP. It is anticipated that DI-CUP will be widely and dependably used in the real-time observations of various ultrafast dynamics.

Advancements in fluorescence lifetime imaging microscopy Instrumentation: Towards high speed and 3D

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging tool offering molecular specific insights into samples through the measurement of fluorescence decay time, with promising applications in diverse research fields. However, to acquire two-dimensional lifetime images, conventional FLIM relies on extensive scanning in both the spatial and temporal domain, resulting in much slower acquisition rates compared to intensity-based approaches. This problem is further magnified in three-dimensional imaging, as it necessitates additional scanning along the depth axis. Recent advancements have aimed to enhance the speed and three-dimensional imaging capabilities of FLIM. This review explores the progress made in addressing these challenges and discusses potential directions for future developments in FLIM instrumentation.

Spatial-temporal characterization of photoemission in a streak-mode dynamic transmission electron microscope

A long-standing motivation driving high-speed electron microscopy development is to capture phase transformations and material dynamics in real time with high spatial and temporal resolution. Current dynamic transmission electron microscopes (DTEMs) are limited to nanosecond temporal resolution and the ability to capture only a few frames of a transient event. With the motivation to overcome these limitations, we present our progress in developing a streak-mode DTEM (SM-DTEM) and demonstrate the recovery of picosecond images with high frame sequence depth. We first demonstrate that a zero-dimensional (0D) SM-DTEM can provide temporal information on any local region of interest with a 0.37 μm diameter, a 20-GHz sampling rate, and 1200 data points in the recorded trace. We use this method to characterize the temporal profile of the photoemitted electron pulse, finding that it deviates from the incident ultraviolet laser pulse and contains an unexpected peak near its onset. Then, we demonstrate a two-dimensional (2D) SM-DTEM, which uses compressed-sensing-based tomographic imaging to recover a full spatiotemporal photoemission profile over a 1.85-μm-diameter field of view with nanoscale spatial resolution, 370-ps inter-frame interval, and 140-frame sequence depth in a 50-ns time window. Finally, a perspective is given on the instrumental modifications necessary to further develop this promising technique with the goal of decreasing the time to capture a 2D SM-DTEM dataset.

Single-shot videography with multiplex structured illumination using an interferometer

Frequency recognition algorithm for multiple exposures (FRAME) is a high-speed videography technique that exposes a dynamic object to time-varying structured illumination (SI) and captures two-dimensional transients in a single shot. Conventional FRAME requires light splitting to increase the number of frames per shot, thereby resulting in optical loss and a limited number of frames per shot. Here, we propose and demonstrate a novel FRAME method which overcomes these problems by utilizing an interferometer to generate a time-varying SI without light splitting. Combining this method with a pulsed laser enables low-cost, high-speed videography on a variety of timescales from microseconds.

TDI-based continuous window compressed spatio-temporal imaging capable of flexible voxels post-interpretation

To achieve high frame rates and continuous streaming simultaneously, we propose a compressed spatio-temporal imaging framework implemented by combining time-delay-integration sensors and coded exposure. Without additional optical coding elements and subsequent calibration required, this electronic-domain modulation enables a more compact and robust hardware structure, compared to the existing imaging modalities. By exploiting the intra-line charge transfer mechanism, we achieve a super-resolution in both temporal and spatial domains, thus multiplying the frame rate to millions of frames-per-second. In addition, the forward model with post-tunable coefficients, and two reconstruction strategies proposed therefrom, facilitate a flexible voxels post-interpretation. Finally, the effectiveness of the proposed framework is demonstrated by both numerical simulations and proof-of-concept experiments. With the prominent advantages of prolonged time window and flexible voxels post-interpretation, the proposed system will be suitable for imaging random, non-repetitive, or long-term events.

Dynamic nitrogen vacancy magnetometry by single-shot optical streaking microscopy

Nitrogen vacancy diamonds have emerged as sensitive solid-state magnetic field sensors capable of producing diffraction limited and sub-diffraction field images. Here, for the first time, to our knowledge, we extend those measurements to high-speed imaging, which can be readily applied to analyze currents and magnetic field dynamics in circuits on a microscopic scale. To overcome detector acquisition rate limitations, we designed an optical streaking nitrogen vacancy microscope to acquire two-dimensional spatiotemporal kymograms. We demonstrate magnetic field wave imaging with micro-scale spatial extent and ~400 μs temporal resolution. In validating this system, we detected magnetic fields down to 10 μT for 40 Hz magnetic fields using single-shot imaging and captured the spatial transit of an electromagnetic needle at streak rates as high as 110 μm/ms. This design has the capability to be readily extended to full 3D video acquisition by utilizing compressed sensing techniques and a potential for further improvement of spatial resolution, acquisition speed, and sensitivity. The device opens opportunities to many potential applications where transient magnetic events can be isolated to a single spatial axis, such as acquiring spatially propagating action potentials for brain imaging and remotely interrogating integrated circuits.

FISI: frequency domain integration sequential imaging at 1.26×1013 frames per second and 108 lines per millimeter

High spatial resolution on the image plane (intrinsic spatial resolution) has always been a problem for ultrafast imaging. Single-shot ultrafast imaging methods can achieve high spatial resolution on the object plane through amplification systems but with low intrinsic spatial resolutions. We present frequency domain integration sequential imaging (FISI), which encodes a transient dynamic by an inversed 4f (IFF) system and decodes it using optical spatial frequencies recognition (OFR), which overcomes the limitation of the spatial frequencies recognition algorithm. In an experiment on the process of an air plasma channel, FISI achieved shadow imaging of the channel with a framing rate of 1.26×10¹³ fps and an intrinsic spatial resolution of 108 lp/mm (the spatial resolution on the image plane). Owing to its excellent framing time and high intrinsic spatial resolution, FISI can probe both repeatable and unrepeatable ultrafast phenomena, such as laser-induced damage, plasma physics, and shockwave interactions in living cells with high quality.

Shuffled rolling shutter for snapshot temporal imaging

We propose a modification to the rolling shutter mechanism found in CMOS detectors by shuffling the pixels in every scanline. This potential hardware modification improves the sampling of the space-time datacube, allowing the recovery of high-speed videos from a single image using either tensor completion methods or reconstruction algorithms often used for compressive temporal video. We also present a design methodology for optimal sampling schemes and compare them to random shuffling. Simulations, and experimental results obtained by optically emulating the hardware, demonstrate the ability of the shuffled rolling shutter to capture images that allow reconstructing videos, which would otherwise be impossible when using the traditional rolling shutter mechanism.

Continuously streaming compressed high-speed photography using time delay integration

An imaging system capable of acquiring high-resolution data at a high speed is in demand. However, the amount of optical information captured by a modern camera is limited by the data transfer bandwidth of electronics, resulting in a reduced spatial and temporal resolution. To overcome this problem, we developed continuously streaming compressed high-speed photography, which can record a dynamic scene with an unprecedented space-bandwidth-time product. By performing compressed imaging in a time-delay-integration manner, we continuously recorded a 0.85 megapixel video at 200 kHz, corresponding to an information flux of 170 gigapixels per second.

Fast wide-field upconversion luminescence lifetime thermometry enabled by single-shot compressed ultrahigh-speed imaging

Photoluminescence lifetime imaging of upconverting nanoparticles is increasingly featured in recent progress in optical thermometry. Despite remarkable advances in photoluminescent temperature indicators, existing optical instruments lack the ability of wide-field photoluminescence lifetime imaging in real time, thus falling short in dynamic temperature mapping. Here, we report video-rate upconversion temperature sensing in wide field using single-shot photoluminescence lifetime imaging thermometry (SPLIT). Developed from a compressed-sensing ultrahigh-speed imaging paradigm, SPLIT first records wide-field luminescence intensity decay compressively in two views in a single exposure. Then, an algorithm, built upon the plug-and-play alternating direction method of multipliers, is used to reconstruct the video, from which the extracted lifetime distribution is converted to a temperature map. Using the core/shell NaGdF4:Er3+,Yb3+/NaGdF4 upconverting nanoparticles as the lifetime-based temperature indicators, we apply SPLIT in longitudinal wide-field temperature monitoring beneath a thin scattering medium. SPLIT also enables video-rate temperature mapping of a moving biological sample at single-cell resolution.

Sparsity-Based Recovery of Three-Dimensional Photoacoustic Images from Compressed Single-Shot Optical Detection

Photoacoustic (PA) imaging combines optical excitation with ultrasonic detection to achieve high-resolution imaging of biological samples. A high-energy pulsed laser is often used for imaging at multi-centimeter depths in tissue. These lasers typically have a low pulse repetition rate, so to acquire images in real-time, only one pulse of the laser can be used per image. This single pulse necessitates the use of many individual detectors and receive electronics to adequately record the resulting acoustic waves and form an image. Such requirements make many PA imaging systems both costly and complex. This investigation proposes and models a method of volumetric PA imaging using a state-of-the-art compressed sensing approach to achieve real-time acquisition of the initial pressure distribution (IPD) at a reduced level of cost and complexity. In particular, a single exposure of an optical image sensor is used to capture an entire Fabry–Pérot interferometric acoustic sensor. Time resolved encoding as achieved through spatial sweeping with a galvanometer. This optical system further makes use of a random binary mask to set a predetermined subset of pixels to zero, thus enabling recovery of the time-resolved signals. The Two-Step Iterative Shrinking and Thresholding algorithm is used to reconstruct the IPD, harnessing the sparsity naturally occurring in the IPD as well as the additional structure provided by the binary mask. We conduct experiments on simulated data and analyze the performance of our new approach.

Compressed ultrafast tomographic imaging by passive spatiotemporal projections

Existing streak-camera-based two-dimensional (2D) ultrafast imaging techniques are limited by long acquisition time, the trade-off between spatial and temporal resolutions, and a reduced field of view. They also require additional components, customization, or active illumination. Here we develop compressed ultrafast tomographic imaging (CUTI), which passively records 2D transient events with a standard streak camera. By grafting the concept of computed tomography to the spatiotemporal domain, the operations of temporal shearing and spatiotemporal integration in a streak camera's data acquisition can be equivalently expressed as the spatiotemporal projection of an (x,y,t) datacube from a certain angle. Aided by a new, to the best of our knowledge, compressed-sensing reconstruction algorithm, the 2D transient event can be accurately recovered in a few measurements. CUTI is exhibited as a new imaging mode universally adaptable to most streak cameras. Implemented in an image-converter streak camera, CUTI captures the sequential arrival of two spatially modulated ultrashort ultraviolet laser pulses at 0.5 trillion frames per second. Applied to a rotating-mirror streak camera, CUTI records an amination of fast-bouncing balls at 5,000 frames per second.

Compressive Coded Rotating Mirror Camera for High-Speed Imaging

We develop a novel compressive coded rotating mirror (CCRM) camera to capture events at high frame rates in passive mode with a compact instrument design at a fraction of the cost compared to other high-speed imaging cameras. Operation of the CCRM camera is based on amplitude optical encoding (grey scale) and a continuous frame sweep across a low-cost detector using a motorized rotating mirror system which can achieve single pixel shift between adjacent frames. Amplitude encoding and continuous frame overlapping enable the CCRM camera to achieve a high number of captured frames and high temporal resolution without making sacrifices in the spatial resolution. Two sets of dynamic scenes have been captured at up to a 120 Kfps frame rate in both monochrome and colored scales in the experimental demonstrations. The obtained heavily compressed data from the experiment are reconstructed using the optimization algorithm under the compressive sensing (CS) paradigm and the highest sequence depth of 1400 captured frames in a single exposure has been achieved with the highest compression ratio of 368 compared to other CS-based high-speed imaging technologies. Under similar conditions the CCRM camera is 700× faster than conventional rotating mirror based imaging devices and could reach a frame rate of up to 20 Gfps.

Snapshot multidimensional photography through active optical mapping

Multidimensional photography can capture optical fields beyond the capability of conventional image sensors that measure only two-dimensional (2D) spatial distribution of light. By mapping a high-dimensional datacube of incident light onto a 2D image sensor, multidimensional photography resolves the scene along with other information dimensions, such as wavelength and time. However, the application of current multidimensional imagers is fundamentally restricted by their static optical architectures and measurement schemes—the mapping relation between the light datacube voxels and image sensor pixels is fixed. To overcome this limitation, we propose tunable multidimensional photography through active optical mapping. A high-resolution spatial light modulator, referred to as an active optical mapper, permutes and maps the light datacube voxels onto sensor pixels in an arbitrary and programmed manner. The resultant system can readily adapt the acquisition scheme to the scene, thereby maximising the measurement flexibility. Through active optical mapping, we demonstrate our approach in two niche implementations: hyperspectral imaging and ultrafast imaging.

Multidimensional photography has traditionally been restricted by their static optical architectures and measurement schemes. Here, the authors present a tunable multidimensional photography approach employing active optical mapping, which allows them to adapt the acquisition schemes to the scene.

Punching holes in light: recent progress in single-shot coded-aperture optical imaging

Single-shot coded-aperture optical imaging physically captures a code-aperture-modulated optical signal in one exposure and then recovers the scene via computational image reconstruction. Recent years have witnessed dazzling advances in various modalities in this hybrid imaging scheme in concomitant technical improvement and widespread applications in physical, chemical and biological sciences. This review comprehensively surveys state-of-the-art single-shot coded-aperture optical imaging. Based on the detected photon tags, this field is divided into six categories: planar imaging, depth imaging, light-field imaging, temporal imaging, spectral imaging, and polarization imaging. In each category, we start with a general description of the available techniques and design principles, then provide two representative examples of active-encoding and passive-encoding approaches, with a particular emphasis on their methodology and applications as well as their advantages and challenges. Finally, we envision prospects for further technical advancement in this field.

Beyond MHz image recordings using LEDs and the FRAME concept

Many important scientific questions in physics, chemistry and biology rely on high-speed optical imaging techniques for their investigations. These techniques are either passive, relying on the rapid readout of photoactive elements, or active, relying on the illumination properties of specially designed pulse trains. Currently, MHz imaging speeds are difficult to realize; passive methods, being dictated by electronics, cause the unification of high spatial resolution with high frame rates to be very challenging, while active methods rely on expensive and complex hardware such as femto- and picosecond laser sources. Here we present an accessible temporally resolved imaging system for shadowgraphy based on multiplexed LED illumination that is capable of producing four images at MHz frame rates. Furthermore as the LEDs are independent of each other, any light burst configuration can be obtained, allowing for instance the simultaneous determination of low- and high speed events in parallel. To the best of the authors' knowledge, this is the fastest high speed imaging system that does not rely on pulsed lasers or fast detectors, in this case reaching up to 4.56 MHz.

Single-shot ultrafast imaging attaining 70 trillion frames per second

Real-time imaging of countless femtosecond dynamics requires extreme speeds orders of magnitude beyond the limits of electronic sensors. Existing femtosecond imaging modalities either require event repetition or provide single-shot acquisition with no more than 10¹³ frames per second (fps) and 3 × 10² frames. Here, we report compressed ultrafast spectral photography (CUSP), which attains several new records in single-shot multi-dimensional imaging speeds. In active mode, CUSP achieves both 7 × 10¹³ fps and 10³ frames simultaneously by synergizing spectral encoding, pulse splitting, temporal shearing, and compressed sensing-enabling unprecedented quantitative imaging of rapid nonlinear light-matter interaction. In passive mode, CUSP provides four-dimensional (4D) spectral imaging at 0.5 × 10¹² fps, allowing the first single-shot spectrally resolved fluorescence lifetime imaging microscopy (SR-FLIM). As a real-time multi-dimensional imaging technology with the highest speeds and most frames, CUSP is envisioned to play instrumental roles in numerous pivotal scientific studies without the need for event repetition.

A versatile, low-cost, snapshot multidimensional imaging approach based on structured light

The behaviour and function of dynamic samples can be investigated using optical imaging approaches with high temporal resolution and multidimensional acquisition. Snapshot techniques have been developed in order to meet these demands, however they are often designed to study a specific parameter, such as spectral properties, limiting their applicability. Here we present and demonstrate a frequency recognition algorithm for multiple exposures (FRAME) snapshot imaging approach, which can be reconfigured to capture polarization, temporal, depth-of-focus and spectral information by simply changing the filters used. FRAME is implemented by splitting the emitted light from a sample into four channels, filtering the light and then applying a unique spatial modulation encoding before recombining all the channels. The multiplexed information is collected in a single exposure using a single detector and extracted in post processing of the Fourier transform of the collected image, where each channel image is located in a distinct region of the Fourier domain. The approach allows for individual intensity control in each channel, has easily interchangeable filters and can be used in conjunction with, in principle, all 2D detectors, making it a low cost and versatile snapshot multidimensional imaging technique.

Real-time high-speed three-dimensional surface imaging using band-limited illumination profilometry with a CoaXPress interface

High-speed three-dimensional (3D) surface imaging by structured-light profilometry is currently driven by numerous applications. However, the limited speeds in fringe pattern projection, image acquisition, and data transmission have strained the existing methods from reaching kilohertz-level acquisition, processing, and display of 3D information during the occurrence of dynamic events (i.e., in real time). To overcome these limitations, we have developed band-limited illumination profilometry (BLIP) with a CoaXPress interface (CI), which enables real-time high-speed 3D surface imaging. We have demonstrated the system's performance by imaging various static and fast-moving 3D objects in real time. We have also applied this system in fluid mechanics by imaging dynamics of a flag, which allowed observation of the wave propagation, gravity-induced phase mismatch, and asymmetric flapping motion. We expect CI-BLIP to find diverse scientific and industrial applications.

Interferometry-free noncontact photoacoustic detection method based on speckle correlation change

Optical speckle patterns occur when a coherent optical wavefront is randomized but such stochastic yet deterministic information about the medium can be decoded. A simple setup is inspired to monitor the decorrelation of speckle patterns within the memory effect range when the medium is photoacoustically perturbated. Experimentally, a linear relationship is confirmed between the speckle correlation change and the peak-to-peak amplitude of the ultrasonic transducer-detected photoacoustic waves, and the detection sensitivity is comparable. Such a plain specklegram-based method may find special interests when no direct contact is allowed between the sample and the photoacoustic detector.