A trillion frames per second: the techniques and applications of light-in-flight photography

Swept coded aperture real-time femtophotography

Single-shot real-time femtophotography is indispensable for imaging ultrafast dynamics during their times of occurrence. Despite their advantages over conventional multi-shot approaches, existing techniques confront restricted imaging speed or degraded data quality by the deployed optoelectronic devices and face challenges in the application scope and acquisition accuracy. They are also hindered by the limitations in the acquirable information imposed by the sensing models. Here, we overcome these challenges by developing swept coded aperture real-time femtophotography (SCARF). This computational imaging modality enables all-optical ultrafast sweeping of a static coded aperture during the recording of an ultrafast event, bringing full-sequence encoding of up to 156.3 THz to every pixel on a CCD camera. We demonstrate SCARF's single-shot ultrafast imaging ability at tunable frame rates and spatial scales in both reflection and transmission modes. Using SCARF, we image ultrafast absorption in a semiconductor and ultrafast demagnetization of a metal alloy.

Spatial-temporal characterization of photoemission in a streak-mode dynamic transmission electron microscope

A long-standing motivation driving high-speed electron microscopy development is to capture phase transformations and material dynamics in real time with high spatial and temporal resolution. Current dynamic transmission electron microscopes (DTEMs) are limited to nanosecond temporal resolution and the ability to capture only a few frames of a transient event. With the motivation to overcome these limitations, we present our progress in developing a streak-mode DTEM (SM-DTEM) and demonstrate the recovery of picosecond images with high frame sequence depth. We first demonstrate that a zero-dimensional (0D) SM-DTEM can provide temporal information on any local region of interest with a 0.37 μm diameter, a 20-GHz sampling rate, and 1200 data points in the recorded trace. We use this method to characterize the temporal profile of the photoemitted electron pulse, finding that it deviates from the incident ultraviolet laser pulse and contains an unexpected peak near its onset. Then, we demonstrate a two-dimensional (2D) SM-DTEM, which uses compressed-sensing-based tomographic imaging to recover a full spatiotemporal photoemission profile over a 1.85-μm-diameter field of view with nanoscale spatial resolution, 370-ps inter-frame interval, and 140-frame sequence depth in a 50-ns time window. Finally, a perspective is given on the instrumental modifications necessary to further develop this promising technique with the goal of decreasing the time to capture a 2D SM-DTEM dataset.

Imaging operator in indirect imaging correlography

Indirect imaging correlography (IIC) is a coherent imaging technique that provides access to the autocorrelation of the albedo of objects obscured from line-of-sight. This technique is used to recover sub-mm resolution images of obscured objects at large standoffs in non-line-of-sight (NLOS) imaging. However, predicting the exact resolving power of IIC in any given NLOS scene is complicated by the interplay between several factors, including object position and pose. This work puts forth a mathematical model for the imaging operator in IIC to accurately predict the images of objects in NLOS imaging scenes. Using the imaging operator, expressions for the spatial resolution as a function of scene parameters such as object position and pose are derived and validated experimentally. In addition, a self-supervised deep neural network framework to reconstruct images of objects from their autocorrelation is proposed. Using this framework, objects with ≈ 250 μ m features, located at 1 mt standoffs in an NLOS scene, are successfully reconstructed.

Flatfield ultrafast imaging with single-shot non-synchronous array photography

We present a method for acquiring a sequence of time-resolved images in a single shot, called single-shot non-synchronous array photography (SNAP). In SNAP, a pulsed laser beam is split by a diffractive optical element into an array of angled beamlets whose illumination fronts remain perpendicular to the optical axis. Different time delays are imparted to each beamlet by an echelon, enabling them to probe ultrafast dynamics in rapid succession. The beamlets are imaged onto different regions of a camera by a lenslet array. Because the illumination fronts remain flat (head-on) independently of beamlet angle, the exposure time in SNAP is fundamentally limited only by the laser pulse duration, akin to a "global shutter" in conventional imaging. We demonstrate SNAP by capturing the evolution of a laser induced plasma filament over 20 frames at an average rate of 4.2 trillion frames per second (Tfps) and a peak rate of 5.7 Tfps.

Intensity-corrected 4D light-in-flight imaging

Light-in-flight (LIF) imaging is the measurement and reconstruction of light's path as it moves and interacts with objects. It is well known that relativistic effects can result in apparent velocities that differ significantly from the speed of light. However, less well known is that Rayleigh scattering and the effects of imaging optics can lead to observed intensities changing by several orders of magnitude along light's path. We develop a model that enables us to correct for all of these effects, thus we can accurately invert the observed data and reconstruct the true intensity-corrected optical path of a laser pulse as it travels in air. We demonstrate the validity of our model by observing the photon arrival time and intensity distribution obtained from single-photon avalanche detector (SPAD) array data for a laser pulse propagating towards and away from the camera. We can then reconstruct the true intensity-corrected path of the light in four dimensions (three spatial dimensions and time).

Continuous-capture microwave imaging

Light-in-flight sensing has emerged as a promising technique in image reconstruction applications at various wavelengths. We report a microwave imaging system that uses an array of transmitters and a single receiver operating in continuous transmit-receive mode. Captures take a few microseconds and the corresponding images cover a spatial range of tens of square meters with spatial resolution of 0.1 meter. The images are the result of a dot product between a reconstruction matrix and the captured signal with no prior knowledge of the scene. The reconstruction matrix uses an engineered electromagnetic field mask to create unique random time patterns at every point in the scene and correlates it with the captured signal to determine the corresponding voxel value. We report the operation of the system through simulations and experiment in a laboratory scene. We demonstrate through-wall real-time imaging, tracking, and observe second-order images from specular reflections.

Single-point material recognition by quantum parametric mode sorting and photon counting

We explore an active illumination approach to remote material recognition, based on quantum parametric mode sorting and single-photon detection. By measuring a photon's time of flight at picosecond resolution, 97.8% recognition is demonstrated by illuminating only a single point on the materials. Thanks to the exceptional detection sensitivity and noise rejection, a high recognition accuracy of 96.1% is achieved even when the materials are occluded by a lossy and multiscattering obscurant.

Compressed ultrafast tomographic imaging by passive spatiotemporal projections

Existing streak-camera-based two-dimensional (2D) ultrafast imaging techniques are limited by long acquisition time, the trade-off between spatial and temporal resolutions, and a reduced field of view. They also require additional components, customization, or active illumination. Here we develop compressed ultrafast tomographic imaging (CUTI), which passively records 2D transient events with a standard streak camera. By grafting the concept of computed tomography to the spatiotemporal domain, the operations of temporal shearing and spatiotemporal integration in a streak camera's data acquisition can be equivalently expressed as the spatiotemporal projection of an (x,y,t) datacube from a certain angle. Aided by a new, to the best of our knowledge, compressed-sensing reconstruction algorithm, the 2D transient event can be accurately recovered in a few measurements. CUTI is exhibited as a new imaging mode universally adaptable to most streak cameras. Implemented in an image-converter streak camera, CUTI captures the sequential arrival of two spatially modulated ultrashort ultraviolet laser pulses at 0.5 trillion frames per second. Applied to a rotating-mirror streak camera, CUTI records an amination of fast-bouncing balls at 5,000 frames per second.

Super-resolution time-resolved imaging using computational sensor fusion

Imaging across both the full transverse spatial and temporal dimensions of a scene with high precision in all three coordinates is key to applications ranging from LIDAR to fluorescence lifetime imaging. However, compromises that sacrifice, for example, spatial resolution at the expense of temporal resolution are often required, in particular when the full 3-dimensional data cube is required in short acquisition times. We introduce a sensor fusion approach that combines data having low-spatial resolution but high temporal precision gathered with a single-photon-avalanche-diode (SPAD) array with data that has high spatial but no temporal resolution, such as that acquired with a standard CMOS camera. Our method, based on blurring the image on the SPAD array and computational sensor fusion, reconstructs time-resolved images at significantly higher spatial resolution than the SPAD input, upsampling numerical data by a factor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$12 \times 12$$\end{document}12×12, and demonstrating up to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$4 \times 4$$\end{document}4×4 upsampling of experimental data. We demonstrate the technique for both LIDAR applications and FLIM of fluorescent cancer cells. This technique paves the way to high spatial resolution SPAD imaging or, equivalently, FLIM imaging with conventional microscopes at frame rates accelerated by more than an order of magnitude.

Fluorescence lifetime imaging with a megapixel SPAD camera and neural network lifetime estimation

Fluorescence lifetime imaging microscopy (FLIM) is a key technology that provides direct insight into cell metabolism, cell dynamics and protein activity. However, determining the lifetimes of different fluorescent proteins requires the detection of a relatively large number of photons, hence slowing down total acquisition times. Moreover, there are many cases, for example in studies of cell collectives, where wide-field imaging is desired. We report scan-less wide-field FLIM based on a 0.5 MP resolution, time-gated Single Photon Avalanche Diode (SPAD) camera, with acquisition rates up to 1 Hz. Fluorescence lifetime estimation is performed via a pre-trained artificial neural network with 1000-fold improvement in processing times compared to standard least squares fitting techniques. We utilised our system to image HT1080-human fibrosarcoma cell line as well as Convallaria. The results show promise for real-time FLIM and a viable route towards multi-megapixel fluorescence lifetime images, with a proof-of-principle mosaic image shown with 3.6 MP.

Punching holes in light: recent progress in single-shot coded-aperture optical imaging

Single-shot coded-aperture optical imaging physically captures a code-aperture-modulated optical signal in one exposure and then recovers the scene via computational image reconstruction. Recent years have witnessed dazzling advances in various modalities in this hybrid imaging scheme in concomitant technical improvement and widespread applications in physical, chemical and biological sciences. This review comprehensively surveys state-of-the-art single-shot coded-aperture optical imaging. Based on the detected photon tags, this field is divided into six categories: planar imaging, depth imaging, light-field imaging, temporal imaging, spectral imaging, and polarization imaging. In each category, we start with a general description of the available techniques and design principles, then provide two representative examples of active-encoding and passive-encoding approaches, with a particular emphasis on their methodology and applications as well as their advantages and challenges. Finally, we envision prospects for further technical advancement in this field.

Noise-tolerant single photon sensitive three-dimensional imager

Active imagers capable of reconstructing 3-dimensional (3D) scenes in the presence of strong background noise are highly desirable for many sensing and imaging applications. A key to this capability is the time-resolving photon detection that distinguishes true signal photons from the noise. To this end, quantum parametric mode sorting (QPMS) can achieve signal to noise exceeding by far what is possible with typical linear optics filters, with outstanding performance in isolating temporally and spectrally overlapping noise. Here, we report a QPMS-based 3D imager with exceptional detection sensitivity and noise tolerance. With only 0.0006 detected signal photons per pulse, we reliably reconstruct the 3D profile of an obscured scene, despite 34-fold spectral-temporally overlapping noise photons, within the 6 ps detection window (amounting to 113,000 times noise per 20 ns detection period). Our results highlight a viable approach to suppress background noise and measurement errors of single photon imager operation in high-noise environments.

Imagers capable of reconstructing three-dimensional scenes in the presence of strong background noise are desirable for many remote sensing and imaging applications. Here, the authors report an imager operating in photon-starved and noise-polluted environments through quantum parametric mode sorting.

A Single-Shot Non-Line-of-Sight Range-Finder

The ability to locate a target around a corner is crucial in situations where it is impractical or unsafe to physically move around the obstruction. However, current techniques are limited to long acquisition times as they rely on single-photon counting for precise arrival time measurements. Here, we demonstrate a single-shot non-line-of-sight range-finding method operating at 10 Hz and capable of detecting a moving human target up to distances of 3 m around a corner. Due to the potential data acquisition speeds, this technique will find applications in search and rescue and autonomous vehicles.

Action Recognition Using Single-Pixel Time-of-Flight Detection

Action recognition is a challenging task that plays an important role in many robotic systems, which highly depend on visual input feeds. However, due to privacy concerns, it is important to find a method which can recognise actions without using visual feed. In this paper, we propose a concept for detecting actions while preserving the test subject's privacy. Our proposed method relies only on recording the temporal evolution of light pulses scattered back from the scene. Such data trace to record one action contains a sequence of one-dimensional arrays of voltage values acquired by a single-pixel detector at 1 GHz repetition rate. Information about both the distance to the object and its shape are embedded in the traces. We apply machine learning in the form of recurrent neural networks for data analysis and demonstrate successful action recognition. The experimental results show that our proposed method could achieve on average 96.47 % accuracy on the actions walking forward, walking backwards, sitting down, standing up and waving hand, using recurrent neural network.

A light-in-flight single-pixel camera for use in the visible and short-wave infrared

Single-pixel cameras reconstruct images from a stream of spatial projection measurements recorded with a single-element detector, which itself has no spatial resolution. This enables the creation of imaging systems that can take advantage of the ultra-fast response times of single-element detectors. Here we present a single-pixel camera with a temporal resolution of 200 ps in the visible and short-wave infrared wavelengths, used here to study the transit time of distinct spatial modes transmitted through few-mode and orbital angular momentum mode conserving optical fiber. Our technique represents a way to study the spatial and temporal characteristics of light propagation in multimode optical fibers, which may find use in optical fiber design and communications.

Time-of-Flight Imaging at 10 ps Resolution with an ICCD Camera

ICCD cameras can record low light events with extreme temporal resolution. Thus, they are used in a variety of bio-medical applications for single photon time of flight measurements and LIDAR measurements. In this paper, we present a method which allows improvement of the temporal resolution of ICCD cameras down to 10 ps (from the native 200 ps of our model), thus placing ICCD cameras at a better temporal resolution than SPAD cameras and in direct competition with streak cameras. The higher temporal resolution can serve for better tracking and visualization of the information carried in time-of-flight measurements.