All Photons Imaging Through Volumetric Scattering

Learning-based real-time imaging through dynamic scattering media

Imaging through dynamic scattering media is one of the most challenging yet fascinating problems in optics, with applications spanning from biological detection to remote sensing. In this study, we propose a comprehensive learning-based technique that facilitates real-time, non-invasive, incoherent imaging of real-world objects through dense and dynamic scattering media. We conduct extensive experiments, demonstrating the capability of our technique to see through turbid water and natural fog. The experimental results indicate that the proposed technique surpasses existing approaches in numerous aspects and holds significant potential for imaging applications across a broad spectrum of disciplines.

Imaging inside highly scattering media using hybrid deep learning and analytical algorithm

Imaging through highly scattering media is a challenging problem with numerous applications in biomedical and remote-sensing fields. Existing methods that use analytical or deep learning tools are limited by simplified forward models or a requirement for prior physical knowledge, resulting in blurry images or a need for large training databases. To address these limitations, we propose a hybrid scheme called Hybrid-DOT that combines analytically derived image estimates with a deep learning network. Our analysis demonstrates that Hybrid-DOT outperforms a state-of-the-art ToF-DOT algorithm by improving the PSNR ratio by 4.6 dB and reducing the resolution by a factor of 2.5. Furthermore, when compared to a deep learning stand-alone model, Hybrid-DOT achieves a 0.8 dB increase in PSNR, 1.5 times the resolution, and a significant reduction in the required dataset size (factor of 1.6-3). The proposed model remains effective at higher depths, providing similar improvements for up to 160 mean-free paths.

NeuWS: Neural wavefront shaping for guidestar-free imaging through static and dynamic scattering media

Diffraction-limited optical imaging through scattering media has the potential to transform many applications such as airborne and space-based imaging (through the atmosphere), bioimaging (through skin and human tissue), and fiber-based imaging (through fiber bundles). Existing wavefront shaping methods can image through scattering media and other obscurants by optically correcting wavefront aberrations using high-resolution spatial light modulators-but these methods generally require (i) guidestars, (ii) controlled illumination, (iii) point scanning, and/or (iv) statics scenes and aberrations. We propose neural wavefront shaping (NeuWS), a scanning-free wavefront shaping technique that integrates maximum likelihood estimation, measurement modulation, and neural signal representations to reconstruct diffraction-limited images through strong static and dynamic scattering media without guidestars, sparse targets, controlled illumination, nor specialized image sensors. We experimentally demonstrate guidestar-free, wide field-of-view, high-resolution, diffraction-limited imaging of extended, nonsparse, and static/dynamic scenes captured through static/dynamic aberrations.

Terahertz Nonlinear Ghost Imaging via Plane Decomposition: Toward Near-Field Micro-Volumetry

Terahertz time-domain imaging targets the reconstruction of the full electromagnetic morphology of an object. In this spectral range, the near-field propagation strongly affects the information in the space-time domain in items with microscopic features. While this often represents a challenge, as the information needs to be disentangled to obtain high image fidelity, here, we show that such a phenomenon can enable three-dimensional microscopy. Specifically, we investigate the capability of the time-resolved nonlinear ghost imaging methodology to implement field-sensitive micro-volumetry by plane decomposition. We leverage the temporally resolved, field-sensitive detection to "refocus" an image plane at an arbitrary distance from the source, which defines the near-field condition, and within a microscopic sample. Since space-time coupling rapidly evolves and diffuses within subwavelength length scales, our technique can separate and discriminate the information originating from different planes at different depths. Our approach is particularly suitable for objects with sparse micrometric details. Building upon this principle, we demonstrate complex, time-domain volumetry resolving internal object planes with subwavelength resolution, discussing the range of applicability of our technique.

Unrolled-DOT: an interpretable deep network for diffuse optical tomography

SIGNIFICANCE

Imaging through scattering media is critical in many biomedical imaging applications, such as breast tumor detection and functional neuroimaging. Time-of-flight diffuse optical tomography (ToF-DOT) is one of the most promising methods for high-resolution imaging through scattering media. ToF-DOT and many traditional DOT methods require an image reconstruction algorithm. Unfortunately, this algorithm often requires long computational runtimes and may produce lower quality reconstructions in the presence of model mismatch or improper hyperparameter tuning.

AIM

We used a data-driven unrolled network as our ToF-DOT inverse solver. The unrolled network is faster than traditional inverse solvers and achieves higher reconstruction quality by accounting for model mismatch.

APPROACH

Our model "Unrolled-DOT" uses the learned iterative shrinkage thresholding algorithm. In addition, we incorporate a refinement U-Net and Visual Geometry Group (VGG) perceptual loss to further increase the reconstruction quality. We trained and tested our model on simulated and real-world data and benchmarked against physics-based and learning-based inverse solvers.

RESULTS

In experiments on real-world data, Unrolled-DOT outperformed learning-based algorithms and achieved over 10× reduction in runtime and mean-squared error, compared to traditional physics-based solvers.

CONCLUSION

We demonstrated a learning-based ToF-DOT inverse solver that achieves state-of-the-art performance in speed and reconstruction quality, which can aid in future applications for noninvasive biomedical imaging.

Deterministic Terahertz Wave Control in Scattering Media

Scattering-assisted synthesis of broadband optical pulses is recognized to have a cross-disciplinary fundamental and application importance. Achieving full-waveform synthesis generally requires means for assessing the instantaneous electric field, i.e., the absolute electromagnetic phase. These are generally not accessible to established methodologies for scattering-assisted pulse envelope and phase shaping. The lack of field sensitivity also results in complex indirect approaches to evaluate the scattering space-time properties. The terahertz frequency domain potentially offers some distinctive new possibilities, thanks to the availability of methods to perform absolute measurements of the scattered electric field, as opposed to optical intensity-based diagnostics. An interesting conceptual question is whether this additional degree of freedom can lead to different types of methodologies toward wave shaping and direct field-waveform control. In this work, we theoretically investigate a deterministic scheme to achieve broadband, spatiotemporal waveform control of terahertz fields mediated by a scattering medium. Direct field access via time-domain spectroscopy enables a process in which the field and scattering matrix of the medium are assessed with minimal experimental efforts. Then, illumination conditions for an arbitrary targeted output field waveform are deterministically retrieved through numerical inversion. In addition, complete field knowledge enables reconstructing field distributions with complex phase profiles, as in the case of phase-only masks and optical vortices, a significantly challenging task for traditional implementations at optical frequencies based on intensity measurements aided with interferometric techniques.

Imaging Dynamics Beneath Turbid Media via Parallelized Single-Photon Detection

Noninvasive optical imaging through dynamic scattering media has numerous important biomedical applications but still remains a challenging task. While standard diffuse imaging methods measure optical absorption or fluorescent emission, it is also well-established that the temporal correlation of scattered coherent light diffuses through tissue much like optical intensity. Few works to date, however, have aimed to experimentally measure and process such temporal correlation data to demonstrate deep-tissue video reconstruction of decorrelation dynamics. In this work, a single-photon avalanche diode array camera is utilized to simultaneously monitor the temporal dynamics of speckle fluctuations at the single-photon level from 12 different phantom tissue surface locations delivered via a customized fiber bundle array. Then a deep neural network is applied to convert the acquired single-photon measurements into video of scattering dynamics beneath rapidly decorrelating tissue phantoms. The ability to reconstruct images of transient (0.1-0.4 s) dynamic events occurring up to 8 mm beneath a decorrelating tissue phantom with millimeter-scale resolution is demonstrated, and it is highlighted how the model can flexibly extend to monitor flow speed within buried phantom vessels.

High resolution TCSPC imaging of diffuse light with a one-dimensional SPAD array scanning system

We report a time-correlated single-photon counting (TCSPC) imaging system based on a line-scanning architecture. The system benefits from the high fill-factor, active area, and large dimension of an advanced CMOS single-photon avalanche diode (SPAD) array line-sensor. A two-dimensional image is constructed using a moving mirror to scan the line-sensor field-of-view (FOV) across the target, to enable the efficient acquisition of a two-dimensional 0.26 Mpixel TCSPC image. We demonstrate the capabilities of the system for TCSPC imaging and locating objects obscured in scattering media - specifically to locate a series of discrete point sources of light along an optical fibre submerged in a highly scattering solution. We demonstrate that by selectively imaging using early arriving photons which have undergone less scattering than later arriving photons, our TCSPC imaging system is able to locate the position of discrete point sources of light than a non-time-resolved imaging system.

A boundary migration model for imaging within volumetric scattering media

Effectively imaging within volumetric scattering media is of great importance and challenging especially in macroscopic applications. Recent works have demonstrated the ability to image through scattering media or within the weak volumetric scattering media using spatial distribution or temporal characteristics of the scattered field. Here, we focus on imaging Lambertian objects embedded in highly scattering media, where signal photons are dramatically attenuated during propagation and highly coupled with background photons. We address these challenges by providing a time-to-space boundary migration model (BMM) of the scattered field to convert the scattered measurements in spectral form to the scene information in the temporal domain using all of the optical signals. The experiments are conducted under two typical scattering scenarios: 2D and 3D Lambertian objects embedded in the polyethylene foam and the fog, which demonstrate the effectiveness of the proposed algorithm. It outperforms related works including time gating in terms of reconstruction precision and scattering strength. Even though the proportion of signal photons is only 0.75%, Lambertian objects located at more than 25 transport mean free paths (TMFPs), corresponding to the round-trip scattering length of more than 50 TMFPs, can be reconstructed. Also, the proposed method provides low reconstruction complexity and millisecond-scale runtime, which significantly benefits its application.

Imaging in scattering media is challenging due to signal attenuation and strong coupling of scattered and signal photons. The authors present a boundary migration model of the scattered field, converting scattered measurements in spectral form to scene information in temporal domain, and image Lambertian objects in highly scattering media.

Optical reciprocity induced wavefront shaping for axial and lateral shifting of focus through a scattering medium

Light propagating along a reversed path experiences the same transmission coefficient as in the forward direction, independent of the path complexity. This is called the optical reciprocity of light, which is valid for not too intense scattering media as well. Hence, by utilizing the reciprocity principle, the proposed novel technique can achieve axially and laterally tunable focus, non-invasively, through a scattering media without a priori knowledge or modeling of its scattering properties. Moreover, the uniqueness of the proposed technique lies in the fact that the illumination and detection are on the same side of the scattering media.

Model Study of Transient Imaging With Multi-Frequency Time-of-Flight Sensors

As an emerging imaging modality, transient imaging that records the transient information of light transport has significantly shaped our understanding of scenes. In spite of the great progress made in computer vision and optical imaging fields, commonly used multi-frequency time-of-flight (ToF) sensors are still afflicted with the band-limited modulation frequency and long acquisition process. To overcome such barriers, more effective image-formation schemes and reconstruction algorithms are highly desired. In this paper, we propose a compressive transient imaging model, without any priori knowledge, by constructing a near-tight-frame based representation of the ToF imaging principle. We prove that the compressibility of sensor measurements can be presented in the Fourier domain and held in the frame, and the ToF measurements possess multi-scale characteristics. Solving the inverse problems in transient imaging with our proposed model consists of two major steps, including a compressed-sensing-based approach for full measurement recovery, which essentially reduces the capture time, and a wavelet-based transient image reconstruction framework, which realizes adaptive transient image reconstruction and achieves highly accurate reconstruction results. The compressive transient imaging model is suitable for various existing multi-frequency ToF sensors and requires no hardware modifications. Experimental results using synthetic and real online datasets demonstrate its promising performance.

Compressed ultrafast tomographic imaging by passive spatiotemporal projections

Existing streak-camera-based two-dimensional (2D) ultrafast imaging techniques are limited by long acquisition time, the trade-off between spatial and temporal resolutions, and a reduced field of view. They also require additional components, customization, or active illumination. Here we develop compressed ultrafast tomographic imaging (CUTI), which passively records 2D transient events with a standard streak camera. By grafting the concept of computed tomography to the spatiotemporal domain, the operations of temporal shearing and spatiotemporal integration in a streak camera's data acquisition can be equivalently expressed as the spatiotemporal projection of an (x,y,t) datacube from a certain angle. Aided by a new, to the best of our knowledge, compressed-sensing reconstruction algorithm, the 2D transient event can be accurately recovered in a few measurements. CUTI is exhibited as a new imaging mode universally adaptable to most streak cameras. Implemented in an image-converter streak camera, CUTI captures the sequential arrival of two spatially modulated ultrashort ultraviolet laser pulses at 0.5 trillion frames per second. Applied to a rotating-mirror streak camera, CUTI records an amination of fast-bouncing balls at 5,000 frames per second.

Non-invasive single photon imaging through strongly scattering media

Non-invasive optical imaging through opaque and multi-scattering media remains highly desirable across many application domains. The random scattering and diffusion of light in such media inflict exponential decay and aberration, prohibiting diffraction-limited imaging. By non-interferometric few picoseconds optical gating of backscattered photons, we demonstrate single photon sensitive non-invasive 3D imaging of targets occluded by strongly scattering media with optical thicknesses reaching 9.5l_s (19l_s round trip). It achieves diffraction-limited imaging of a target placed 130 cm away through the opaque media, with millimeter lateral and depth resolution while requiring only one photon detection out of 50,000 probe pulses. Our single photon sensitive imaging technique does not require wavefront shaping nor computationally-intensive image reconstruction algorithms, promising practical solutions for diffraction-limited imaging through highly opaque and diffusive media with low illumination power.

Super-resolution time-resolved imaging using computational sensor fusion

Imaging across both the full transverse spatial and temporal dimensions of a scene with high precision in all three coordinates is key to applications ranging from LIDAR to fluorescence lifetime imaging. However, compromises that sacrifice, for example, spatial resolution at the expense of temporal resolution are often required, in particular when the full 3-dimensional data cube is required in short acquisition times. We introduce a sensor fusion approach that combines data having low-spatial resolution but high temporal precision gathered with a single-photon-avalanche-diode (SPAD) array with data that has high spatial but no temporal resolution, such as that acquired with a standard CMOS camera. Our method, based on blurring the image on the SPAD array and computational sensor fusion, reconstructs time-resolved images at significantly higher spatial resolution than the SPAD input, upsampling numerical data by a factor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$12 \times 12$$\end{document}12×12, and demonstrating up to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$4 \times 4$$\end{document}4×4 upsampling of experimental data. We demonstrate the technique for both LIDAR applications and FLIM of fluorescent cancer cells. This technique paves the way to high spatial resolution SPAD imaging or, equivalently, FLIM imaging with conventional microscopes at frame rates accelerated by more than an order of magnitude.

Role of late photons in diffuse optical imaging

The ability to image through turbid media, such as organic tissues, is a highly attractive prospect for biological and medical imaging. This is challenging, however, due to the highly scattering properties of tissues which scramble the image information. The earliest photons that arrive at the detector are often associated with ballistic transmission, whilst the later photons are associated with complex paths due to multiple independent scattering events and are therefore typically considered to be detrimental to the final image formation process. In this work, we report on the importance of these highly diffuse, "late" photons for computational time-of-flight diffuse optical imaging. In thick scattering materials, >80 transport mean free paths, we provide evidence that including late photons in the inverse retrieval enhances the image reconstruction quality. We also show that the late photons alone have sufficient information to retrieve images of a similar quality to early photon gated data. This result emphasises the importance in the strongly diffusive regime of fully time-resolved imaging techniques.

Three-dimensional imaging through scattering media based on confocal diffuse tomography

Optical imaging techniques, such as light detection and ranging (LiDAR), are essential tools in remote sensing, robotic vision, and autonomous driving. However, the presence of scattering places fundamental limits on our ability to image through fog, rain, dust, or the atmosphere. Conventional approaches for imaging through scattering media operate at microscopic scales or require a priori knowledge of the target location for 3D imaging. We introduce a technique that co-designs single-photon avalanche diodes, ultra-fast pulsed lasers, and a new inverse method to capture 3D shape through scattering media. We demonstrate acquisition of shape and position for objects hidden behind a thick diffuser (≈6 transport mean free paths) at macroscopic scales. Our technique, confocal diffuse tomography, may be of considerable value to the aforementioned applications.

Cross-talk reduction in a multiplexed synchroscan streak camera with simultaneous calibration

The streak camera is a picosecond resolution photodetector with parallel input capability; however, the degree of multiplexing is limited by crosstalk and temporal uncertainty in the sweeping field. We introduced a fixed time delay between adjacent fibers to reduce crosstalk in the synchroscan mode. The fixed delay and a tunable electronic delay between the input pulse and the synchroscan unit allows robust separation modes between the streaks, while spatial and temporal nonlinearities can be calibrated in. The efficacy of the design is demonstrated through a 100-fold multiplexed confocal fluorescence lifetime imaging microscope in live cells.

Noninvasive three-dimensional imaging through scattering media by three-dimensional speckle correlation

We present a method for noninvasive three-dimensional imaging through scattering media by using a three-dimensional memory effect in scattering phenomena. In the proposed method, an object in a scattering medium is reconstructed from a three-dimensional autocorrelation of speckle images captured by axially scanning an image sensor, based on a three-dimensional phase retrieval algorithm. We experimentally demonstrated our method with a lensless setup by using a three-dimensionally printed object between diffusers.

A trillion frames per second: the techniques and applications of light-in-flight photography

Cameras capable of capturing videos at a trillion frames per second allow to freeze light in motion, a very counterintuitive capability when related to our everyday experience in which light appears to travel instantaneously. By combining this capability with computational imaging techniques, new imaging opportunities emerge such as 3D imaging of scenes that are hidden behind a corner, the study of relativistic distortion effects, imaging through diffusive media and imaging of ultrafast optical processes such as laser ablation, supercontinuum and plasma generation. We provide an overview of the main techniques that have been developed for ultra-high speed photography with a particular focus on 'light-in-flight' imaging, i.e. applications where the key element is the imaging of light itself at frame rates that allow to freeze its motion and therefore extract information that would otherwise be blurred out and lost.

Review of structured light in diffuse optical imaging

Diffuse optical imaging probes deep living tissue enabling structural, functional, metabolic, and molecular imaging. Recently, due to the availability of spatial light modulators, wide-field quantitative diffuse optical techniques have been implemented, which benefit greatly from structured light methodologies. Such implementations facilitate the quantification and characterization of depth-resolved optical and physiological properties of thick and deep tissue at fast acquisition speeds. We summarize the current state of work and applications in the three main techniques leveraging structured light: spatial frequency-domain imaging, optical tomography, and single-pixel imaging. The theory, measurement, and analysis of spatial frequency-domain imaging are described. Then, advanced theories, processing, and imaging systems are summarized. Preclinical and clinical applications on physiological measurements for guidance and diagnosis are summarized. General theory and method development of tomographic approaches as well as applications including fluorescence molecular tomography are introduced. Lastly, recent developments of single-pixel imaging methodologies and applications are reviewed.

Polarimetric image recovery method combining histogram stretching for underwater imaging

The underwater imaging could be severely degraded by the scattering media because of the backscattered light and signal attenuation, especially in the case of strong scattering for dense turbid medium. In this paper, we propose an improved method for recovering the underwater image combining the histogram stretching and polarimetric recovery in a proper way. In this method, we stretch the histograms of the orthogonal polarization images while maintaining the polarization relation between them, and then, based on the processed orthogonal polarization images, the recovered image with higher quality can be obtained by the traditional polarimetric recovery method. Several groups of experimental results demonstrate that the quality of underwater images can be effectively enhanced by our method, and its performance is better than that of the traditional polarimetric recovery method. In particular, the proposed method is also quite effective in the condition of dense turbid medium.

Efficient illumination angle self-calibration in Fourier ptychography

Fourier ptychography captures intensity images with varying source patterns (illumination angles) in order to computationally reconstruct large space-bandwidth-product images. Accurate knowledge of the illumination angles is necessary for good image quality; hence, calibration methods are crucial, despite often being impractical or slow. Here, we propose a fast, robust, and accurate self-calibration algorithm that uses only experimentally collected data and general knowledge of the illumination setup. First, our algorithm makes a fast direct estimate of the brightfield illumination angles based on image processing. Then, a more computationally intensive spectral correlation method is used inside the iterative solver to further refine the angle estimates of both brightfield and darkfield images. We demonstrate our method for correcting large and small misalignment artifacts in 2D and 3D Fourier ptychography with different source types: an LED array, a galvo-steered laser, and a high-NA quasi-dome LED illuminator.

Ballistic and snake photon imaging for locating optical endomicroscopy fibres

We demonstrate determination of the location of the distal-end of a fibre-optic device deep in tissue through the imaging of ballistic and snake photons using a time resolved single-photon detector array. The fibre was imaged with centimetre resolution, within clinically relevant settings and models. This technique can overcome the limitations imposed by tissue scattering in optically determining the in vivo location of fibre-optic medical instruments.

A new Monte Carlo code for light transport in biological tissue

The aim of this work was to develop an event-by-event Monte Carlo code for light transport (called MCLTmx) to identify and quantify ballistic, diffuse, and absorbed photons, as well as their interaction coordinates inside the biological tissue. The mean free path length was computed between two interactions for scattering or absorption processes, and if necessary scatter angles were calculated, until the photon disappeared or went out of region of interest. A three-layer array (air-tissue-air) was used, forming a semi-infinite sandwich. The light source was placed at (0,0,0), emitting towards (0,0,1). The input data were: refractive indices, target thickness (0.02, 0.05, 0.1, 0.5, and 1 cm), number of particle histories, and λ from which the code calculated: anisotropy, scattering, and absorption coefficients. Validation presents differences less than 0.1% compared with that reported in the literature. The MCLTmx code discriminates between ballistic and diffuse photons, and inside of biological tissue, it calculates: specular reflection, diffuse reflection, ballistics transmission, diffuse transmission and absorption, and all parameters dependent on wavelength and thickness. The MCLTmx code can be useful for light transport inside any medium by changing the parameters that describe the new medium: anisotropy, dispersion and attenuation coefficients, and refractive indices for specific wavelength.

Object classification through scattering media with deep learning on time resolved measurement

We demonstrate an imaging technique that allows identification and classification of objects hidden behind scattering media and is invariant to changes in calibration parameters within a training range. Traditional techniques to image through scattering solve an inverse problem and are limited by the need to tune a forward model with multiple calibration parameters (like camera field of view, illumination position etc.). Instead of tuning a forward model and directly inverting the optical scattering, we use a data driven approach and leverage convolutional neural networks (CNN) to learn a model that is invariant to calibration parameters variations within the training range and nearly invariant beyond that. This effectively allows robust imaging through scattering conditions that is not sensitive to calibration. The CNN is trained with a large synthetic dataset generated with a Monte Carlo (MC) model that contains random realizations of major calibration parameters. The method is evaluated with a time-resolved camera and multiple experimental results are provided including pose estimation of a mannequin hidden behind a paper sheet with 23 correct classifications out of 30 tests in three poses (76.6% accuracy on real-world measurements). This approach paves the way towards real-time practical non line of sight (NLOS) imaging applications.