Inference in artificial intelligence with deep optics and photonics

Thin film ferroelectric photonic-electronic memory

To reduce system complexity and bridge the interface between electronic and photonic circuits, there is a high demand for a non-volatile memory that can be accessed both electrically and optically. However, practical solutions are still lacking when considering the potential for large-scale complementary metal-oxide semiconductor compatible integration. Here, we present an experimental demonstration of a non-volatile photonic-electronic memory based on a 3-dimensional monolithic integrated ferroelectric-silicon ring resonator. We successfully demonstrate programming and erasing the memory using both electrical and optical methods, assisted by optical-to-electrical-to-optical conversion. The memory cell exhibits a high optical extinction ratio of 6.6 dB at a low working voltage of 5 V and an endurance of 4 × 104 cycles. Furthermore, the multi-level storage capability is analyzed in detail, revealing stable performance with a raw bit-error-rate smaller than 5.9 × 10-2. This ground-breaking work could be a key technology enabler for future hybrid electronic-photonic systems, targeting a wide range of applications such as photonic interconnect, high-speed data communication, and neuromorphic computing.

Terahertz spoof plasmonic neural network for diffractive information recognition and processing

All-optical diffractive neural networks, as analog artificial intelligence accelerators, leverage parallelism and analog computation for complex data processing. However, their low space transmission efficiency or large spatial dimensions hinder miniaturization and broader application. Here, we propose a terahertz spoof plasmonic neural network on a planar diffractive platform for direct multi-target recognition. Our approach employs a spoof surface plasmon polariton coupler array to construct a diffractive network layer, resulting in a compact, efficient, and easily integrable architecture. We designed three schemes: basis vector classification, multi-user recognition, and MNIST handwritten digit classification. Experimental results reveal that the terahertz spoof plasmonic neural network successfully classifies basis vectors, recognizes multi-user orientation information, and directly processes handwritten digits using a designed input framework comprising a metal grating array, transmitters, and receivers. This work broadens the application of terahertz plasmonic metamaterials, paving the way for terahertz on-chip integration, intelligent communication, and advanced computing systems.

Curriculum learning for ab initio deep learned refractive optics

Deep optical optimization has recently emerged as a new paradigm for designing computational imaging systems using only the output image as the objective. However, it has been limited to either simple optical systems consisting of a single element such as a diffractive optical element or metalens, or the fine-tuning of compound lenses from good initial designs. Here we present a DeepLens design method based on curriculum learning, which is able to learn optical designs of compound lenses ab initio from randomly initialized surfaces without human intervention, therefore overcoming the need for a good initial design. We demonstrate the effectiveness of our approach by fully automatically designing both classical imaging lenses and a large field-of-view extended depth-of-field computational lens in a cellphone-style form factor, with highly aspheric surfaces and a short back focal length.

Fully forward mode training for optical neural networks

Optical computing promises to improve the speed and energy efficiency of machine learning applications1-6. However, current approaches to efficiently train these models are limited by in silico emulation on digital computers. Here we develop a method called fully forward mode (FFM) learning, which implements the compute-intensive training process on the physical system. The majority of the machine learning operations are thus efficiently conducted in parallel on site, alleviating numerical modelling constraints. In free-space and integrated photonics, we experimentally demonstrate optical systems with state-of-the-art performances for a given network size. FFM learning shows training the deepest optical neural networks with millions of parameters achieves accuracy equivalent to the ideal model. It supports all-optical focusing through scattering media with a resolution of the diffraction limit; it can also image in parallel the objects hidden outside the direct line of sight at over a kilohertz frame rate and can conduct all-optical processing with light intensity as weak as subphoton per pixel (5.40 × 1018- operations-per-second-per-watt energy efficiency) at room temperature. Furthermore, we prove that FFM learning can automatically search non-Hermitian exceptional points without an analytical model. FFM learning not only facilitates orders-of-magnitude-faster learning processes, but can also advance applied and theoretical fields such as deep neural networks, ultrasensitive perception and topological photonics.

Partial coherence enhances parallelized photonic computing

Advancements in optical coherence control1-5 have unlocked many cutting-edge applications, including long-haul communication, light detection and ranging (LiDAR) and optical coherence tomography6-8. Prevailing wisdom suggests that using more coherent light sources leads to enhanced system performance and device functionalities9-11. Our study introduces a photonic convolutional processing system that takes advantage of partially coherent light to boost computing parallelism without substantially sacrificing accuracy, potentially enabling larger-size photonic tensor cores. The reduction of the degree of coherence optimizes bandwidth use in the photonic convolutional processing system. This breakthrough challenges the traditional belief that coherence is essential or even advantageous in integrated photonic accelerators, thereby enabling the use of light sources with less rigorous feedback control and thermal-management requirements for high-throughput photonic computing. Here we demonstrate such a system in two photonic platforms for computing applications: a photonic tensor core using phase-change-material photonic memories that delivers parallel convolution operations to classify the gaits of ten patients with Parkinson's disease with 92.2% accuracy (92.7% theoretically) and a silicon photonic tensor core with embedded electro-absorption modulators (EAMs) to facilitate 0.108 tera operations per second (TOPS) convolutional processing for classifying the Modified National Institute of Standards and Technology (MNIST) handwritten digits dataset with 92.4% accuracy (95.0% theoretically).

Pyramid diffractive optical networks for unidirectional image magnification and demagnification

Diffractive deep neural networks (D2NNs) are composed of successive transmissive layers optimized using supervised deep learning to all-optically implement various computational tasks between an input and output field-of-view. Here, we present a pyramid-structured diffractive optical network design (which we term P-D2NN), optimized specifically for unidirectional image magnification and demagnification. In this design, the diffractive layers are pyramidally scaled in alignment with the direction of the image magnification or demagnification. This P-D2NN design creates high-fidelity magnified or demagnified images in only one direction, while inhibiting the image formation in the opposite direction-achieving the desired unidirectional imaging operation using a much smaller number of diffractive degrees of freedom within the optical processor volume. Furthermore, the P-D2NN design maintains its unidirectional image magnification/demagnification functionality across a large band of illumination wavelengths despite being trained with a single wavelength. We also designed a wavelength-multiplexed P-D2NN, where a unidirectional magnifier and a unidirectional demagnifier operate simultaneously in opposite directions, at two distinct illumination wavelengths. Furthermore, we demonstrate that by cascading multiple unidirectional P-D2NN modules, we can achieve higher magnification factors. The efficacy of the P-D2NN architecture was also validated experimentally using terahertz illumination, successfully matching our numerical simulations. P-D2NN offers a physics-inspired strategy for designing task-specific visual processors.

Advancements and Challenges in the Integration of Indium Arsenide and Van der Waals Heterostructures

The strategic integration of low-dimensional InAs-based materials and emerging van der Waals systems is advancing in various scientific fields, including electronics, optics, and magnetics. With their unique properties, these InAs-based van der Waals materials and devices promise further miniaturization of semiconductor devices in line with Moore's Law. However, progress in this area lags behind other 2D materials like graphene and boron nitride. Challenges include synthesizing pure crystalline phase InAs nanostructures and single-atomic-layer 2D InAs films, both vital for advanced van der Waals heterostructures. Also, diverse surface state effects on InAs-based van der Waals devices complicate their performance evaluation. This review discusses the experimental advances in the van der Waals epitaxy of InAs-based materials and the working principles of InAs-based van der Waals devices. Theoretical achievements in understanding and guiding the design of InAs-based van der Waals systems are highlighted. Focusing on advancing novel selective area growth and remote epitaxy, exploring multi-functional applications, and incorporating deep learning into first-principles calculations are proposed. These initiatives aim to overcome existing bottlenecks and accelerate transformative advancements in integrating InAs and van der Waals heterostructures.

Super-resolution diffractive neural network for all-optical direction of arrival estimation beyond diffraction limits

Wireless sensing of the wave propagation direction from radio sources lays the foundation for communication, radar, navigation, etc. However, the existing signal processing paradigm for the direction of arrival estimation requires the radio frequency electronic circuit to demodulate and sample the multichannel baseband signals followed by a complicated computing process, which places the fundamental limit on its sensing speed and energy efficiency. Here, we propose the super-resolution diffractive neural networks (S-DNN) to process electromagnetic (EM) waves directly for the DOA estimation at the speed of light. The multilayer meta-structures of S-DNN generate super-oscillatory angular responses in local angular regions that can perform the all-optical DOA estimation with angular resolutions beyond the diffraction limit. The spatial-temporal multiplexing of passive and reconfigurable S-DNNs is utilized to achieve high-resolution DOA estimation over a wide field of view. The S-DNN is validated for the DOA estimation of multiple radio sources over 5 GHz frequency bandwidth with estimation latency over two to four orders of magnitude lower than the state-of-the-art commercial devices in principle. The results achieve the angular resolution over an order of magnitude, experimentally demonstrated with four times, higher than diffraction-limited resolution. We also apply S-DNN's edge computing capability, assisted by reconfigurable intelligent surfaces, for extremely low-latency integrated sensing and communication with low power consumption. Our work is a significant step towards utilizing photonic computing processors to facilitate various wireless sensing and communication tasks with advantages in both computing paradigms and performance over electronic computing.

Non-convex optimization for inverse problem solving in computer-generated holography

Computer-generated holography is a promising technique that modulates user-defined wavefronts with digital holograms. Computing appropriate holograms with faithful reconstructions is not only a problem closely related to the fundamental basis of holography but also a long-standing challenge for researchers in general fields of optics. Finding the exact solution of a desired hologram to reconstruct an accurate target object constitutes an ill-posed inverse problem. The general practice of single-diffraction computation for synthesizing holograms can only provide an approximate answer, which is subject to limitations in numerical implementation. Various non-convex optimization algorithms are thus designed to seek an optimal solution by introducing different constraints, frameworks, and initializations. Herein, we overview the optimization algorithms applied to computer-generated holography, incorporating principles of hologram synthesis based on alternative projections and gradient descent methods. This is aimed to provide an underlying basis for optimized hologram generation, as well as insights into the cutting-edge developments of this rapidly evolving field for potential applications in virtual reality, augmented reality, head-up display, data encryption, laser fabrication, and metasurface design.

Nanowatt all-optical 3D perception for mobile robotics

Three-dimensional (3D) perception is vital to drive mobile robotics' progress toward intelligence. However, state-of-the-art 3D perception solutions require complicated postprocessing or point-by-point scanning, suffering computational burden, latency of tens of milliseconds, and additional power consumption. Here, we propose a parallel all-optical computational chipset 3D perception architecture (Aop3D) with nanowatt power and light speed. The 3D perception is executed during the light propagation over the passive chipset, and the captured light intensity distribution provides a direct reflection of the depth map, eliminating the need for extensive postprocessing. The prototype system of Aop3D is tested in various scenarios and deployed to a mobile robot, demonstrating unprecedented performance in distance detection and obstacle avoidance. Moreover, Aop3D works at a frame rate of 600 hertz and a power consumption of 33.3 nanowatts per meta-pixel experimentally. Our work is promising toward next-generation direct 3D perception techniques with light speed and high energy efficiency.

The Application Potential of Artificial Intelligence and Numerical Simulation in the Research and Formulation Design of Drilling Fluid Gel Performance

Drilling fluid is pivotal for efficient drilling. However, the gelation performance of drilling fluids is influenced by various complex factors, and traditional methods are inefficient and costly. Artificial intelligence and numerical simulation technologies have become transformative tools in various disciplines. This work reviews the application of four artificial intelligence techniques-expert systems, artificial neural networks (ANNs), support vector machines (SVMs), and genetic algorithms-and three numerical simulation techniques-computational fluid dynamics (CFD) simulations, molecular dynamics (MD) simulations, and Monte Carlo simulations-in drilling fluid design and performance optimization. It analyzes the current issues in these studies, pointing out that challenges in applying these two technologies to drilling fluid gelation performance research include difficulties in obtaining field data and overly idealized model assumptions. From the literature review, it can be estimated that 52.0% of the papers are related to ANNs. Leakage issues are the primary concern for practitioners studying drilling fluid gelation performance, accounting for over 17% of research in this area. Based on this, and in conjunction with the technical requirements of drilling fluids and the development needs of drilling intelligence theory, three development directions are proposed: (1) Emphasize feature engineering and data preprocessing to explore the application potential of interpretable artificial intelligence. (2) Establish channels for open access to data or large-scale oil and gas field databases. (3) Conduct in-depth numerical simulation research focusing on the microscopic details of the spatial network structure of drilling fluids, reducing or even eliminating data dependence.

Advancements in fluorescence lifetime imaging microscopy Instrumentation: Towards high speed and 3D

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging tool offering molecular specific insights into samples through the measurement of fluorescence decay time, with promising applications in diverse research fields. However, to acquire two-dimensional lifetime images, conventional FLIM relies on extensive scanning in both the spatial and temporal domain, resulting in much slower acquisition rates compared to intensity-based approaches. This problem is further magnified in three-dimensional imaging, as it necessitates additional scanning along the depth axis. Recent advancements have aimed to enhance the speed and three-dimensional imaging capabilities of FLIM. This review explores the progress made in addressing these challenges and discusses potential directions for future developments in FLIM instrumentation.

Large-scale photonic computing with nonlinear disordered media

Neural networks find widespread use in scientific and technological applications, yet their implementations in conventional computers have encountered bottlenecks due to ever-expanding computational needs. Photonic computing is a promising neuromorphic platform with potential advantages of massive parallelism, ultralow latency and reduced energy consumption but mostly for computing linear operations. Here we demonstrate a large-scale, high-performance nonlinear photonic neural system based on a disordered polycrystalline slab composed of lithium niobate nanocrystals. Mediated by random quasi-phase-matching and multiple scattering, linear and nonlinear optical speckle features are generated as the interplay between the simultaneous linear random scattering and the second-harmonic generation, defining a complex neural network in which the second-order nonlinearity acts as internal nonlinear activation functions. Benchmarked against linear random projection, such nonlinear mapping embedded with rich physical computational operations shows improved performance across a large collection of machine learning tasks in image classification, regression and graph classification. Demonstrating up to 27,648 input and 3,500 nonlinear output nodes, the combination of optical nonlinearity and random scattering serves as a scalable computing engine for diverse applications.

Integrated photonic encoder for low power and high-speed image processing

Modern lens designs are capable of resolving greater than 10 gigapixels, while advances in camera frame-rate and hyperspectral imaging have made data acquisition rates of Terapixel/second a real possibility. The main bottlenecks preventing such high data-rate systems are power consumption and data storage. In this work, we show that analog photonic encoders could address this challenge, enabling high-speed image compression using orders-of-magnitude lower power than digital electronics. Our approach relies on a silicon-photonics front-end to compress raw image data, foregoing energy-intensive image conditioning and reducing data storage requirements. The compression scheme uses a passive disordered photonic structure to perform kernel-type random projections of the raw image data with minimal power consumption and low latency. A back-end neural network can then reconstruct the original images with structural similarity exceeding 90%. This scheme has the potential to process data streams exceeding Terapixel/second using less than 100 fJ/pixel, providing a path to ultra-high-resolution data and image acquisition systems.

Multiplexable all-optical nonlinear activator for optical computing

As an alternative solution to surpass electronic neural networks, optical neural networks (ONNs) offer significant advantages in terms of energy consumption and computing speed. Despite the optical hardware platform could provide an efficient approach to realizing neural network algorithms than traditional hardware, the lack of optical nonlinearity limits the development of ONNs. Here, we proposed and experimentally demonstrated an all-optical nonlinear activator based on the stimulated Brillouin scattering (SBS). Utilizing the exceptional carrier dynamics of SBS, our activator supports two types of nonlinear functions, saturable absorption and rectified linear unit (Relu) models. Moreover, the proposed activator exhibits large dynamic response bandwidth (∼11.24 GHz), low nonlinear threshold (∼2.29 mW), high stability, and wavelength division multiplexing identities. These features have potential advantages for the physical realization of optical nonlinearities. As a proof of concept, we verify the performance of the proposed activator as an ONN nonlinear mapping unit via numerical simulations. Simulation shows that our approach achieves comparable performance to the activation functions commonly used in computers. The proposed approach provides support for the realization of all-optical neural networks.

Monolayer directional metasurface for all-optical image classifier doublet

Diffractive deep neural networks, known for their passivity, high scalability, and high efficiency, offer great potential in holographic imaging, target recognition, and object classification. However, previous endeavors have been hampered by spatial size and alignment. To address these issues, this study introduces a monolayer directional metasurface, aimed at reducing spatial constraints and mitigating alignment issues. Utilizing this methodology, we use MNIST datasets to train diffractive deep neural networks and realize digital classification, revealing that the metasurface can achieve excellent digital image classification results, and the classification accuracy of ideal phase mask plates and metasurface for phase-only modulation can reach 84.73% and 84.85%, respectively. Despite a certain loss of degrees of freedom compared to multi-layer phase mask plates, the single-layer metasurface is easier to fabricate and align, thereby improving spatial utilization efficiency.

An optoacoustic field-programmable perceptron for recurrent neural networks

Recurrent neural networks (RNNs) can process contextual information such as time series signals and language. But their tracking of internal states is a limiting factor, motivating research on analog implementations in photonics. While photonic unidirectional feedforward neural networks (NNs) have demonstrated big leaps, bi-directional optical RNNs present a challenge: the need for a short-term memory that (i) programmable and coherently computes optical inputs, (ii) minimizes added noise, and (iii) allows scalability. Here, we experimentally demonstrate an optoacoustic recurrent operator (OREO) which meets (i, ii, iii). OREO contextualizes the information of an optical pulse sequence via acoustic waves. The acoustic waves link different optical pulses, capturing their information and using it to manipulate subsequent operations. OREO's all-optical control on a pulse-by-pulse basis offers simple reconfigurability and is used to implement a recurrent drop-out and pattern recognition of 27 optical pulse patterns. Finally, we introduce OREO as bi-directional perceptron for new classes of optical NNs.

Photorefraction Simulates Well the Plasticity of Neural Synaptic Connections

In recent years, the need for systems capable of achieving the dynamic learning and information storage efficiency of the biological brain has led to the emergence of neuromorphic research. In particular, neuromorphic optics was born with the idea of reproducing the functional and structural properties of the biological brain. In this context, solitonic neuromorphic research has demonstrated the ability to reproduce dynamic and plastic structures capable of learning and storing through conformational changes in the network. In this paper, we demonstrate that solitonic neural networks are capable of mimicking the functional behaviour of biological neural tissue, in terms of synaptic formation procedures and dynamic reinforcement.

Multichannel meta-imagers for accelerating machine vision

Rapid developments in machine vision technology have impacted a variety of applications, such as medical devices and autonomous driving systems. These achievements, however, typically necessitate digital neural networks with the downside of heavy computational requirements and consequent high energy consumption. As a result, real-time decision-making is hindered when computational resources are not readily accessible. Here we report a meta-imager designed to work together with a digital back end to offload computationally expensive convolution operations into high-speed, low-power optics. In this architecture, metasurfaces enable both angle and polarization multiplexing to create multiple information channels that perform positively and negatively valued convolution operations in a single shot. We use our meta-imager for object classification, achieving 98.6% accuracy in handwritten digits and 88.8% accuracy in fashion images. Owing to its compactness, high speed and low power consumption, our approach could find a wide range of applications in artificial intelligence and machine vision applications.

Multi-channel broadband nonvolatile programmable modal switch

Mode-division multiplexing (MDM) in chip-scale photonics is paramount to sustain data capacity growth and reduce power consumption. However, its scalability hinges on developing efficient and dynamic modal switches. Existing active modal switches suffer from substantial static power consumption, large footprints, and narrow bandwidth. Here, we present, for the first time, to the best of our knowledge, a novel multiport, broadband, non-volatile, and programmable modal switch designed for on-chip MDM systems. Our design leverages the unique properties of integrating nanoscale phase-change materials (PCM) within a silicon photonic architecture. This enables independent manipulation of spatial modes, allowing for dynamic, non-volatile, and selective routing to six distinct output ports. Crucially, our switch outperforms current dynamic modal switches by offering non-volatile, energy-efficient multiport functionality and excels in performance metrics. Our switch exhibits exceptional broadband operating bandwidth exceeding 70 nm, with low loss (< 1 dB), and a high extinction ratio (> 10 dB). Our framework provides a step forward in chip-scale MDM, paving the way for future green and scalable data centers and high-performance computers.

Cross-layer transmission realized by light-emitting memristor for constructing ultra-deep neural network with transfer learning ability

Deep neural networks have revolutionized several domains, including autonomous driving, cancer detection, and drug design, and are the foundation for massive artificial intelligence models. However, hardware neural network reports still mainly focus on shallow networks (2 to 5 layers). Implementing deep neural networks in hardware is challenging due to the layer-by-layer structure, resulting in long training times, signal interference, and low accuracy due to gradient explosion/vanishing. Here, we utilize negative ultraviolet photoconductive light-emitting memristors with intrinsic parallelism and hardware-software co-design to achieve electrical information's optical cross-layer transmission. We propose a hybrid ultra-deep photoelectric neural network and an ultra-deep super-resolution reconstruction neural network using light-emitting memristors and cross-layer block, expanding the networks to 54 and 135 layers, respectively. Further, two networks enable transfer learning, approaching or surpassing software-designed networks in multi-dataset recognition and high-resolution restoration tasks. These proposed strategies show great potential for high-precision multifunctional hardware neural networks and edge artificial intelligence.

Data-driven coordinated attention deep learning for high-fidelity brain imaging denoising and inpainting

Deep learning offers promise in enhancing low-quality images by addressing weak fluorescence signals, especially in deep in vivo mouse brain imaging. However, current methods struggle with photon scarcity and noise within in vivo deep mouse brains, and often neglecting tissue preservation. In this study, we propose an innovative in vivo cortical fluorescence image restoration approach, combining signal enhancement, denoising, and inpainting. We curated a deep brain cortical image dataset and developed a novel deep brain coordinate attention restoration network (DeepCAR), integrating coordinate attention with optimized residual networks. Our method swiftly and accurately restores deep cortex images exceeding 800 μm, preserving small-scale tissue structures. It boosts the peak signal-to-noise ratio (PSNR) by 6.94 dB for weak signals and 11.22 dB for large noisy images. Crucially, we validate the effectiveness on external datasets with diverse noise distributions, structural features compared to those in our training data, showcasing real-time high-performance image restoration capabilities.

Structured light enhanced machine learning for fiber bend sensing

The intricate optical distortions that occur when light interacts with complex media, such as few- or multi-mode optical fiber, often appear random in origin and are a fundamental source of error for communication and sensing systems. We propose the use of orbital angular momentum (OAM) feature extraction to mitigate phase-noise and allow for the use of intermodal-coupling as an effective tool for fiber sensing. OAM feature extraction is achieved by passive all-optical OAM demultiplexing, and we demonstrate fiber bend tracking with 94.1% accuracy. Conversely, an accuracy of only 14% was achieved for determining the same bend positions when using a convolutional-neural-network trained with intensity measurements of the output of the fiber. Further, OAM feature extraction used 120 times less information for training compared to intensity image based measurements. This work indicates that structured light enhanced machine learning could be used in a wide range of future sensing technologies.

General-purpose programmable photonic processor for advanced radiofrequency applications

A general-purpose photonic processor can be built integrating a silicon photonic programmable core in a technology stack comprising an electronic monitoring and controlling layer and a software layer for resource control and programming. This processor can leverage the unique properties of photonics in terms of ultra-high bandwidth, high-speed operation, and low power consumption while operating in a complementary and synergistic way with electronic processors. These features are key in applications such as next-generation 5/6 G wireless systems where reconfigurable filtering, frequency conversion, arbitrary waveform generation, and beamforming are currently provided by microwave photonic subsystems that cannot be scaled down. Here we report the first general-purpose programmable processor with the remarkable capability to implement all the required basic functionalities of a microwave photonic system by suitable programming of its resources. The processor is fabricated in silicon photonics and incorporates the full photonic/electronic and software stack.

Diffractive optical computing in free space

Structured optical materials create new computing paradigms using photons, with transformative impact on various fields, including machine learning, computer vision, imaging, telecommunications, and sensing. This Perspective sheds light on the potential of free-space optical systems based on engineered surfaces for advancing optical computing. Manipulating light in unprecedented ways, emerging structured surfaces enable all-optical implementation of various mathematical functions and machine learning tasks. Diffractive networks, in particular, bring deep-learning principles into the design and operation of free-space optical systems to create new functionalities. Metasurfaces consisting of deeply subwavelength units are achieving exotic optical responses that provide independent control over different properties of light and can bring major advances in computational throughput and data-transfer bandwidth of free-space optical processors. Unlike integrated photonics-based optoelectronic systems that demand preprocessed inputs, free-space optical processors have direct access to all the optical degrees of freedom that carry information about an input scene/object without needing digital recovery or preprocessing of information. To realize the full potential of free-space optical computing architectures, diffractive surfaces and metasurfaces need to advance symbiotically and co-evolve in their designs, 3D fabrication/integration, cascadability, and computing accuracy to serve the needs of next-generation machine vision, computational imaging, mathematical computing, and telecommunication technologies.

Parallel edge extraction operators on chip speed up photonic convolutional neural networks

We experimentally establish a 3 × 3 cross-shaped micro-ring resonator (MRR) array-based photonic multiplexing architecture relying on silicon photonics to achieve parallel edge extraction operations in images for photonic convolution neural networks. The main mathematical operations involved are convolution. Precisely, a faster convolutional calculation speed of up to four times is achieved by extracting four feature maps simultaneously with the same photonic hardware's structure and power consumption, where a maximum computility of 0.742 TOPS at an energy cost of 48.6 mW and a convolution accuracy of 95.1% is achieved in an MRR array chip. In particular, our experimental results reveal that this system using parallel edge extraction operators instead of universal operators can improve the imaging recognition accuracy for CIFAR-10 dataset by 6.2% within the same computing time, reaching a maximum of 78.7%. This work presents high scalability and efficiency of parallel edge extraction chips, furnishing a novel, to the best of our knowledge, approach to boost photonic computing speed.

Wide-field color imaging through multimode fiber with single wavelength illumination: plug-and-play approach

Multimode fiber (MMF) is extensively studied for its ability to transmit light modes in parallel, potentially minimizing optical fiber size in imaging. However, current research predominantly focuses on grayscale imaging, with limited attention to color studies. Existing colorization methods often involve costly white light lasers or multiple light sources, increasing optical system expenses and space. To achieve wide-field color images with typical monochromatic illumination MMF imaging system, we proposed a data-driven "colorization" approach and a neural network called SpeckleColorNet, merging U-Net and conditional GAN (cGAN) architectures, trained by a combined loss function. This approach, demonstrated on a 2-meter MMF system with single-wavelength illumination and the Peripheral Blood Cell (PBC) dataset, outperforms grayscale imaging and alternative colorization methods in readability, definition, detail, and accuracy. Our method aims to integrate MMF into clinical medicine and industrial monitoring, offering cost-effective high-fidelity color imaging. It serves as a plug-and-play replacement for conventional grayscale algorithms in MMF systems, eliminating the need for additional hardware.

All-optical image denoising using a diffractive visual processor

Image denoising, one of the essential inverse problems, targets to remove noise/artifacts from input images. In general, digital image denoising algorithms, executed on computers, present latency due to several iterations implemented in, e.g., graphics processing units (GPUs). While deep learning-enabled methods can operate non-iteratively, they also introduce latency and impose a significant computational burden, leading to increased power consumption. Here, we introduce an analog diffractive image denoiser to all-optically and non-iteratively clean various forms of noise and artifacts from input images - implemented at the speed of light propagation within a thin diffractive visual processor that axially spans <250 × λ, where λ is the wavelength of light. This all-optical image denoiser comprises passive transmissive layers optimized using deep learning to physically scatter the optical modes that represent various noise features, causing them to miss the output image Field-of-View (FoV) while retaining the object features of interest. Our results show that these diffractive denoisers can efficiently remove salt and pepper noise and image rendering-related spatial artifacts from input phase or intensity images while achieving an output power efficiency of ~30-40%. We experimentally demonstrated the effectiveness of this analog denoiser architecture using a 3D-printed diffractive visual processor operating at the terahertz spectrum. Owing to their speed, power-efficiency, and minimal computational overhead, all-optical diffractive denoisers can be transformative for various image display and projection systems, including, e.g., holographic displays.

Imaging privacy threats from an ambient light sensor

Embedded sensors in smart devices pose privacy risks, often unintentionally leaking user information. We investigate how combining an ambient light sensor with a device display can capture an image of touch interaction without a camera. By displaying a known video sequence, we use the light sensor to capture reflected light intensity variations partially blocked by the touching hand, formulating an inverse problem similar to single-pixel imaging. Because of the sensors' heavy quantization and low sensitivity, we propose an inversion algorithm involving an ℓp-norm dequantizer and a deep denoiser as natural image priors, to reconstruct images from the screen's perspective. We demonstrate touch interactions and eavesdropping hand gestures on an off-the-shelf Android tablet. Despite limitations in resolution and speed, we aim to raise awareness of potential security/privacy threats induced by the combination of passive and active components in smart devices and promote the development of ways to mitigate them.

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

OP-FCNN: an optronic fully convolutional neural network for imaging through scattering media

Imaging through scattering media is a classical inverse issue in computational imaging. In recent years, deep learning(DL) methods have excelled in speckle reconstruction by extracting the correlation of speckle patterns. However, high-performance DL-based speckle reconstruction also costs huge hardware computation and energy consumption. Here, we develop an opto-electronic DL method with low computation complexity for imaging through scattering media. We design the "end-to-end" optronic structure for speckle reconstruction, namely optronic fully convolutional neural network (OP-FCNN). In OP-FCNN, we utilize lens groups and spatial light modulators to implement the convolution, down/up-sampling, and skip connection in optics, which significantly reduces the computational complexity by two orders of magnitude, compared with the digital CNN. Moreover, the reconfigurable and scalable structure supports the OP-FCNN to further improve imaging performance and accommodate object datasets of varying complexity. We utilize MNIST handwritten digits, EMNIST handwritten letters, fashion MNIST, and MIT-CBCL-face datasets to validate the OP-FCNN imaging performance through random diffusers. Our OP-FCNN reveals a good balance between computational complexity and imaging performance. The average imaging performance on four datasets achieves 0.84, 0.91, 0.79, and 16.3dB for JI, PCC, SSIM, and PSNR, respectively. The OP-FCNN paves the way for all-optical systems in imaging through scattering media.

Interfacial Effect on the Transient Dielectric Function and Charge Transfer in a Monolayer WS2/Si Heterojunction

Monolayer tungsten disulfide (WS2) is a highly promising material for silicon photonics. Thus, the WS2/Si interface plays a very important role due to the interfacial complex effects and abundant states. Among them, the effect of charge transfer on exciton dynamics and the optoelectronic property is determined by the dielectric function, which is very crucial for the performance of optoelectronic devices. However, research on the exciton dynamics or the transient dielectric function of WS2 in such WS2/Si junctions is still rare. In this work, both the transient dielectric function and charge transfer of WS2/Si heterojunctions are analyzed based on the transient reflectance spectra measured by the pump-probe spectrometer. The dynamic processes of the A exciton, affected by charge transfer within the WS2/Si heterojunction, are interpreted. Moreover, the transient dielectric function of WS2 is quantitatively analyzed. The dielectric function of WS2 exhibits a notable 19% change, persisting for more than 180 ps within the WS2/Si heterojunction. These findings can pave the way for the advancement of silicon photonic devices based on WS2.

Simulating an Integrated Photonic Image Classifier for Diffractive Neural Networks

The slowdown of Moore's law and the existence of the "von Neumann bottleneck" has led to electronic-based computing systems under von Neumann's architecture being unable to meet the fast-growing demand for artificial intelligence computing. However, all-optical diffractive neural networks provide a possible solution to this challenge. They can outperform conventional silicon-based electronic neural networks due to the significantly higher speed of the propagation of optical signals (≈108 m.s-1) compared to electrical signals (≈105 m.s-1), their parallelism in nature, and their low power consumption. The integrated diffractive deep neural network (ID2NN) uses an on-chip fully passive photonic approach to achieve the functionality of neural networks (matrix-vector operations) and can be fabricated via the CMOS process, which is technologically more amenable to implementing an artificial intelligence processor. In this paper, we present a detailed design framework for the integrated diffractive deep neural network and corresponding silicon-on-insulator integration implementation through Python-based simulations. The performance of our proposed ID2NN was evaluated by solving image classification problems using the MNIST dataset.

Quantitative comparison of the computational complexity of optical, digital and hybrid neural network architectures for image classification tasks

By implementing neuromorphic paradigms in processing visual information, machine learning became crucial in an ever-increasing number of applications of our everyday lives, ever more performing but also computationally demanding. While a pre-processing of the information passively in the optical domain, before optical-electronic conversion, can reduce the computational requirements for a machine learning task, a comprehensive analysis of computational requirements for hybrid optical-digital neural networks is thus far missing. In this work we critically compare and analyze the performance of different optical, digital and hybrid neural network architectures with respect to their classification accuracy and computational requirements for analog classification tasks of different complexity. We show that certain hybrid architectures exhibit a reduction of computational requirements of a factor >10 while maintaining their performance. This may inspire a new generation of co-designed optical-digital neural network architectures, aimed for applications that require low power consumption like remote sensing devices.

Neuromorphic Computing via Fission-based Broadband Frequency Generation

The performance limitations of traditional computer architectures have led to the rise of brain-inspired hardware, with optical solutions gaining popularity due to the energy efficiency, high speed, and scalability of linear operations. However, the use of optics to emulate the synaptic activity of neurons has remained a challenge since the integration of nonlinear nodes is power-hungry and, thus, hard to scale. Neuromorphic wave computing offers a new paradigm for energy-efficient information processing, building upon transient and passively nonlinear interactions between optical modes in a waveguide. Here, an implementation of this concept is presented using broadband frequency conversion by coherent higher-order soliton fission in a single-mode fiber. It is shown that phase encoding on femtosecond pulses at the input, alongside frequency selection and weighting at the system output, makes transient spectro-temporal system states interpretable and allows for the energy-efficient emulation of various digital neural networks. The experiments in a compact, fully fiber-integrated setup substantiate an anticipated enhancement in computational performance with increasing system nonlinearity. The findings suggest that broadband frequency generation, accessible on-chip and in-fiber with off-the-shelf components, may challenge the traditional approach to node-based brain-inspired hardware design, ultimately leading to energy-efficient, scalable, and dependable computing with minimal optical hardware requirements.

Overlapping speckle correlation algorithm for high-resolution imaging and tracking of objects in unknown scattering media

Optical imaging in scattering media is important to many fields but remains challenging. Recent methods have focused on imaging through thin scattering layers or thicker scattering media with prior knowledge of the sample, but this still limits practical applications. Here, we report an imaging method named 'speckle kinetography' that enables high-resolution imaging in unknown scattering media with thicknesses up to about 6 transport mean free paths. Speckle kinetography non-invasively records a series of incoherent speckle images accompanied by object motion and the inherently retained object information is extracted through an overlapping speckle correlation algorithm to construct the object's autocorrelation for imaging. Under single-colour light-emitting diode, white light, and fluorescence illumination, we experimentally demonstrate 1 μm resolution imaging and tracking of objects moving in scattering samples, while reducing the requirements for prior knowledge. We anticipate this method will enable imaging in currently inaccessible scenarios.

Backpropagation-free training of deep physical neural networks

Recent successes in deep learning for vision and natural language processing are attributed to larger models but come with energy consumption and scalability issues. Current training of digital deep learning models primarily relies on backpropagation that is unsuitable for physical implementation. Here, we proposed a simple deep neural network architecture augmented by a physical local learning (PhyLL) algorithm, enabling supervised and unsupervised training of deep physical neural networks, without detailed knowledge of the nonlinear physical layer's properties. We trained diverse wave-based physical neural networks in vowel and image classification experiments, showcasing our approach's universality. Our method shows advantages over other hardware-aware training schemes by improving training speed, enhancing robustness, and reducing power consumption by eliminating the need for system modelling and thus decreasing digital computation.

Complex-valued matrix-vector multiplication system for a large-scale optical FFT

Recent advancements in optical convolutional neural networks (CNNs) and radar signal processing systems have brought an increasing need for the adoption of optical fast Fourier transform (OFFT). Presently, the fast Fourier transform (FFT) is executed using electronic means within prevailing architectures. However, this electronic approach faces limitations in terms of both speed and power consumption. Concurrently, existing OFFT systems struggle to balance the demands of large-scale processing and high precision simultaneously. In response, we introduce a novel, to the best of our knowledge, solution: a complex-valued matrix-vector system harnessed through wavelength selective switches (WSSs) for the realization of a 24-input optical FFT, achieving a high-accuracy level of 5.4 bits. This study capitalizes on the abundant wavelength resources available to present a feasible solution for an optical FFT system with a large N.

Photonic convolutional neural network with robustness against wavelength deviations

We experimentally explore the practicality of integrated multiwavelength laser arrays (MLAs) for photonic convolutional neural network (PCNN). MLAs represent excellent performance for PCNN, except for imperfect wavelength spacings due to fabrication variation. Therefore, the performance of PCNN with non-ideal wavelength spacing is investigated experimentally and numerically for the first time. The results show that there exists a certain tolerance for wavelength deviation on the degradation of the structural information of the extracted feature map, leading to the robustness of photonic recognition accuracy under non-ideal wavelength spacing. The results suggest that scalable MLAs could serve as an alternative source for the PCNN, to support low-cost optical computing scenarios. For a benchmark classification task of MNIST handwritten digits, the photonic prediction accuracy of 91.2% for stride 1 × 1 scheme using the testing dataset are experimentally obtained at speeds on the order of tera operations per second, compared to 94.14% on computer. The robust performance, flexible spectral control, low cost, large bandwidth and parallel processing capability of the PCNN driven by scalable MLAs may broaden the application possibilities of photonic neural networks in next generation data computing applications.

All-analog photoelectronic chip for high-speed vision tasks

Photonic computing enables faster and more energy-efficient processing of vision data1-5. However, experimental superiority of deployable systems remains a challenge because of complicated optical nonlinearities, considerable power consumption of analog-to-digital converters (ADCs) for downstream digital processing and vulnerability to noises and system errors1,6-8. Here we propose an all-analog chip combining electronic and light computing (ACCEL). It has a systemic energy efficiency of 74.8 peta-operations per second per watt and a computing speed of 4.6 peta-operations per second (more than 99% implemented by optics), corresponding to more than three and one order of magnitude higher than state-of-the-art computing processors, respectively. After applying diffractive optical computing as an optical encoder for feature extraction, the light-induced photocurrents are directly used for further calculation in an integrated analog computing chip without the requirement of analog-to-digital converters, leading to a low computing latency of 72 ns for each frame. With joint optimizations of optoelectronic computing and adaptive training, ACCEL achieves competitive classification accuracies of 85.5%, 82.0% and 92.6%, respectively, for Fashion-MNIST, 3-class ImageNet classification and time-lapse video recognition task experimentally, while showing superior system robustness in low-light conditions (0.14 fJ μm-2 each frame). ACCEL can be used across a broad range of applications such as wearable devices, autonomous driving and industrial inspections.

Reproduction of mode-locked pulses by spectrotemporal domain-informed deep learning

The accurate reproduction of unique pulse states in a mode-locked fiber laser is an important scientific issue and has wide applications in the laser industry. We present what we believe to be a novel method for automatically and precisely reproducing targeted soliton states in a mode-locked fiber laser by spectrotemporal domain-informed deep learning. Targeted solitons are experimentally reproduced via a superior matching process with a spectrotemporal mean square error (MSE) of 3.99 × 10-5. The outstanding feature of our reproduction algorithm is that the pulse information in both the spectral and temporal domains is jointly adopted for reconstructing targeted soliton states from white noise, rather than establishing arbitrary mode-locked pulse states, as described in previous studies. Additionally, a single-layer perceptron model is proposed to retrieve the phase distribution of a mode-locked pulse, validating the physical completeness of our reproduction approach. Our approach advances ultrafast laser technology, enabling the precise control of pulse dynamics in applications such as optical communication and nonlinear optics.

Snapshot Mueller spectropolarimeter imager

We introduce an imaging system that can simultaneously record complete Mueller polarization responses for a set of wavelength channels in a single image capture. The division-of-focal-plane concept combines a multiplexed illumination scheme based on Fourier optics together with an integrated telescopic light-field imaging system. Polarization-resolved imaging is achieved using broadband nanostructured plasmonic polarizers as functional pinhole apertures. The recording of polarization and wavelength information on the image sensor is highly interpretable. We also develop a calibration approach based on a customized neural network architecture that can produce calibrated measurements in real-time. As a proof-of-concept demonstration, we use our calibrated system to accurately reconstruct a thin film thickness map from a four-inch wafer. We anticipate that our concept will have utility in metrology, machine vision, computational imaging, and optical computing platforms.

High-frame-rate reconfigurable diffractive neural network based on superpixels

The existing implementations of reconfigurable diffractive neural networks rely on both a liquid-crystal spatial light modulator and a digital micromirror device, which results in complexity in the alignment of the optical system and a constrained computational speed. Here, we propose a superpixel diffractive neural network that leverages solely a digital micromirror device to control the neuron bias and connection. This approach considerably simplifies the optical system and achieves a computational speed of 326 Hz per neural layer. We validate our method through experiments in digit classification, achieving an accuracy of 82.6%, and action recognition, attaining a perfect accuracy of 100%. Our findings demonstrate the effectiveness of the superpixel diffractive neural network in simplifying the optical system and enhancing computational speed, opening up new possibilities for real-time optical information processing applications.

All-Optical Data Processing with Photon-Avalanching Nanocrystalline Photonic Synapse

Data processing and storage in electronic devices are typically performed as a sequence of elementary binary operations. Alternative approaches, such as neuromorphic or reservoir computing, are rapidly gaining interest where data processing is relatively slow, but can be performed in a more comprehensive way or massively in parallel, like in neuronal circuits. Here, time-domain all-optical information processing capabilities of photon-avalanching (PA) nanoparticles at room temperature are discovered. Demonstrated functionality resembles properties found in neuronal synapses, such as: paired-pulse facilitation and short-term internal memory, in situ plasticity, multiple inputs processing, and all-or-nothing threshold response. The PA-memory-like behavior shows capability of machine-learning-algorithm-free feature extraction and further recognition of 2D patterns with simple 2 input artificial neural network. Additionally, high nonlinearity of luminescence intensity in response to photoexcitation mimics and enhances spike-timing-dependent plasticity that is coherent in nature with the way a sound source is localized in animal neuronal circuits. Not only are yet unexplored fundamental properties of photon-avalanche luminescence kinetics studied, but this approach, combined with recent achievements in photonics, light confinement and guiding, promises all-optical data processing, control, adaptive responsivity, and storage on photonic chips.

Multi-wavelength diffractive neural network with the weighting method

Recently, the diffractive deep neural network (D2NN) has demonstrated the advantages to achieve large-scale computational tasks in terms of high speed, low power consumption, parallelism, and scalability. A typical D2NN with cascaded diffractive elements is designed for monochromatic illumination. Here, we propose a framework to achieve the multi-wavelength D2NN (MW-D2NN) based on the method of weight coefficients. In training, each wavelength is assigned a specific weighting and their output planes construct the wavelength weighting loss function. The trained MW-D2NN can implement the classification of images of handwritten digits at multi-wavelength incident beams. The designed 3-layers MW-D2NN achieves a simulation classification accuracy of 83.3%. We designed a 1-layer MW-D2NN. The simulation and experiment classification accuracy are 71.4% and 67.5% at RGB wavelengths. Furthermore, the proposed MW-D2NN can be extended to intelligent machine vision systems for multi-wavelength and incoherent illumination.

Referenceless characterization of complex media using physics-informed neural networks

In this work, we present a method to characterize the transmission matrices of complex scattering media using a physics-informed, multi-plane neural network (MPNN) without the requirement of a known optical reference field. We use this method to accurately measure the transmission matrix of a commercial multi-mode fiber without the problems of output-phase ambiguity and dark spots, leading to up to 58% improvement in focusing efficiency compared with phase-stepping holography. We demonstrate how our method is significantly more noise-robust than phase-stepping holography and show how it can be generalized to characterize a cascade of transmission matrices, allowing one to control the propagation of light between independent scattering media. This work presents an essential tool for accurate light control through complex media, with applications ranging from classical optical networks, biomedical imaging, to quantum information processing.

High-throughput terahertz imaging: progress and challenges

Many exciting terahertz imaging applications, such as non-destructive evaluation, biomedical diagnosis, and security screening, have been historically limited in practical usage due to the raster-scanning requirement of imaging systems, which impose very low imaging speeds. However, recent advancements in terahertz imaging systems have greatly increased the imaging throughput and brought the promising potential of terahertz radiation from research laboratories closer to real-world applications. Here, we review the development of terahertz imaging technologies from both hardware and computational imaging perspectives. We introduce and compare different types of hardware enabling frequency-domain and time-domain imaging using various thermal, photon, and field image sensor arrays. We discuss how different imaging hardware and computational imaging algorithms provide opportunities for capturing time-of-flight, spectroscopic, phase, and intensity image data at high throughputs. Furthermore, the new prospects and challenges for the development of future high-throughput terahertz imaging systems are briefly introduced.

Single-pixel image reconstruction using coherent nonlinear optics

We propose and experimentally demonstrate a novel, to the best of our knowledge, hybrid optoelectronic system that utilizes mode-selective frequency upconversion, single-pixel detection, and a deep neural network to achieve the reliable reconstruction of two-dimensional (2D) images from a noise-contaminated database of handwritten digits. Our system is designed to maximize the multi-scale structural similarity index measure (MS-SSIM) and minimize the mean absolute error (MAE) during the training process. Through extensive evaluation, we have observed that the reconstructed images exhibit high-quality results, with a peak signal-to-noise ratio (PSNR) reaching approximately 20 dB and a structural similarity index measure (SSIM) of around 0.85. These impressive metrics demonstrate the effectiveness and fidelity of our image reconstruction technique. The versatility of our approach allows its application in various fields, including Lidar, compressive imaging, volumetric reconstruction, and so on.