Inference in artificial intelligence with deep optics and photonics

Electromagnetic wave-based extreme deep learning with nonlinear time-Floquet entanglement

Wave-based analog signal processing holds the promise of extremely fast, on-the-fly, power-efficient data processing, occurring as a wave propagates through an artificially engineered medium. Yet, due to the fundamentally weak non-linearities of traditional electromagnetic materials, such analog processors have been so far largely confined to simple linear projections such as image edge detection or matrix multiplications. Complex neuromorphic computing tasks, which inherently require strong non-linearities, have so far remained out-of-reach of wave-based solutions, with a few attempts that implemented non-linearities on the digital front, or used weak and inflexible non-linear sensors, restraining the learning performance. Here, we tackle this issue by demonstrating the relevance of time-Floquet physics to induce a strong non-linear entanglement between signal inputs at different frequencies, enabling a power-efficient and versatile wave platform for analog extreme deep learning involving a single, uniformly modulated dielectric layer and a scattering medium. We prove the efficiency of the method for extreme learning machines and reservoir computing to solve a range of challenging learning tasks, from forecasting chaotic time series to the simultaneous classification of distinct datasets. Our results open the way for optical wave-based machine learning with high energy efficiency, speed and scalability.

LOEN: Lensless opto-electronic neural network empowered machine vision

Machine vision faces bottlenecks in computing power consumption and large amounts of data. Although opto-electronic hybrid neural networks can provide assistance, they usually have complex structures and are highly dependent on a coherent light source; therefore, they are not suitable for natural lighting environment applications. In this paper, we propose a novel lensless opto-electronic neural network architecture for machine vision applications. The architecture optimizes a passive optical mask by means of a task-oriented neural network design, performs the optical convolution calculation operation using the lensless architecture, and reduces the device size and amount of calculation required. We demonstrate the performance of handwritten digit classification tasks with a multiple-kernel mask in which accuracies of as much as 97.21% were achieved. Furthermore, we optimize a large-kernel mask to perform optical encryption for privacy-protecting face recognition, thereby obtaining the same recognition accuracy performance as no-encryption methods. Compared with the random MLS pattern, the recognition accuracy is improved by more than 6%.

Photonic-aware neural networks

Photonics-based neural networks promise to outperform electronic counterparts, accelerating neural network computations while reducing power consumption and footprint. However, these solutions suffer from physical layer constraints arising from the underlying analog photonic hardware, impacting the resolution of computations (in terms of effective number of bits), requiring the use of positive-valued inputs, and imposing limitations in the fan-in and in the size of convolutional kernels. To abstract these constraints, in this paper we introduce the concept of Photonic-Aware Neural Network (PANN) architectures, i.e., deep neural network models aware of the photonic hardware constraints. Then, we devise PANN training schemes resorting to quantization strategies aimed to obtain the required neural network parameters in the fixed-point domain, compliant with the limited resolution of the underlying hardware. We finally carry out extensive simulations exploiting PANNs in image classification tasks on well-known datasets (MNIST, Fashion-MNIST, and Cifar-10) with varying bitwidths (i.e., 2, 4, and 6 bits). We consider two kernel sizes and two pooling schemes for each PANN model, exploiting $$2\times 2$$ 2 × 2 and $$3\times 3$$ 3 × 3 convolutional kernels, and max and average pooling, the latter more amenable to an optical implementation. $$3\times 3$$ 3 × 3 kernels perform better than $$2\times 2$$ 2 × 2 counterparts, while max and average pooling provide comparable results, with the latter performing better on MNIST and Cifar-10. The accuracy degradation due to the photonic hardware constraints is quite limited, especially on MNIST and Fashion-MNIST, demonstrating the feasibility of PANN approaches on computer vision tasks.

Integrated photonic metasystem for image classifications at telecommunication wavelength

Miniaturized image classifiers are potential for revolutionizing their applications in optical communication, autonomous vehicles, and healthcare. With subwavelength structure enabled directional diffraction and dispersion engineering, the light propagation through multi-layer metasurfaces achieves wavelength-selective image recognitions on a silicon photonic platform at telecommunication wavelength. The metasystems implement high-throughput vector-by-matrix multiplications, enabled by near 10³ nanoscale phase shifters as weight elements within 0.135 mm² footprints. The diffraction manifested computing capability incorporates the fabrication and measurement related phase fluctuations, and thus the pre-trained metasystem can handle uncertainties in inputs without post-tuning. Here we demonstrate three functional metasystems: a 15-pixel spatial pattern classifier that reaches near 90% accuracy with femtosecond inputs, a multi-channel wavelength demultiplexer, and a hyperspectral image classifier. The diffractive metasystem provides an alternative machine learning architecture for photonic integrated circuits, with densely integrated phase shifters, spatially multiplexed throughput, and data processing capabilities.

Metasystem architectures are attractive alternatives to waveguide-based integrated photonic processors due to the subwavelength structures. Here, the authors report a 1D passive silicon photonic metasystem with near 90% spatial pattern classification accuracy at telecommunication wavelength.

Single photon imaging based on a photon driven sparse sampling

Single photon three-dimensional (3D) imager can capture 3D profile details and see through obscuring objects with high sensitivity, making it promising in sensing and imaging applications. The key capabilities of such 3D imager lie on its depth resolution and multi-return discrimination. For conventional pulsed single photon lidar, these capabilities are limited by transmitter bandwidth and receiver bandwidth simultaneously. A single photon imager is proposed and experimentally demonstrated to implement time-resolved and multi-return imaging. Time-to-frequency conversion is performed to achieve millimetric depth resolution. Experimental results show that the depth resolution is better than 4.5 mm, even though time jitter of the SPAD reaches 1 ns and time resolution of the TCSPC module reaches 10 ns. Furthermore, photon driven sparse sampling mechanism allows us to discriminate multiple near surfaces, no longer limited by the receiver bandwidth. The simplicity of the system hardware enables low-cost and compact 3D imaging.

Extracting Lamb wave vibrating modes with convolutional neural network

In recent years, micro-acoustic devices, such as surface acoustic wave (SAW) devices, and bulk acoustic wave (BAW) devices have been widely used in the areas of Internet of Things and mobile communication. With the increasing demand of information transmission speed, working frequencies of micro-acoustic devices are becoming much higher. To meet the emerging demand, Lamb wave devices with characteristics that are fit for high working frequency come into being. However, Lamb wave devices have more complicated vibrating modes than SAW and BAW devices. Methods used for SAW and BAW devices are no longer suitable for the mode extraction of Lamb wave devices. To solve this difficulty, this paper proposed a method based on machine learning with convolutional neural network to achieve automatic identification. The great ability to handle large amount of images makes it a good option for vibrating mode recognition and extraction. With a pre-trained model, we are able to identify and extract the first two anti-symmetric and symmetric modes of Lamb waves in varisized plate structures. After the successful use of this method in Lamb wave modes automatic extraction, it can be extended to all micro-acoustic devices and all other wave types. The proposed method will further promote the application of the Lamb wave devices.

Orthogonality of diffractive deep neural network

Some rules of the diffractive deep neural network (D²NN) are discovered. They reveal that the inner product of any two optical fields in D²NN is invariant and the D²NN acts as a unitary transformation for optical fields. If the output intensities of the two inputs are separated spatially, the input fields must be orthogonal. These rules imply that the D²NN is not only suitable for the classification of general objects but also more suitable for applications aimed at optical orthogonal modes. Our simulation shows the D²NN performs well in applications like mode conversion, mode multiplexing/demultiplexing, and optical mode recognition.

Knowledge distillation circumvents nonlinearity for optical convolutional neural networks

In recent years, convolutional neural networks (CNNs) have enabled ubiquitous image processing applications. As such, CNNs require fast forward propagation runtime to process high-resolution visual streams in real time. This is still a challenging task even with state-of-the-art graphics and tensor processing units. The bottleneck in computational efficiency primarily occurs in the convolutional layers. Performing convolutions in the Fourier domain is a promising way to accelerate forward propagation since it transforms convolutions into elementwise multiplications, which are considerably faster to compute for large kernels. Furthermore, such computation could be implemented using an optical 4f system with orders of magnitude faster operation. However, a major challenge in using this spectral approach, as well as in an optical implementation of CNNs, is the inclusion of a nonlinearity between each convolutional layer, without which CNN performance drops dramatically. Here, we propose a spectral CNN linear counterpart (SCLC) network architecture and its optical implementation. We propose a hybrid platform with an optical front end to perform a large number of linear operations, followed by an electronic back end. The key contribution is to develop a knowledge distillation (KD) approach to circumvent the need for nonlinear layers between the convolutional layers and successfully train such networks. While the KD approach is known in machine learning as an effective process for network pruning, we adapt the approach to transfer the knowledge from a nonlinear network (teacher) to a linear counterpart (student), where we can exploit the inherent parallelism of light. We show that the KD approach can achieve performance that easily surpasses the standard linear version of a CNN and could approach the performance of the nonlinear network. Our simulations show that the possibility of increasing the resolution of the input image allows our proposed 4f optical linear network to perform more efficiently than a nonlinear network with the same accuracy on two fundamental image processing tasks: (i) object classification and (ii) semantic segmentation.

Free-Space Optical Merging via Meta-Grating Inverse-Design

Despite various advances in achieving arbitrary optics steering, one of the longstanding challenges is to achieve optical merging for combining multidirectional beams through single-time reflection/transmission in free space. Typically, dual-directional beam merging is conducted by combining half-transmission and half-reflection using beam splitters; however, it leads to a bulky system with stray light and low merging efficiency. The difficulty of free-space beam merging lies in imparting respective distinct wavevectors to different directional beams. Herein, we originally proposed and successfully demonstrated the free-space optical merging (FOM) functionality based on the inverse-designed meta-grating architecture in the visible regime. By utilizing the inverse problem solver, two proposed meta-grating schemes experimentally enable merging of dual-directional beams into the same outgoing angle for the first time merely through single-time reflection. We envision that the creation of free-space merging performance can be widely applicable to the future optical system and facilitate the miniature optical devices and integration.

Nonlinear optical response of inverse-designed integrated photonic devices

Gradient-based optimization combined with the adjoint method has been demonstrated to be an efficient way to design a nano-structure with a vast number of degrees of freedom. However, most inverse-designed photonic devices are applied as linear photonic devices. Here, we demonstrate the nonlinear optical response in inverse-designed integrated splitters fabricated on a SiN platform. The splitting ratio is tunable under different incident powers. The thermo-optical effect can be used as an effective approach for adjusting the nonlinear optical response threshold and modulation depth of the device. These promising results indicate the great potential of inverse-designed photonic devices in nonlinear optics and optical communications.

Insect-inspired nanofibrous polyaniline multi-scale films for hybrid polarimetric imaging with scattered light

We demonstrate a bio-inspired coating for novel imaging and sensing designs: the coating sorts different colors and linear polarizations. This coating, composed of conducting, nanofibrous polyaniline in an inverse opal film (PANI-IOF), is inexpensive and can feasibly be deposited over large areas on a range of flexible and non-flat substrates. With PANI IOFs, light is scattered into azimuthally polarized Debye rings. Subsequently, the diffracted speckle patterns carry compressed representations of the polarized illumination, which we reconstruct using shallow neural networks.

Ultralow-loss waveguide crossing for photonic integrated circuits by using inverted tapers

An ultralow-loss silicon planar waveguide crossing operating in the O-band was experimentally demonstrated based on the Gaussian beam synthesis method. Elliptical parabolic inverted tapers were introduced in our design to reduce the crossing loss. According to the measurement results, the proposed device exhibits an insertion loss of 0.008 dB, which is the lowest reported loss for planar silicon waveguide crossings operating in the O-band. The device exhibits a low crosstalk below -40 dB over a 40 nm wavelength range with a compact footprint of 18 × 18 µm² and can be fabricated in a complementary metal-oxide-semiconductor-compatible process.

Uni-Messe: Unified Rule-Based Message Delivery Service for Efficient Context-Aware Service Integration

Rule-based systems, which are the typical technology used to realize context-aware services, have been independently implemented in various smart services. The challenges of these systems are the versatility of action, looseness, and the coding that is needed to describe the conditional branches. The purpose of this study was to support the realization of service coordination and smart services using context-aware technology by converting rule-based systems into services. In the proposed method, we designed and implemented the architecture of a new service: Unified Rule-Based Message Delivery Service (Uni-messe), which is an application-neutral rule management and evaluation service for rule-based systems. The core part of the Uni-messe proposal is the combination of a Pub/Sub and a rule-based system, and the proposal of a new event–condition–route (ECR) rule-based system. We applied Uni-messe to an audio information presentation system (ALPS) and indoor location sensing technology to construct concrete smart services, and then compared and evaluated the implementation to “if this then that” (IFTTT), which is a typical service coordination technology. Moreover, we analyzed the characteristics of other rule-based systems that have been serviced in previous studies and compared them to Uni-messe. This study shows that Uni-messe can provide services that simultaneously combine versatility, ease of conditional description, looseness, context independence, and user interface (UI), which cannot be achieved using conventional rule-based system services. By using Uni-messe, advanced heterogeneous distributed service coordination using rule-based systems and the construction of context-aware services can be performed easily.

Space-efficient optical computing with an integrated chip diffractive neural network

Large-scale, highly integrated and low-power-consuming hardware is becoming progressively more important for realizing optical neural networks (ONNs) capable of advanced optical computing. Traditional experimental implementations need N² units such as Mach-Zehnder interferometers (MZIs) for an input dimension N to realize typical computing operations (convolutions and matrix multiplication), resulting in limited scalability and consuming excessive power. Here, we propose the integrated diffractive optical network for implementing parallel Fourier transforms, convolution operations and application-specific optical computing using two ultracompact diffractive cells (Fourier transform operation) and only N MZIs. The footprint and energy consumption scales linearly with the input data dimension, instead of the quadratic scaling in the traditional ONN framework. A ~10-fold reduction in both footprint and energy consumption, as well as equal high accuracy with previous MZI-based ONNs was experimentally achieved for computations performed on the MNIST and Fashion-MNIST datasets. The integrated diffractive optical network (IDNN) chip demonstrates a promising avenue towards scalable and low-power-consumption optical computational chips for optical-artificial-intelligence.

Here, we propose the integrated diffractive optical network for implementing parallel Fourier transforms, convolution operations and application-specific optical computing with reduced footprint and energy consumption.

Computer-free computational imaging: optical computing for seeing through random media

Diffractive Deep Neural Network enables computer-free, all-optical "computational imaging" for seeing through unknown random diffusers at the speed of light.

Photonic matrix multiplication lights up photonic accelerator and beyond

Matrix computation, as a fundamental building block of information processing in science and technology, contributes most of the computational overheads in modern signal processing and artificial intelligence algorithms. Photonic accelerators are designed to accelerate specific categories of computing in the optical domain, especially matrix multiplication, to address the growing demand for computing resources and capacity. Photonic matrix multiplication has much potential to expand the domain of telecommunication, and artificial intelligence benefiting from its superior performance. Recent research in photonic matrix multiplication has flourished and may provide opportunities to develop applications that are unachievable at present by conventional electronic processors. In this review, we first introduce the methods of photonic matrix multiplication, mainly including the plane light conversion method, Mach-Zehnder interferometer method and wavelength division multiplexing method. We also summarize the developmental milestones of photonic matrix multiplication and the related applications. Then, we review their detailed advances in applications to optical signal processing and artificial neural networks in recent years. Finally, we comment on the challenges and perspectives of photonic matrix multiplication and photonic acceleration.

Imaging through diffuse media using multi-mode vortex beams and deep learning

Optical imaging through diffuse media is a challenging issue and has attracted applications in many fields such as biomedical imaging, non-destructive testing, and computer-assisted surgery. However, light interaction with diffuse media leads to multiple scattering of the photons in the angular and spatial domain, severely degrading the image reconstruction process. In this article, a novel method to image through diffuse media using multiple modes of vortex beams and a new deep learning network named "LGDiffNet" is derived. A proof-of-concept numerical simulation is conducted using this method, and the results are experimentally verified. In this technique, the multiple modes of Gaussian and Laguerre-Gaussian beams illuminate the displayed digits dataset number, and the beams are then propagated through the diffuser before being captured on the beam profiler. Furthermore, we investigated whether imaging through diffuse media using multiple modes of vortex beams instead of Gaussian beams improves the imaging system's imaging capability and enhances the network's reconstruction ability. Our results show that illuminating the diffuser using vortex beams and employing the "LGDiffNet" network provides enhanced image reconstruction compared to existing modalities. An enhancement of ~ 1 dB, in terms of PSNR, is achieved using this method when a highly scattering diffuser of grit 220 and width 2 mm (7.11 times the mean free path) is used. No additional optimizations or reference beams were used in the imaging system, revealing the robustness of the "LGDiffNet" network and the adaptability of the imaging system for practical applications in medical imaging.

Harnessing optoelectronic noises in a photonic generative network

Integrated optoelectronics is emerging as a promising platform of neural network accelerator, which affords efficient in-memory computing and high bandwidth interconnectivity. The inherent optoelectronic noises, however, make the photonic systems error-prone in practice. It is thus imperative to devise strategies to mitigate and, if possible, harness noises in photonic computing systems. Here, we demonstrate a photonic generative network as a part of a generative adversarial network (GAN). This network is implemented with a photonic core consisting of an array of programable phase-change memory cells to perform four-element vector-vector dot multiplication. The GAN can generate a handwritten number ("7") in experiments and full 10 digits in simulation. We realize an optical random number generator, apply noise-aware training by injecting additional noise, and demonstrate the network's resilience to hardware nonidealities. Our results suggest the resilience and potential of more complex photonic generative networks based on large-scale, realistic photonic hardware.

An optical neural network using less than 1 photon per multiplication

Deep learning has become a widespread tool in both science and industry. However, continued progress is hampered by the rapid growth in energy costs of ever-larger deep neural networks. Optical neural networks provide a potential means to solve the energy-cost problem faced by deep learning. Here, we experimentally demonstrate an optical neural network based on optical dot products that achieves 99% accuracy on handwritten-digit classification using ~3.1 detected photons per weight multiplication and ~90% accuracy using ~0.66 photons (~2.5 × 10⁻¹⁹ J of optical energy) per weight multiplication. The fundamental principle enabling our sub-photon-per-multiplication demonstration—noise reduction from the accumulation of scalar multiplications in dot-product sums—is applicable to many different optical-neural-network architectures. Our work shows that optical neural networks can achieve accurate results using extremely low optical energies.

Though theory suggests that highly energy efficient optical neural networks (ONNs) based on optical matrix-vector multipliers are possible, an experimental validation is lacking. Here, the authors report an ONN with >90% accuracy image classification using <1 detected photon per scalar multiplication.

Deep physical neural networks trained with backpropagation

Deep-learning models have become pervasive tools in science and engineering. However, their energy requirements now increasingly limit their scalability1. Deep-learning accelerators2-9 aim to perform deep learning energy-efficiently, usually targeting the inference phase and often by exploiting physical substrates beyond conventional electronics. Approaches so far10-22 have been unable to apply the backpropagation algorithm to train unconventional novel hardware in situ. The advantages of backpropagation have made it the de facto training method for large-scale neural networks, so this deficiency constitutes a major impediment. Here we introduce a hybrid in situ-in silico algorithm, called physics-aware training, that applies backpropagation to train controllable physical systems. Just as deep learning realizes computations with deep neural networks made from layers of mathematical functions, our approach allows us to train deep physical neural networks made from layers of controllable physical systems, even when the physical layers lack any mathematical isomorphism to conventional artificial neural network layers. To demonstrate the universality of our approach, we train diverse physical neural networks based on optics, mechanics and electronics to experimentally perform audio and image classification tasks. Physics-aware training combines the scalability of backpropagation with the automatic mitigation of imperfections and noise achievable with in situ algorithms. Physical neural networks have the potential to perform machine learning faster and more energy-efficiently than conventional electronic processors and, more broadly, can endow physical systems with automatically designed physical functionalities, for example, for robotics23-26, materials27-29 and smart sensors30-32.

Deep learning multi-shot 3D localization microscopy using hybrid optical-electronic computing

Current 3D localization microscopy approaches are fundamentally limited in their ability to image thick, densely labeled specimens. Here, we introduce a hybrid optical-electronic computing approach that jointly optimizes an optical encoder (a set of multiple, simultaneously imaged 3D point spread functions) and an electronic decoder (a neural-network-based localization algorithm) to optimize 3D localization performance under these conditions. With extensive simulations and biological experiments, we demonstrate that our deep-learning-based microscope achieves significantly higher 3D localization accuracy than existing approaches, especially in challenging scenarios with high molecular density over large depth ranges.

Artificial intelligence-powered microfluidics for nanomedicine and materials synthesis

Artificial intelligence (AI) is an emerging technology with great potential, and its robust calculation and analysis capabilities are unmatched by traditional calculation tools. With the promotion of deep learning and open-source platforms, the threshold of AI has also become lower. Combining artificial intelligence with traditional fields to create new fields of high research and application value has become a trend. AI has been involved in many disciplines, such as medicine, materials, energy, and economics. The development of AI requires the support of many kinds of data, and microfluidic systems can often mine object data on a large scale to support AI. Due to the excellent synergy between the two technologies, excellent research results have emerged in many fields. In this review, we briefly review AI and microfluidics and introduce some applications of their combination, mainly in nanomedicine and material synthesis. Finally, we discuss the development trend of the combination of the two technologies.

Two-color optically addressed spatial light modulator as a generic spatiotemporal system

Nonlinear spatiotemporal systems are the basis for countless physical phenomena in such diverse fields as ecology, optics, electronics, and neuroscience. The canonical approach to unify models originating from different fields is the normal form description, which determines the generic dynamical aspects and different bifurcation scenarios. Realizing different types of dynamical systems via one experimental platform that enables continuous transition between normal forms through tuning accessible system parameters is, therefore, highly relevant. Here, we show that a transmissive, optically addressed spatial light modulator under coherent optical illumination and optical feedback coupling allows tuning between pitchfork, transcritical, and saddle-node bifurcations of steady states. We demonstrate this by analytically deriving the system's dynamical equations in correspondence to the normal forms of the associated bifurcations and confirm these results via extensive numerical simulations. Our model describes a nematic liquid crystal device using nano-dimensional dichalcogenide (a-As ₂S ₃) glassy thin films as photo sensors and alignment layers, and we use device parameters obtained from experimental characterization. Optical coupling, for example, using diffraction, holography, or integrated unitary maps allows implementing a variety of system topologies of technological relevance for neural networks and potentially Ising or XY-Hamiltonian models with ultralow energy consumption.

Optical meta-waveguides for integrated photonics and beyond

The growing maturity of nanofabrication has ushered massive sophisticated optical structures available on a photonic chip. The integration of subwavelength-structured metasurfaces and metamaterials on the canonical building block of optical waveguides is gradually reshaping the landscape of photonic integrated circuits, giving rise to numerous meta-waveguides with unprecedented strength in controlling guided electromagnetic waves. Here, we review recent advances in meta-structured waveguides that synergize various functional subwavelength photonic architectures with diverse waveguide platforms, such as dielectric or plasmonic waveguides and optical fibers. Foundational results and representative applications are comprehensively summarized. Brief physical models with explicit design tutorials, either physical intuition-based design methods or computer algorithms-based inverse designs, are cataloged as well. We highlight how meta-optics can infuse new degrees of freedom to waveguide-based devices and systems, by enhancing light-matter interaction strength to drastically boost device performance, or offering a versatile designer media for manipulating light in nanoscale to enable novel functionalities. We further discuss current challenges and outline emerging opportunities of this vibrant field for various applications in photonic integrated circuits, biomedical sensing, artificial intelligence and beyond.

Efficient color imaging through unknown opaque scattering layers via physics-aware learning

Color imaging with scattered light is crucial to many practical applications and becomes one of the focuses in optical imaging fields. More physics theories have been introduced in the deep learning (DL) approach for the optical tasks and improve the imaging capability a lot. Here, an efficient color imaging method is proposed in reconstructing complex objects hidden behind unknown opaque scattering layers, which can obtain high reconstruction fidelity in spatial structure and accurate restoration in color information by training with only one diffuser. More information is excavated by utilizing the scattering redundancy and promotes the physics-aware DL approach to reconstruct the color objects hidden behind unknown opaque scattering layers with robust generalization capability by an efficient means. This approach gives impetus to color imaging through dynamic scattering media and provides an enlightening reference for solving complex inverse problems based on physics-aware DL methods.

Architecture agnostic algorithm for reconfigurable optical interferometer programming

We develop the learning algorithm to build an architecture agnostic model of a reconfigurable optical interferometer. A procedure of programming a unitary transformation of optical modes of an interferometer either follows an analytical expression yielding a unitary matrix given a set of phase shifts or requires an optimization routine if an analytic decomposition does not exist. Our algorithm adopts a supervised learning strategy which matches a model of an interferometer to a training set populated by samples produced by a device under study. A simple optimization routine uses the trained model to output phase shifts corresponding to a desired unitary transformation of the interferometer with a given architecture. Our result provides the recipe for efficient tuning of interferometers even without rigorous analytical description which opens opportunity to explore new architectures of the interferometric circuits.

Universal linear optics by programmable multimode interference

We introduce a constructive algorithm for universal linear electromagnetic transformations between the N input and N output modes of a dielectric slab. The approach uses out-of-plane phase modulation programmed down to N² degrees of freedom. The total area of these modulators equals that of the entire slab: our scheme makes optimal use of the available area for optical modulation. We also present error correction schemes that enable high-fidelity unitary transformations at large N. This "programmable multimode interferometer" (ProMMI) thus translates the algorithmic simplicity of Mach-Zehnder meshes into a holographically programmed slab, yielding DoF-limited compactness and error tolerance while eliminating the dominant sidewall-related optical losses and directional-coupler-related patterning challenges.

Chinese Language Feature Analysis Based on Multilayer Self-Organizing Neural Network and Data Mining Techniques

As one of the oldest languages in the world, Chinese has a long cultural history and unique language charm. The multilayer self-organizing neural network and data mining techniques have been widely used and can achieve high-precision prediction in different fields. However, they are hardly applied to Chinese language feature analysis. In order to accurately analyze the characteristics of Chinese language, this paper uses the multilayer self-organizing neural network and the corresponding data mining technology for feature recognition and then compared it with other different types of neural network algorithms. The results show that the multilayer self-organizing neural network can make the accuracy, recall, and F1 score of feature recognition reach 68.69%, 80.21%, and 70.19%, respectively, when there are many samples. Under the influence of strong noise, it keeps high efficiency of feature analysis. This shows that the multilayer self-organizing neural network has superior performance and can provide strong support for Chinese language feature analysis.

Roadmap on digital holography [Invited]

This Roadmap article on digital holography provides an overview of a vast array of research activities in the field of digital holography. The paper consists of a series of 25 sections from the prominent experts in digital holography presenting various aspects of the field on sensing, 3D imaging and displays, virtual and augmented reality, microscopy, cell identification, tomography, label-free live cell imaging, and other applications. Each section represents the vision of its author to describe the significant progress, potential impact, important developments, and challenging issues in the field of digital holography.

Compressive hyperspectral image classification using a 3D coded convolutional neural network

Hyperspectral image classification (HIC) is an active research topic in remote sensing. Hyperspectral images typically generate large data cubes posing big challenges in data acquisition, storage, transmission and processing. To overcome these limitations, this paper develops a novel deep learning HIC approach based on compressive measurements of coded-aperture snapshot spectral imagers (CASSI), without reconstructing the complete hyperspectral data cube. A new kind of deep learning strategy, namely 3D coded convolutional neural network (3D-CCNN), is proposed to efficiently solve for the classification problem, where the hardware-based coded aperture is regarded as a pixel-wise connected network layer. An end-to-end training method is developed to jointly optimize the network parameters and the coded apertures with periodic structures. The accuracy of classification is effectively improved by exploiting the synergy between the deep learning network and coded apertures. The superiority of the proposed method is assessed over the state-of-the-art HIC methods on several hyperspectral datasets.

Deep learning: an efficient method for plasmonic design of geometric nanoparticles

Plasmons of noble metal nanoparticles have played an important role in energy transfer research, biomedical sensing, drug preparation and photocatalysis in recent years. The scattering spectra of nanoparticles are dramatically affected by numerous complex parameters, such as morphology, material, and volumes, making the parameters design a necessary step before the experiment. However, the plasmonic design is limited by several difficulties, such as the high degree of freedom, expensive trial and error costs. Herein, a plasmonic design method based on dual strategy deep learning is proposed, which can provide the design proposals according to the required spectrum. To make the model closer to the real experimental situation, the boundary element method was used to build a scattering spectra dataset of geometric nanoparticles (>1200 000 samples). Driven by the above data, the artificial intelligence (AI) model learns the relationship between spectral features and design parameters. Then, the performance statistics of the model were implemented from multiple dimensions, and a high design precision of over 90% was achieved in testing cases (testing samples >120 000). Moreover, to verify the realizability of the proposed model, the scattering spectra of nanoparticles designed by AI were constructed using a dark-field microscope system. The experimental results showed that the deviation between the target and the actual spectra was very small and within the acceptable range. This proves the realizability of the AI model proposed in this paper, and sheds light on the application of AI in plasmonic design.

Integrated random projection and dimensionality reduction by propagating light in photonic lattices

It is proposed that the propagation of light in disordered photonic lattices can be harnessed as a random projection that preserves distances between a set of projected vectors. This mapping is enabled by the complex evolution matrix of a photonic lattice with diagonal disorder, which turns out to be a random complex Gaussian matrix. Thus, by collecting the output light from a random subset of the waveguide channels, one can perform an embedding from a higher- to a lower-dimensional space that respects the Johnson-Lindenstrauss lemma and nearly preserves the Euclidean distances. The distance-preserving random projection through photonic lattices requires intermediate disorder levels that allow diffusive propagation of light. The proposed scheme can be utilized as a simple and powerful integrated dimension reduction stage that can greatly reduce the burden of a subsequent neural computation stage.

Analog optical deconvolution computing for wavefront coding based on nanoantennas metasurfaces

Analog optical computing based on metasurfaces has attracted much attention for achieving high-speed calculating without the electronic processing unit. Wavefront coding imaging systems involve the joint design of an encoded image-capturing module and decoding postprocessing algorithms to obtain a required final image. The decoding postprocessing algorithms, as a typical deconvolution computation, require an additional electronic processing unit to yield a high-quality decoded image. We demonstrate an analog optical deconvolution computing kernel based on nanoantennas metasurfaces for the postprocessing calculation of wavefront coding systems. Numerical simulations are presented to prove that the encoded point spread function can be refocused through a designed optical computing metasurface. The proposed approach opens an opportunity for real-time recovering images in wavefront coding optical systems.

All-optical synthesis of an arbitrary linear transformation using diffractive surfaces

Spatially-engineered diffractive surfaces have emerged as a powerful framework to control light-matter interactions for statistical inference and the design of task-specific optical components. Here, we report the design of diffractive surfaces to all-optically perform arbitrary complex-valued linear transformations between an input (N_i) and output (N_o), where N_i and N_o represent the number of pixels at the input and output fields-of-view (FOVs), respectively. First, we consider a single diffractive surface and use a matrix pseudoinverse-based method to determine the complex-valued transmission coefficients of the diffractive features/neurons to all-optically perform a desired/target linear transformation. In addition to this data-free design approach, we also consider a deep learning-based design method to optimize the transmission coefficients of diffractive surfaces by using examples of input/output fields corresponding to the target transformation. We compared the all-optical transformation errors and diffraction efficiencies achieved using data-free designs as well as data-driven (deep learning-based) diffractive designs to all-optically perform (i) arbitrarily-chosen complex-valued transformations including unitary, nonunitary, and noninvertible transforms, (ii) 2D discrete Fourier transformation, (iii) arbitrary 2D permutation operations, and (iv) high-pass filtered coherent imaging. Our analyses reveal that if the total number (N) of spatially-engineered diffractive features/neurons is ≥N_i × N_o, both design methods succeed in all-optical implementation of the target transformation, achieving negligible error. However, compared to data-free designs, deep learning-based diffractive designs are found to achieve significantly larger diffraction efficiencies for a given N and their all-optical transformations are more accurate for N < N_i × N_o. These conclusions are generally applicable to various optical processors that employ spatially-engineered diffractive surfaces.

Polarisation optics for biomedical and clinical applications: a review

Many polarisation techniques have been harnessed for decades in biological and clinical research, each based upon measurement of the vectorial properties of light or the vectorial transformations imposed on light by objects. Various advanced vector measurement/sensing techniques, physical interpretation methods, and approaches to analyse biomedically relevant information have been developed and harnessed. In this review, we focus mainly on summarising methodologies and applications related to tissue polarimetry, with an emphasis on the adoption of the Stokes-Mueller formalism. Several recent breakthroughs, development trends, and potential multimodal uses in conjunction with other techniques are also presented. The primary goal of the review is to give the reader a general overview in the use of vectorial information that can be obtained by polarisation optics for applications in biomedical and clinical research.

Ultra-dense perfect optical orbital angular momentum multiplexed holography

Optical orbital angular momentum (OAM) has been recently implemented in holography technologies as an independent degree of freedom for boosting information capacity. However, the holography capacity and fidelity suffer from the limited space-bandwidth product (SBP) and the channel crosstalk, albeit the OAM mode set exploited as multiplexing channels is theoretically unbounded. Here, we propose the ultra-dense perfect OAM holography, in which the OAM modes are discriminated both radially and angularly. As such, the perfect OAM mode set constructs the two-dimensional spatial division multiplexed holography (conventional OAM holography is 1D). The extending degree of freedom enhances the holography capacity and fidelity. We have demonstrated an ultra-fine fractional OAM holography with the topological charge resolution of 0.01. A 20-digit OAM-encoded holography encryption has also been exhibited. It harnesses only five angular OAM topological charges ranging from -16 to +16. The SBP efficiency is about 20 times larger than the conventional phase-only OAM holography. This work paves the way to compact, high-security and high-capacity holography.

Fossil Energy in the Framework of Sustainable Development: Analysis of Prospects and Development of Forecast Scenarios

In the next 20 years, the fossil energy must become a guarantor of the sustainable development of the energy sector for future generations. Significant threats represent hurdles in this transition. This study identified current global trends in the energy sector and the prospects for the development of energy until 2035. The importance of risk assessment in scenario forecasting based on expert judgments was proven. Three contrasting scenarios, #StayHome, #StayAlone, and #StayEffective, for the development of fossil energy, all based on comprehensive analysis of global risks by expert survey and factor analysis, were developed. It was concluded that fossil energy is mandatory with integration of advanced technologies at every stage of the production of traditional energy and of renewable energy as an integral part of the modern energy sector. Based on the results of the study, nine ambitious programs for the development of sustainable energy are presented. They require the creation and the utilization of a single interactive digital platform adapted to this purpose. It is a passport mandatory for the flexible interaction of energy production, its transmission, and its consumption in the perspective of having a future sustainable, reliable, and secured energy sector.

Materials for emergent silicon-integrated optical computing

Progress in computing architectures is approaching a paradigm shift: traditional computing based on digital complementary metal-oxide semiconductor technology is nearing physical limits in terms of miniaturization, speed, and, especially, power consumption. Consequently, alternative approaches are under investigation. One of the most promising is based on a "brain-like" or neuromorphic computation scheme. Another approach is quantum computing using photons. Both of these approaches can be realized using silicon photonics, and at the heart of both technologies is an efficient, ultra-low power broad band optical modulator. As silicon modulators suffer from relatively high power consumption, materials other than silicon itself have to be considered for the modulator. In this Perspective, we present our view on such materials. We focus on oxides showing a strong linear electro-optic effect that can also be integrated with Si, thus capitalizing on new materials to enable the devices and circuit architectures that exploit shifting computational machine learning paradigms, while leveraging current manufacturing infrastructure. This is expected to result in a new generation of computers that consume less power and possess a larger bandwidth.

Waveguide-lattice-based architecture for multichannel optical transformations

We consider waveguide lattices as the architecture to implement a wide range of multiport transformations. In this architecture, a particular transfer matrix is obtained by setting step-wise profiles of propagation constants experienced by a field evolving in a lattice. To investigate the capabilities of this architecture, we numerically study the implementation of random transfer matrices as well as several notable cases, such as the discrete Fourier transform, the Hadamard, and permutation matrices. We show that waveguide lattice schemes are more compact than their traditional lumped-parameter counterparts, thus the proposed architecture may be beneficial for photonic information processing systems of the future.

Scalable optical learning operator

Today's heavy machine learning tasks are fueled by large datasets. Computing is performed with power-hungry processors whose performance is ultimately limited by the data transfer to and from memory. Optics is a powerful means of communicating and processing information, and there is currently intense interest in optical information processing for realizing high-speed computations. Here we present and experimentally demonstrate an optical computing framework called scalable optical learning operator, which is based on spatiotemporal effects in multimode fibers for a range of learning tasks including classifying COVID-19 X-ray lung images, speech recognition and predicting age from images of faces. The presented framework addresses the energy scaling problem of existing systems without compromising speed. We leverage simultaneous, linear and nonlinear interaction of spatial modes as a computation engine. We numerically and experimentally show the ability of the method to execute several different tasks with accuracy comparable with a digital implementation.

Multiple-view D2NNs array: realizing robust 3D object recognition

As an optical-based classifier of the physical neural network, the independent diffractive deep neural network (D²NN) can be utilized to learn the single-view spatial featured mapping between the input lightfields and the truth labels by preprocessing a large number of training samples. However, it is still not enough to approach or even reach a satisfactory classification accuracy on three-dimensional (3D) targets owing to already losing lots of effective lightfield information on other view fields. This Letter presents a multiple-view D²NNs array (MDA) scheme that provides a significant inference improvement compared with individual D²NN or Res-D²NN by constructing a different complementary mechanism and then merging all base learners of distinct views on an electronic computer. Furthermore, a robust multiple-view D²NNs array (r-MDA) framework is demonstrated to resist the redundant spatial features of invalid lightfields due to severe optical disturbances.

Quo Vadis, Metasurfaces?

The full manipulation of intrinsic properties of electromagnetic waves has become the central target in various modern optical technologies. Optical metasurfaces have been suggested for a complete control of light-matter interaction with subwavelength structures, and they have been explored widely in the past decade for creating next-generation multifunctional flat-optics devices. The current studies of metasurfaces have reached a mature stage where common materials, basic optical physics, and conventional engineering tools have been explored extensively for various applications such as light bending, metalenses, metaholograms, and many others. A natural question is where the future research on metasurfaces will be going: Quo vadis, metasurfaces? In this Mini Review, we provide perspectives on the future developments of optical metasurfaces. Specifically, we highlight recent progresses on hybrid metasurfaces employing low-dimensional materials and discuss biomedical, computational, and quantum applications of metasurfaces, followed by discussions of challenges and foreseeing the future of metasurface physics and engineering.

Phase-stable repetition rate multiplication of dual-comb spectroscopy based on a cascaded Mach-Zehnder interferometer

In this Letter, we demonstrate a passive all-fiber pulse delay method for repetition rate multiplication of dual-comb spectroscopy. By combining a cascaded Mach-Zehnder interferometer and digital error correction, a mode-resolved spectrum with improved acquisition speed and sensitivity can be obtained. This technique has the strengths of compact, broadband, high energetic efficiency, and low complexity. Due to the use of an adaptive post-processing algorithm, sophisticated closed-loop feedback electronics are not required, which provides a simple and effective scheme to break through the physical limitation of the repetition frequency of the frequency comb for phase-stable dual-comb applications.

Robust and efficient single-pixel image classification with nonlinear optics

We present a hybrid image classifier by feature-sensitive image upconversion, single pixel photodetection, and deep learning, aiming at fast processing of high-resolution images. It uses partial Fourier transform to extract the images' signature features in both the original and Fourier domains, thereby significantly increasing the classification accuracy and robustness. Tested on the Modified National Institute of Standards and Technology handwritten digit images and verified by simulation, it boosts accuracy from 81.25% (by Fourier-domain processing) to 99.23%, and achieves 83% accuracy for highly contaminated images whose signal-to-noise ratio is only -17dB. Our approach could prove useful for fast lidar data processing, high-resolution image recognition, occluded target identification, and atmosphere monitoring.

Nanoprinted high-neuron-density optical linear perceptrons performing near-infrared inference on a CMOS chip

Optical machine learning has emerged as an important research area that, by leveraging the advantages inherent to optical signals, such as parallelism and high speed, paves the way for a future where optical hardware can process data at the speed of light. In this work, we present such optical devices for data processing in the form of single-layer nanoscale holographic perceptrons trained to perform optical inference tasks. We experimentally show the functionality of these passive optical devices in the example of decryptors trained to perform optical inference of single or whole classes of keys through symmetric and asymmetric decryption. The decryptors, designed for operation in the near-infrared region, are nanoprinted on complementary metal-oxide-semiconductor chips by galvo-dithered two-photon nanolithography with axial nanostepping of 10 nm1,2, achieving a neuron density of >500 million neurons per square centimetre. This power-efficient commixture of machine learning and on-chip integration may have a transformative impact on optical decryption3, sensing4, medical diagnostics5 and computing6,7.