Freely scalable and reconfigurable optical hardware for deep learning

BayesianSpikeFusion: accelerating spiking neural network inference via Bayesian fusion of early prediction

Spiking neural networks (SNNs) have garnered significant attention due to their notable energy efficiency. However, conventional SNNs rely on spike firing frequency to encode information, necessitating a fixed sampling time and leaving room for further optimization. This study presents a novel approach to reduce sampling time and conserve energy by extracting early prediction results from the intermediate layer of the network and integrating them with the final layer's predictions in a Bayesian fashion. Experimental evaluations conducted on image classification tasks using MNIST, CIFAR-10, and CIFAR-100 datasets demonstrate the efficacy of our proposed method when applied to VGGNets and ResNets models. Results indicate a substantial energy reduction of 38.8% in VGGNets and 48.0% in ResNets, illustrating the potential for achieving significant efficiency gains in spiking neural networks. These findings contribute to the ongoing research in enhancing the performance of SNNs, facilitating their deployment in resource-constrained environments. Our code is available on GitHub: https://github.com/hanebarla/BayesianSpikeFusion.

Single-shot optical neural network

Analog optical and electronic hardware has emerged as a promising alternative to digital electronics to improve the efficiency of deep neural networks (DNNs). However, previous work has been limited in scalability (input vector length K ≈ 100 elements) or has required nonstandard DNN models and retraining, hindering widespread adoption. Here, we present an analog, CMOS-compatible DNN processor that uses free-space optics to reconfigurably distribute an input vector and optoelectronics for static, updatable weighting and the nonlinearity-with K ≈ 1000 and beyond. We demonstrate single-shot-per-layer classification of the MNIST, Fashion-MNIST, and QuickDraw datasets with standard fully connected DNNs, achieving respective accuracies of 95.6, 83.3, and 79.0% without preprocessing or retraining. We also experimentally determine the fundamental upper bound on throughput (∼0.9 exaMAC/s), set by the maximum optical bandwidth before substantial increase in error. Our combination of wide spectral and spatial bandwidths enables highly efficient computing for next-generation DNNs.

Parallel photonic acceleration processor for matrix-matrix multiplication

We propose and experimentally demonstrate a highly parallel photonic acceleration processor based on a wavelength division multiplexing (WDM) system and a non-coherent Mach-Zehnder interferometer (MZI) array for matrix-matrix multiplication. The dimensional expansion is achieved by WDM devices, which play a crucial role in realizing matrix-matrix multiplication together with the broadband characteristics of an MZI. We implemented a 2 × 2 arbitrary nonnegative valued matrix using a reconfigurable 8 × 8 MZI array structure. Through experimentation, we verified that this structure could achieve 90.5% inference accuracy in a classification task for the Modified National Institute of Standards and Technology (MNIST) handwritten dataset. This provides a new effective solution for large-scale integrated optical computing systems based on convolution acceleration processors.

Photonic online learning: a perspective

Emerging neuromorphic hardware promises to solve certain problems faster and with higher energy efficiency than traditional computing by using physical processes that take place at the device level as the computational primitives in neural networks. While initial results in photonic neuromorphic hardware are very promising, such hardware requires programming or "training" that is often power-hungry and time-consuming. In this article, we examine the online learning paradigm, where the machinery for training is built deeply into the hardware itself. We argue that some form of online learning will be necessary if photonic neuromorphic hardware is to achieve its true potential.

Co-designed metaoptoelectronic deep learning

A metaoptical system is co-designed with electronic hardware to implement deep learning image recognition. The optical convolution block includes a reflective metasurface to perform one layer of a deep neural network. The optical and digital components are jointly optimized to perform an image classification task attaining 65% accuracy, which is close to the 66% accuracy of a fully-digital network where the optical block is replaced by a digital convolution layer.

Delocalized photonic deep learning on the internet's edge

Advanced machine learning models are currently impossible to run on edge devices such as smart sensors and unmanned aerial vehicles owing to constraints on power, processing, and memory. We introduce an approach to machine learning inference based on delocalized analog processing across networks. In this approach, named Netcast, cloud-based "smart transceivers" stream weight data to edge devices, enabling ultraefficient photonic inference. We demonstrate image recognition at ultralow optical energy of 40 attojoules per multiply (<1 photon per multiply) at 98.8% (93%) classification accuracy. We reproduce this performance in a Boston-area field trial over 86 kilometers of deployed optical fiber, wavelength multiplexed over 3 terahertz of optical bandwidth. Netcast allows milliwatt-class edge devices with minimal memory and processing to compute at teraFLOPS rates reserved for high-power (>100 watts) cloud computers.

Quantization-aware training for low precision photonic neural networks

Recent advances in Deep Learning (DL) fueled the interest in developing neuromorphic hardware accelerators that can improve the computational speed and energy efficiency of existing accelerators. Among the most promising research directions towards this is photonic neuromorphic architectures, which can achieve femtojoule per MAC efficiencies. Despite the benefits that arise from the use of neuromorphic architectures, a significant bottleneck is the use of expensive high-speed and precision analog-to-digital (ADCs) and digital-to-analog conversion modules (DACs) required to transfer the electrical signals, originating from the various Artificial Neural Networks (ANNs) operations (inputs, weights, etc.) in the photonic optical engines. The main contribution of this paper is to study quantization phenomena in photonic models, induced by DACs/ADCs, as an additional noise/uncertainty source and to provide a photonics-compliant framework for training photonic DL models with limited precision, allowing for reducing the need for expensive high precision DACs/ADCs. The effectiveness of the proposed method is demonstrated using different architectures, ranging from fully connected and convolutional networks to recurrent architectures, following recent advances in photonic DL.

Two-layer integrated photonic architectures with multiport photodetectors for high-fidelity and energy-efficient matrix multiplications

Photonic integrated circuits (PICs) are emerging as a promising tool for accelerating matrix multiplications in deep learning. Previous PIC architectures, primarily focusing on the matrix-vector multiplication (MVM), have large hardware errors that increase with the device scale. In this work, we propose a novel PIC architecture for MVM, which features an intrinsically small hardware error that does not increase with the device scale. Moreover, we further develop this concept and propose a PIC architecture for the general matrix-matrix multiplication (GEMM), which allows the GEMM to be directly performed on a photonic chip with a high energy efficiency unattainable by parallel or sequential MVMs. This work provides a promising approach to realize a high fidelity and high energy efficiency optical computing platform.

A Review of Optical Neural Networks

With the continuous miniaturization of conventional integrated circuits, obstacles such as excessive cost, increased resistance to electronic motion, and increased energy consumption are gradually slowing down the development of electrical computing and constraining the application of deep learning. Optical neuromorphic computing presents various opportunities and challenges compared with the realm of electronics. Algorithms running on optical hardware have the potential to meet the growing computational demands of deep learning and artificial intelligence. Here, we review the development of optical neural networks and compare various research proposals. We focus on fiber-based neural networks. Finally, we describe some new research directions and challenges.

Photonic matrix multiplication lights up photonic accelerator and beyond

Matrix computation, as a fundamental building block of information processing in science and technology, contributes most of the computational overheads in modern signal processing and artificial intelligence algorithms. Photonic accelerators are designed to accelerate specific categories of computing in the optical domain, especially matrix multiplication, to address the growing demand for computing resources and capacity. Photonic matrix multiplication has much potential to expand the domain of telecommunication, and artificial intelligence benefiting from its superior performance. Recent research in photonic matrix multiplication has flourished and may provide opportunities to develop applications that are unachievable at present by conventional electronic processors. In this review, we first introduce the methods of photonic matrix multiplication, mainly including the plane light conversion method, Mach-Zehnder interferometer method and wavelength division multiplexing method. We also summarize the developmental milestones of photonic matrix multiplication and the related applications. Then, we review their detailed advances in applications to optical signal processing and artificial neural networks in recent years. Finally, we comment on the challenges and perspectives of photonic matrix multiplication and photonic acceleration.

Harnessing optoelectronic noises in a photonic generative network

Integrated optoelectronics is emerging as a promising platform of neural network accelerator, which affords efficient in-memory computing and high bandwidth interconnectivity. The inherent optoelectronic noises, however, make the photonic systems error-prone in practice. It is thus imperative to devise strategies to mitigate and, if possible, harness noises in photonic computing systems. Here, we demonstrate a photonic generative network as a part of a generative adversarial network (GAN). This network is implemented with a photonic core consisting of an array of programable phase-change memory cells to perform four-element vector-vector dot multiplication. The GAN can generate a handwritten number ("7") in experiments and full 10 digits in simulation. We realize an optical random number generator, apply noise-aware training by injecting additional noise, and demonstrate the network's resilience to hardware nonidealities. Our results suggest the resilience and potential of more complex photonic generative networks based on large-scale, realistic photonic hardware.

An optical neural network using less than 1 photon per multiplication

Deep learning has become a widespread tool in both science and industry. However, continued progress is hampered by the rapid growth in energy costs of ever-larger deep neural networks. Optical neural networks provide a potential means to solve the energy-cost problem faced by deep learning. Here, we experimentally demonstrate an optical neural network based on optical dot products that achieves 99% accuracy on handwritten-digit classification using ~3.1 detected photons per weight multiplication and ~90% accuracy using ~0.66 photons (~2.5 × 10⁻¹⁹ J of optical energy) per weight multiplication. The fundamental principle enabling our sub-photon-per-multiplication demonstration—noise reduction from the accumulation of scalar multiplications in dot-product sums—is applicable to many different optical-neural-network architectures. Our work shows that optical neural networks can achieve accurate results using extremely low optical energies.

Though theory suggests that highly energy efficient optical neural networks (ONNs) based on optical matrix-vector multipliers are possible, an experimental validation is lacking. Here, the authors report an ONN with >90% accuracy image classification using <1 detected photon per scalar multiplication.