Two-dimensional programmable optical neural network

OP-FCNN: an optronic fully convolutional neural network for imaging through scattering media

Imaging through scattering media is a classical inverse issue in computational imaging. In recent years, deep learning(DL) methods have excelled in speckle reconstruction by extracting the correlation of speckle patterns. However, high-performance DL-based speckle reconstruction also costs huge hardware computation and energy consumption. Here, we develop an opto-electronic DL method with low computation complexity for imaging through scattering media. We design the "end-to-end" optronic structure for speckle reconstruction, namely optronic fully convolutional neural network (OP-FCNN). In OP-FCNN, we utilize lens groups and spatial light modulators to implement the convolution, down/up-sampling, and skip connection in optics, which significantly reduces the computational complexity by two orders of magnitude, compared with the digital CNN. Moreover, the reconfigurable and scalable structure supports the OP-FCNN to further improve imaging performance and accommodate object datasets of varying complexity. We utilize MNIST handwritten digits, EMNIST handwritten letters, fashion MNIST, and MIT-CBCL-face datasets to validate the OP-FCNN imaging performance through random diffusers. Our OP-FCNN reveals a good balance between computational complexity and imaging performance. The average imaging performance on four datasets achieves 0.84, 0.91, 0.79, and 16.3dB for JI, PCC, SSIM, and PSNR, respectively. The OP-FCNN paves the way for all-optical systems in imaging through scattering media.

Neuromorphic Photonics Circuits: Contemporary Review

Neuromorphic photonics is a cutting-edge fusion of neuroscience-inspired computing and photonics technology to overcome the constraints of conventional computing architectures. Its significance lies in the potential to transform information processing by mimicking the parallelism and efficiency of the human brain. Using optics and photonics principles, neuromorphic devices can execute intricate computations swiftly and with impressive energy efficiency. This innovation holds promise for advancing artificial intelligence and machine learning while addressing the limitations of traditional silicon-based computing. Neuromorphic photonics could herald a new era of computing that is more potent and draws inspiration from cognitive processes, leading to advancements in robotics, pattern recognition, and advanced data processing. This paper reviews the recent developments in neuromorphic photonic integrated circuits, applications, and current challenges.

Real-time wavefront correction using diffractive optical networks

Real-time wavefront correction is a challenging problem to present for conventional adaptive optics systems. Here, we present an all-optical system to realize real-time wavefront correction. Using deep learning, the system, which contains only multiple transmissive diffractive layers, is trained to realize high-quality imaging for unknown, random, distorted wavefronts. Once physically fabricated, this passive optical system is physically positioned between the imaging lens and the image plane to all-optically correct unknown, new wavefronts whose wavefront errors are within the training range. Simulated experiments showed that the system designed for the on-axis field of view increases the average imaging Strehl Ratio from 0.32 to 0.94, and the other system intended for multiple fields of view increases the resolvable probability of binary stars from 30.5% to 69.5%. Results suggested that DAOS performed well when performing wavefront correction at the speed of light. The solution of real-time wavefront correction can be applied to other wavelengths and has great application potential in astronomical observation, laser communication, and other fields.

Knowledge distillation circumvents nonlinearity for optical convolutional neural networks

In recent years, convolutional neural networks (CNNs) have enabled ubiquitous image processing applications. As such, CNNs require fast forward propagation runtime to process high-resolution visual streams in real time. This is still a challenging task even with state-of-the-art graphics and tensor processing units. The bottleneck in computational efficiency primarily occurs in the convolutional layers. Performing convolutions in the Fourier domain is a promising way to accelerate forward propagation since it transforms convolutions into elementwise multiplications, which are considerably faster to compute for large kernels. Furthermore, such computation could be implemented using an optical 4f system with orders of magnitude faster operation. However, a major challenge in using this spectral approach, as well as in an optical implementation of CNNs, is the inclusion of a nonlinearity between each convolutional layer, without which CNN performance drops dramatically. Here, we propose a spectral CNN linear counterpart (SCLC) network architecture and its optical implementation. We propose a hybrid platform with an optical front end to perform a large number of linear operations, followed by an electronic back end. The key contribution is to develop a knowledge distillation (KD) approach to circumvent the need for nonlinear layers between the convolutional layers and successfully train such networks. While the KD approach is known in machine learning as an effective process for network pruning, we adapt the approach to transfer the knowledge from a nonlinear network (teacher) to a linear counterpart (student), where we can exploit the inherent parallelism of light. We show that the KD approach can achieve performance that easily surpasses the standard linear version of a CNN and could approach the performance of the nonlinear network. Our simulations show that the possibility of increasing the resolution of the input image allows our proposed 4f optical linear network to perform more efficiently than a nonlinear network with the same accuracy on two fundamental image processing tasks: (i) object classification and (ii) semantic segmentation.

Space-efficient optical computing with an integrated chip diffractive neural network

Large-scale, highly integrated and low-power-consuming hardware is becoming progressively more important for realizing optical neural networks (ONNs) capable of advanced optical computing. Traditional experimental implementations need N² units such as Mach-Zehnder interferometers (MZIs) for an input dimension N to realize typical computing operations (convolutions and matrix multiplication), resulting in limited scalability and consuming excessive power. Here, we propose the integrated diffractive optical network for implementing parallel Fourier transforms, convolution operations and application-specific optical computing using two ultracompact diffractive cells (Fourier transform operation) and only N MZIs. The footprint and energy consumption scales linearly with the input data dimension, instead of the quadratic scaling in the traditional ONN framework. A ~10-fold reduction in both footprint and energy consumption, as well as equal high accuracy with previous MZI-based ONNs was experimentally achieved for computations performed on the MNIST and Fashion-MNIST datasets. The integrated diffractive optical network (IDNN) chip demonstrates a promising avenue towards scalable and low-power-consumption optical computational chips for optical-artificial-intelligence.

Here, we propose the integrated diffractive optical network for implementing parallel Fourier transforms, convolution operations and application-specific optical computing with reduced footprint and energy consumption.

Design framework for metasurface optics-based convolutional neural networks

Deep learning using convolutional neural networks (CNNs) has been shown to significantly outperform many conventional vision algorithms. Despite efforts to increase the CNN efficiency both algorithmically and with specialized hardware, deep learning remains difficult to deploy in resource-constrained environments. In this paper, we propose an end-to-end framework to explore how to optically compute the CNNs in free-space, much like a computational camera. Compared to existing free-space optics-based approaches that are limited to processing single-channel (i.e., gray scale) inputs, we propose the first general approach, based on nanoscale metasurface optics, that can process RGB input data. Our system achieves up to an order of magnitude energy savings and simplifies the sensor design, all the while sacrificing little network accuracy.

Optronic convolutional neural networks of multi-layers with different functions executed in optics for image classification

Although deeper convolutional neural networks (CNNs) generally obtain better performance on classification tasks, they incur higher computation costs. To address this problem, this study proposes the optronic convolutional neural network (OPCNN) in which all computation operations are executed in optics, and data transmission and control are executed in electronics. In OPCNN, we implement convolutional layers with multi input images by the lenslet 4f system, downsampling layers by optical-strided convolution and obtaining nonlinear activation by adjusting the camera's curve and fully connected layers by optical dot product. The OPCNN demonstrates good performance on the classification tasks in simulations and experiments and achieves better performance than other current optical convolutional neural networks by comparison due to the more complex architecture. The scalability of OPCNN contributes to building deeper networks when facing complicated datasets.

Optical frontend for a convolutional neural network

The parallelism of optics and the miniaturization of optical components using nanophotonic structures, such as metasurfaces, present a compelling alternative to electronic implementations of convolutional neural networks. The lack of a low-power optical nonlinearity, however, requires slow and energy-inefficient conversions between the electronic and optical domains. Here, we design an architecture that utilizes a single electrical to optical conversion by designing a free-space optical frontend unit that implements the linear operations of the first layer with the subsequent layers realized electronically. Speed and power analysis of the architecture indicates that the hybrid photonic-electronic architecture outperforms a fully electronic architecture for large image sizes and kernels. Benchmarking of the photonic-electronic architecture on a modified version of AlexNet achieves high classification accuracies on images from the Kaggle's Cats and Dogs challenge and MNIST databases.

Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification

Convolutional neural networks (CNNs) excel in a wide variety of computer vision applications, but their high performance also comes at a high computational cost. Despite efforts to increase efficiency both algorithmically and with specialized hardware, it remains difficult to deploy CNNs in embedded systems due to tight power budgets. Here we explore a complementary strategy that incorporates a layer of optical computing prior to electronic computing, improving performance on image classification tasks while adding minimal electronic computational cost or processing time. We propose a design for an optical convolutional layer based on an optimized diffractive optical element and test our design in two simulations: a learned optical correlator and an optoelectronic two-layer CNN. We demonstrate in simulation and with an optical prototype that the classification accuracies of our optical systems rival those of the analogous electronic implementations, while providing substantial savings on computational cost.

Algorithm for optimizing bipolar interconnection weights with applications in associative memories and multitarget classification

An algorithm for optimizing a bipolar interconnection weight matrix with the Hopfield network is proposed. The effectiveness of this algorithm is demonstrated by computer simulation and optical implementation. In the optical implementation of the neural network the interconnection weights are biased to yield a nonnegative weight matrix. Moreover, a threshold subchannel is added so that the system can realize, in real time, the bipolar weighted summation in a single channel. Preliminary experimental results obtained from the applications in associative memories and multitarget classification with rotation invariance are shown.

Implementation of a large-scale optical neural network by use of a coaxial lenslet array for interconnection

Optical implementation of a large-scale neural network with 32 x 32 neurons is reported. The experimental setup is described, error caused by limited precision of hardware is analyzed, and experimental results are presented.

Power distribution in two-dimensional optical network channels

The power distribution in two-dimensional optical network channels is analyzed. The maximum number of allowable channels as determined by the characteristics of optical detector is identified, in particular, for neural-network and wavelet-transform applications.

Optical associative memory with bipolar edge-enhanced learning that uses a binary spatial light modulator and a BaTiO(3) crystal

An optical associative memory with bipolar edge-enhanced feature learning that uses a ferroelectric liquid-crystal spatial light modulator and a barium titanate crystal is presented. During the learning procedure the bipolar edge-enhanced versions of the patterns are employed, which enable the associative memory to have a high discrimination capability. Experimental results and computer simulations are given.

Coaxial architecture of an optical neural network with a lenslet array

A theoretical limitation on the neuron numbers (N x N) of a spatial interconnection system based on a lenslet array is analyzed. To improve the limitation, we propose a submatrix coaxial interconnection architecture. Off-axial aberration is decreased, and high light efficiency is achieved. A detailed analysis is given.

Incoherent dynamic lenslet array processor

The use of a dynamic lenslet array processor for the implementation of unipolar and bipolar analog inner product, outer product, and vector sum operations is described. Its matrix-vector operations are used as a basis for neural networks and digital circuits. Experimental results of two circuits are presented: a unipolar neural network that computes parity of a 3-bit input word and a digital 3-to-8 decoder circuit.

Four-quadrant optical matrix-vector multiplication machine as a neural-network processor

Optical processors for neural networks are primarily fast matrix-vector multiplication machines that can potentially compete with serial computers owing to their parallelism and their ability to facilitate densely connected networks. However, in most proposed systems the multiplication supports only two quadrants and is thus unable to provide bipolar neuron outputs for increasing network capabilities and learning rate. We propose and demonstrate an opto-electronic four-quadrant matrix-vector multiplier that can be used for feed-forward neural-network recall and learning. Experimental results obtained with common commercial components demonstrate a novel, useful, and reliable approach for fourquadrant matrix-vector multiplication in general and for feed-forward neural-network training and recall in particular.

Optical implementation of the Hamming net

We present an optical implementation of the Hamming net that can be used as an optimum image classifier or an associative memory. We introduce a modified Hamming net, in which the dynamic range requirement of the spatial light modulator can be relaxed and the number of iteration cycles in the second layer (or maxnet) can be reduced. Experimental demonstrations of the optical implementation of the Hamming net are also given.

Incoherent optical associative memory by using synthetic discriminant function filters

A new associative memory system applying incoherent correlators is presented. With this system, which is composed of a nonlinear processor and optical-electronic feedback loop, memorized patterns can be recalled without cross talk. To realize a high association ability and stable operation, a synthetic discriminant function filter is applied. Results of the computer simulation and actual experiments by using the system that is constructed here are demonstrated.

Redundant-interconnection interpattern-association neural network

We have shown that introducing interconnection redundancy can make a neural network more robust. We describe performances under noisy input and partial input that show that the optimum-redundant interconnection improves both the noise tolerance and the pattern discriminability. Simulated and experimental demonstrations are also provided.

Mirror-array optical interconnected neural network

A high-light-efficiency optical neural network that uses a mirror-array interconnection is proposed. Design considerations for the mirror array and experimental demonstration are given.

Space-time-sharing optical neural network

A space-time-sharing optical neural network for implementing a large-scale operation is presented. If the interconnection weight matrix is partitioned into an array of submatrices, a large space-bandwidth pattern can be processed with a smaller neural network. We show that the processing time increases as a square function of the space-bandwidth product of the pattern. To illustrate the space-time-sharing operation, experimental and simulated results are provided.

Optical associative memory using an orthogonalized hologram

A two-dimensional orthogonal model of optical associative memory for storage of nonnegative patterns using a single hologram is presented. Two sets of patterns after prior orthogonal processing are composed and used for hologram recording, which permits the sequential holographic recordings to be carried on spatially separated regions of the recording material. High diffraction efficiency of the hologram is achieved. Computer simulations and optical demonstrations are also given.

Optical neural network with pocket-sized liquid-crystal televisions

A compact optical neural network that uses high-resolution liquid-crystal televisions has been constructed. System design considerations and an experimental demonstration of the liquid-crystal television neural network are reported.

Neural network model using interpattern association

This paper investigates a neural network model-interpattern association (IPA) model-in which the basic logical operations are used to determine the interpattern association (i.e., association between the reference patterns), and simple logical rules are applied to construct tristate interconnections in the network. Computer simulations for the reconstruction of similar English letters embedded in the random noise by the IPA model have shown improved performance compared with the Hopfield model. A 2-D hybrid optical neural network is used to demonstrate the usefulness of the IPA model. Since there are only three gray levels used in the interconnection weight matrix for the IPA model, the dynamic range imposed on a spatial light modulator is rather relaxed, and the interconnections are much simpler than the Hopfield model.