Photonic matrix multiplication lights up photonic accelerator and beyond

Revolutionizing breast cancer Ki-67 diagnosis: ultrasound radiomics and fully connected neural networks (FCNN) combination method

PURPOSE

This study aims to assess the diagnostic value of ultrasound habitat sub-region radiomics feature parameters using a fully connected neural networks (FCNN) combination method L2,1-norm in relation to breast cancer Ki-67 status.

METHODS

Ultrasound images from 528 cases of female breast cancer at the Affiliated Hospital of Xiangnan University and 232 cases of female breast cancer at the Affiliated Rehabilitation Hospital of Xiangnan University were selected for this study. We utilized deep learning methods to automatically outline the gross tumor volume and perform habitat clustering. Subsequently, habitat sub-regions were extracted to identify radiomics features and underwent feature engineering using the L1,2-norm. A prediction model for the Ki-67 status of breast cancer patients was then developed using a FCNN. The model's performance was evaluated using accuracy, area under the curve (AUC), specificity (Spe), positive predictive value (PPV), negative predictive value (NPV), Recall, and F1. In addition, calibration curves and clinical decision curves were plotted for the test set to visually assess the predictive accuracy and clinical benefit of the models.

RESULT

Based on the feature engineering using the L1,2-norm, a total of 9 core features were identified. The predictive model, constructed by the FCNN model based on these 9 features, achieved the following scores: ACC 0.856, AUC 0.915, Spe 0.843, PPV 0.920, NPV 0.747, Recall 0.974, and F1 0.890. Furthermore, calibration curves and clinical decision curves of the validation set demonstrated a high level of confidence in the model's performance and its clinical benefit.

CONCLUSION

Habitat clustering of ultrasound images of breast cancer is effectively supported by the combined implementation of the L1,2-norm and FCNN algorithms, allowing for the accurate classification of the Ki-67 status in breast cancer patients.

Partial coherence enhances parallelized photonic computing

Advancements in optical coherence control1-5 have unlocked many cutting-edge applications, including long-haul communication, light detection and ranging (LiDAR) and optical coherence tomography6-8. Prevailing wisdom suggests that using more coherent light sources leads to enhanced system performance and device functionalities9-11. Our study introduces a photonic convolutional processing system that takes advantage of partially coherent light to boost computing parallelism without substantially sacrificing accuracy, potentially enabling larger-size photonic tensor cores. The reduction of the degree of coherence optimizes bandwidth use in the photonic convolutional processing system. This breakthrough challenges the traditional belief that coherence is essential or even advantageous in integrated photonic accelerators, thereby enabling the use of light sources with less rigorous feedback control and thermal-management requirements for high-throughput photonic computing. Here we demonstrate such a system in two photonic platforms for computing applications: a photonic tensor core using phase-change-material photonic memories that delivers parallel convolution operations to classify the gaits of ten patients with Parkinson's disease with 92.2% accuracy (92.7% theoretically) and a silicon photonic tensor core with embedded electro-absorption modulators (EAMs) to facilitate 0.108 tera operations per second (TOPS) convolutional processing for classifying the Modified National Institute of Standards and Technology (MNIST) handwritten digits dataset with 92.4% accuracy (95.0% theoretically).

Multimodal deep learning using on-chip diffractive optics with in situ training capability

Multimodal deep learning plays a pivotal role in supporting the processing and learning of diverse data types within the realm of artificial intelligence generated content (AIGC). However, most photonic neuromorphic processors for deep learning can only handle a single data modality (either vision or audio) due to the lack of abundant parameter training in optical domain. Here, we propose and demonstrate a trainable diffractive optical neural network (TDONN) chip based on on-chip diffractive optics with massive tunable elements to address these constraints. The TDONN chip includes one input layer, five hidden layers, and one output layer, and only one forward propagation is required to obtain the inference results without frequent optical-electrical conversion. The customized stochastic gradient descent algorithm and the drop-out mechanism are developed for photonic neurons to realize in situ training and fast convergence in the optical domain. The TDONN chip achieves a potential throughput of 217.6 tera-operations per second (TOPS) with high computing density (447.7 TOPS/mm2), high system-level energy efficiency (7.28 TOPS/W), and low optical latency (30.2 ps). The TDONN chip has successfully implemented four-class classification in different modalities (vision, audio, and touch) and achieve 85.7% accuracy on multimodal test sets. Our work opens up a new avenue for multimodal deep learning with integrated photonic processors, providing a potential solution for low-power AI large models using photonic technology.

A perspective on the artificial intelligence's transformative role in advancing diffractive optics

Artificial intelligence (AI) is transforming diffractive optics development through its advanced capabilities in design optimization, pattern generation, fabrication enhancement, performance forecasting, and customization. Utilizing AI algorithms like machine learning, generative models, and transformers, researchers can analyze extensive datasets to refine the design of diffractive optical elements (DOEs) tailored to specific applications and performance requirements. AI-driven pattern generation methods enable the creation of intricate and efficient optical structures that manipulate light with exceptional precision. Furthermore, AI optimizes manufacturing processes by fine-tuning fabrication parameters, resulting in higher quality and productivity. AI models also simulate diffractive optics behavior, accelerating design iterations and facilitating rapid prototyping. This integration of AI into diffractive optics holds tremendous potential to revolutionize optical technology applications across diverse sectors, spanning from imaging and sensing to telecommunications and beyond.

I/O-efficient iterative matrix inversion with photonic integrated circuits

Photonic integrated circuits have been extensively explored for optical processing with the aim of breaking the speed and energy efficiency bottlenecks of digital electronics. However, the input/output (IO) bottleneck remains one of the key barriers. Here we report a photonic iterative processor (PIP) for matrix-inversion-intensive applications. The direct reuse of inputted data in the optical domain unlocks the potential to break the IO bottleneck. We demonstrate notable IO advantages with a lossless PIP for real-valued matrix inversion and integral-differential equation solving, as well as a coherent PIP with optical loops integrated on-chip, enabling complex-valued computation and a net inversion time of 1.2 ns. Furthermore, we estimate at least an order of magnitude enhancement in IO efficiency of a PIP over photonic single-pass processors and the state-of-the-art electronic processors for reservoir training tasks and multiple-input and multiple-output (MIMO) precoding tasks, indicating the huge potential of PIP technology in practical applications.

On-chip silicon photonic micro-ring processor lights up optical image encryption

Optical image encryption has long been an important concept in the fields of photonic network processing and communication. Here, we propose a convolution-like operation-based optical image encryption algorithm exploiting a silicon photonic multiplexing architecture to achieve content security. Particularly, the encryption process is completed in a 3 × 3 cross-shaped photonic micro-ring resonator (MRR) array on chip. For the first time, to the best of our knowledge, this algorithm encodes information in an integrated intensity modulation, effectively reducing the encoding difficulty. Moreover, the high reliability and scalability of optical encryption are ensured using both linear and nonlinear operations on photonic chips according to characteristics of MRRs. As the encryption and decryption experiments show, the image restoration accuracy of our optical encryption algorithm exceeds 99% under real system noise at the pixel level, indicating its noise-robust property. Meanwhile, the peak signal-to-noise ratios of the restored and encrypted images are >60 and <15 dB, respectively, revealing both the high accuracy of the restored image and the small correlation between the encrypted and original images. This work adds to the rapidly expanding field of optical image encryption on photonic chips.

The Goldilocks principle of learning unitaries by interlacing fixed operators with programmable phase shifters on a photonic chip

Programmable photonic integrated circuits represent an emerging technology that amalgamates photonics and electronics, paving the way for light-based information processing at high speeds and low power consumption. Programmable photonics provides a flexible platform that can be reconfigured to perform multiple tasks, thereby holding great promise for revolutionizing future optical networks and quantum computing systems. Over the past decade, there has been constant progress in developing several different architectures for realizing programmable photonic circuits that allow for realizing arbitrary discrete unitary operations with light. Here, we systematically investigate a general family of photonic circuits for realizing arbitrary unitaries based on a simple architecture that interlaces a fixed intervening layer with programmable phase shifter layers. We introduce a criterion for the intervening operator that guarantees the universality of this architecture for representing arbitrary N × N unitary operators with N + 1 phase layers. We explore this criterion for different photonic components, including photonic waveguide lattices and meshes of directional couplers, which allows the identification of several families of photonic components that can serve as the intervening layers in the interlacing architecture. Our findings pave the way for efficiently designing and realizing novel families of programmable photonic integrated circuits for multipurpose analog information processing.

Multiplexable all-optical nonlinear activator for optical computing

As an alternative solution to surpass electronic neural networks, optical neural networks (ONNs) offer significant advantages in terms of energy consumption and computing speed. Despite the optical hardware platform could provide an efficient approach to realizing neural network algorithms than traditional hardware, the lack of optical nonlinearity limits the development of ONNs. Here, we proposed and experimentally demonstrated an all-optical nonlinear activator based on the stimulated Brillouin scattering (SBS). Utilizing the exceptional carrier dynamics of SBS, our activator supports two types of nonlinear functions, saturable absorption and rectified linear unit (Relu) models. Moreover, the proposed activator exhibits large dynamic response bandwidth (∼11.24 GHz), low nonlinear threshold (∼2.29 mW), high stability, and wavelength division multiplexing identities. These features have potential advantages for the physical realization of optical nonlinearities. As a proof of concept, we verify the performance of the proposed activator as an ONN nonlinear mapping unit via numerical simulations. Simulation shows that our approach achieves comparable performance to the activation functions commonly used in computers. The proposed approach provides support for the realization of all-optical neural networks.

On-chip photonic convolution by phase-change in-memory computing cells with quasi-continuous tuning

Matrix multiplication acceleration by on-chip photonic integrated circuits (PICs) is emerging as one of the attractive and promising solutions, offering outstanding benefits in speed and bandwidth as compared to non-photonic approaches. Incorporating nonvolatile phase-change materials into PICs or devices enables optical storage and computing, surpassing their electrical counterparts. In this paper, we propose a design of on-chip photonic convolution for optical in-memory computing by integrating the phase change chalcogenide of Ge2Sb2Se4Te1 (GSST) into an asymmetric directional coupler for constructions of an in-memory computing cell, marrying the advantages of both the large bandwidth of Mach-Zehnder interferometers (MZIs) and the small size of micro-ring resonators (MRRs). Through quasi-continuous electro-thermal tuning of the GSST-integrated in-memory computing cells, numerical calculations about the optical and electro-thermal behaviors during GSST phase transition confirm the tunability of the programmable elements stored in the in-memory computing cells within [-1, 1]. For proof-of-concept verification, we apply the proposed optical convolutional kernel to a typical image edge detection application. As evidenced by the evaluation results, the prototype achieves the same accuracy as the convolution kernel implemented on a common digital computer, demonstrating the feasibility of the proposed scheme for on-chip photonic convolution and optical in-memory computing.

Programmable Photonic Time Circuits for Highly Scalable Universal Unitaries

Programmable photonic circuits (PPCs) have garnered substantial interest for their potential in facilitating deep learning accelerations and universal quantum computations. Although photonic computation using PPCs offers ultrafast operation, energy-efficient matrix calculations, and room-temperature quantum states, its poor scalability hinders integration. This challenge arises from the temporally one-shot operation of propagating light in conventional PPCs, resulting in a light-speed increase in device footprints. Here we propose the concept of programmable photonic time circuits, utilizing time-cycle-based computations analogous to gate cycling in the von Neumann architecture and quantum computation. Our building block is a reconfigurable SU(2) time gate, consisting of two resonators with tunable resonances, and coupled via time-coded dual-channel gauge fields. We demonstrate universal U(N) operations with high fidelity using an assembly of the SU(2) time gates, substantially improving scalability from O(N^{2}) to O(N) in terms of both the footprint and the number of gates. This result paves the way for PPC implementation in very large-scale integration.

Application of a reconfigurable all-optical activation unit based on optical injection into a bistable Fabry-Perot laser in multilayer perceptron neural networks

In this Letter, we theoretically investigate the application of a bistable Fabry-Perot semiconductor laser under optical injection as an all-optical activation unit for multilayer perceptron optical neural networks. The proposed device is programmed to provide reconfigurable sigmoid-like activation functions with adjustable thresholds and saturation points and benchmarked on machine learning image recognition problems. Due to the reconfigurability of the activation unit, the accuracy can be increased by up to 2% simply by adjusting the control parameter of the activation unit to suit the specific problem. For a simple two-layer perceptron neural network, we achieve inference accuracies of up to 95% and 85%, for the MNIST and Fashion-MNIST datasets, respectively.

Conflict-free joint decision by lag and zero-lag synchronization in laser network

With the end of Moore's Law and the increasing demand for computing, photonic accelerators are garnering considerable attention. This is due to the physical characteristics of light, such as high bandwidth and multiplicity, and the various synchronization phenomena that emerge in the realm of laser physics. These factors come into play as computer performance approaches its limits. In this study, we explore the application of a laser network, acting as a photonic accelerator, to the competitive multi-armed bandit problem. In this context, conflict avoidance is key to maximizing environmental rewards. We experimentally demonstrate cooperative decision-making using zero-lag and lag synchronization within a network of four semiconductor lasers. Lag synchronization of chaos realizes effective decision-making and zero-lag synchronization is responsible for the realization of the collision avoidance function. We experimentally verified a low collision rate and high reward in a fundamental 2-player, 2-slot scenario, and showed the scalability of this system. This system architecture opens up new possibilities for intelligent functionalities in laser dynamics.

Parallel edge extraction operators on chip speed up photonic convolutional neural networks

We experimentally establish a 3 × 3 cross-shaped micro-ring resonator (MRR) array-based photonic multiplexing architecture relying on silicon photonics to achieve parallel edge extraction operations in images for photonic convolution neural networks. The main mathematical operations involved are convolution. Precisely, a faster convolutional calculation speed of up to four times is achieved by extracting four feature maps simultaneously with the same photonic hardware's structure and power consumption, where a maximum computility of 0.742 TOPS at an energy cost of 48.6 mW and a convolution accuracy of 95.1% is achieved in an MRR array chip. In particular, our experimental results reveal that this system using parallel edge extraction operators instead of universal operators can improve the imaging recognition accuracy for CIFAR-10 dataset by 6.2% within the same computing time, reaching a maximum of 78.7%. This work presents high scalability and efficiency of parallel edge extraction chips, furnishing a novel, to the best of our knowledge, approach to boost photonic computing speed.

Optical ReLU-like activation function based on a semiconductor laser with optical injection

Artificial neural networks usually consist of successive linear multiply-accumulate operations and nonlinear activation functions. However, most optical neural networks only achieve the linear operation in the optical domain, while the optical implementation of activation function remains challenging. Here we present an optical ReLU-like activation function (with 180° rotation) based on a semiconductor laser subject to the optical injection in an experiment. The ReLU-like function is achieved in a broad regime above the Hopf bifurcation of the injection-locking diagram and is operated in the continuous-wave mode. In particular, the slope of the activation function is reconfigurable by tuning the frequency difference between the master laser and the slave laser.

Orbital angular momentum-mediated machine learning for high-accuracy mode-feature encoding

Machine learning with optical neural networks has featured unique advantages of the information processing including high speed, ultrawide bandwidths and low energy consumption because the optical dimensions (time, space, wavelength, and polarization) could be utilized to increase the degree of freedom. However, due to the lack of the capability to extract the information features in the orbital angular momentum (OAM) domain, the theoretically unlimited OAM states have never been exploited to represent the signal of the input/output nodes in the neural network model. Here, we demonstrate OAM-mediated machine learning with an all-optical convolutional neural network (CNN) based on Laguerre-Gaussian (LG) beam modes with diverse diffraction losses. The proposed CNN architecture is composed of a trainable OAM mode-dispersion impulse as a convolutional kernel for feature extraction, and deep-learning diffractive layers as a classifier. The resultant OAM mode-dispersion selectivity can be applied in information mode-feature encoding, leading to an accuracy as high as 97.2% for MNIST database through detecting the energy weighting coefficients of the encoded OAM modes, as well as a resistance to eavesdropping in point-to-point free-space transmission. Moreover, through extending the target encoded modes into multiplexed OAM states, we realize all-optical dimension reduction for anomaly detection with an accuracy of 85%. Our work provides a deep insight to the mechanism of machine learning with spatial modes basis, which can be further utilized to improve the performances of various machine-vision tasks by constructing the unsupervised learning-based auto-encoder.

Analog spatiotemporal feature extraction for cognitive radio-frequency sensing with integrated photonics

Analog feature extraction (AFE) is an appealing strategy for low-latency and efficient cognitive sensing systems since key features are much sparser than the Nyquist-sampled data. However, applying AFE to broadband radio-frequency (RF) scenarios is challenging due to the bandwidth and programmability bottlenecks of analog electronic circuitry. Here, we introduce a photonics-based scheme that extracts spatiotemporal features from broadband RF signals in the analog domain. The feature extractor structure inspired by convolutional neural networks is implemented on integrated photonic circuits to process RF signals from multiple antennas, extracting valid features from both temporal and spatial dimensions. Because of the tunability of the photonic devices, the photonic spatiotemporal feature extractor is trainable, which enhances the validity of the extracted features. Moreover, a digital-analog-hybrid transfer learning method is proposed for the effective and low-cost training of the photonic feature extractor. To validate our scheme, we demonstrate a radar target recognition task with a 4-GHz instantaneous bandwidth. Experimental results indicate that the photonic analog feature extractor tackles broadband RF signals and reduces the sampling rate of analog-to-digital converters to 1/4 of the Nyquist sampling while maintaining a high target recognition accuracy of 97.5%. Our scheme offers a promising path for exploiting the AFE strategy in the realm of cognitive RF sensing, with the potential to contribute to the efficient signal processing involved in applications such as autonomous driving, robotics, and smart factories.

Correlated optical convolutional neural network with "quantum speedup"

Compared with electrical neural networks, optical neural networks (ONNs) have the potentials to break the limit of the bandwidth and reduce the consumption of energy, and therefore draw much attention in recent years. By far, several types of ONNs have been implemented. However, the current ONNs cannot realize the acceleration as powerful as that indicated by the models like quantum neural networks. How to construct and realize an ONN with the quantum speedup is a huge challenge. Here, we propose theoretically and demonstrate experimentally a new type of optical convolutional neural network by introducing the optical correlation. It is called the correlated optical convolutional neural network (COCNN). We show that the COCNN can exhibit "quantum speedup" in the training process. The character is verified from the two aspects. One is the direct illustration of the faster convergence by comparing the loss function curves of the COCNN with that of the traditional convolutional neural network (CNN). Such a result is compatible with the training performance of the recently proposed quantum convolutional neural network (QCNN). The other is the demonstration of the COCNN's capability to perform the QCNN phase recognition circuit, validating the connection between the COCNN and the QCNN. Furthermore, we take the COCNN analog to the 3-qubit QCNN phase recognition circuit as an example and perform an experiment to show the soundness and the feasibility of it. The results perfectly match the theoretical calculations. Our proposal opens up a new avenue for realizing the ONNs with the quantum speedup, which will benefit the information processing in the era of big data.

Effects of cavity nonlinearities and linear losses on silicon microring-based reservoir computing

Microring resonators (MRRs) are promising devices for time-delay photonic reservoir computing, but the impact of the different physical effects taking place in the MRRs on the reservoir computing performance is yet to be fully understood. We numerically analyze the impact of linear losses as well as thermo-optic and free-carrier effects relaxation times on the prediction error of the time-series task NARMA-10. We demonstrate the existence of three regions, defined by the input power and the frequency detuning between the optical source and the microring resonance, that reveal the cavity transition from linear to nonlinear regimes. One of these regions offers very low error in time-series prediction under relatively low input power and number of nodes while the other regions either lack nonlinearity or become unstable. This study provides insight into the design of the MRR and the optimization of its physical properties for improving the prediction performance of time-delay reservoir computing.

Iterative eigensolver using fixed-point photonic primitive

Photonic computing has potential advantages in speed and energy consumption yet is subject to inaccuracy due to the limited equivalent bitwidth of the analog signal. In this Letter, we demonstrate a configurable, fixed-point coherent photonic iterative solver for numerical eigenvalue problems using shifted inverse iteration. The photonic primitive can accommodate arbitrarily sized sparse matrix-vector multiplication and is deployed to solve eigenmodes in a photonic waveguide structure. The photonic iterative eigensolver does not accumulate errors from each iteration, providing a path toward implementing scientific computing applications on photonic primitives.

A Design of High-Efficiency: Vertical Accumulation Modulators Based on Silicon Photonics

On-chip optical modulators, which are capable of converting electrical signals into optical signals, constitute the foundational components of photonic devices. Photonics modulators exhibiting high modulation efficiency and low insertion loss are highly sought after in numerous critical applications, such as optical phase steering, optical coherent imaging, and optical computing. This paper introduces a novel accumulation-type vertical modulator structure based on a silicon photonics platform. By incorporating a high-K dielectric layer of ZrO2, we have observed an increase in modulation efficiency while maintaining relatively low levels of modulation loss. Through meticulous study and optimization, the simulation results of the final device structure demonstrate a modulation efficiency of 0.16 V·cm, with a mere efficiency-loss product of 8.24 dB·V.

Hybrid lithium tantalite-silicon integrated photonics platform for electro-optic modulation

Integrated electro-optic modulators are key components in photonic integrated circuits. Silicon photonic technology is considered to be promising for large-scale and low-cost integration. However, silicon does not exhibit any Pockels effect, and the electro-optic modulator based on free-carrier dispersion suffers from challenges such as high-power consumption, limited bandwidth, and large optical propagation loss. Here, a new, to the best of our knowledge, hybrid lithium tantalite-silicon platform is proposed for electro-optic modulators based on the Pockels effect. Benefiting from the strong Pockels coefficients of a thin-film lithium tantalite, a hybrid microring-based modulator is demonstrated. The quality factor and the extinction ratio of the hybrid microring are 1.7 × 104 and 10 dB, respectively. The linear bidirectional wavelength tuning efficiency is measured as 12.8 pm/V. The measured 3-dB bandwidth is > 20 GHz. High-quality eye diagrams of 20 Gbps non-return-to-zero signal and 20 Gbps four-level pulse amplitude modulation signals are generated experimentally. The proposed platform extends the toolbox of silicon photonics technology, which paves the way for high-speed modulators and phase shifters in optical communication and optical phased array.

Backpropagation-free training of deep physical neural networks

Recent successes in deep learning for vision and natural language processing are attributed to larger models but come with energy consumption and scalability issues. Current training of digital deep learning models primarily relies on backpropagation that is unsuitable for physical implementation. Here, we proposed a simple deep neural network architecture augmented by a physical local learning (PhyLL) algorithm, enabling supervised and unsupervised training of deep physical neural networks, without detailed knowledge of the nonlinear physical layer's properties. We trained diverse wave-based physical neural networks in vowel and image classification experiments, showcasing our approach's universality. Our method shows advantages over other hardware-aware training schemes by improving training speed, enhancing robustness, and reducing power consumption by eliminating the need for system modelling and thus decreasing digital computation.

Complex-valued matrix-vector multiplication system for a large-scale optical FFT

Recent advancements in optical convolutional neural networks (CNNs) and radar signal processing systems have brought an increasing need for the adoption of optical fast Fourier transform (OFFT). Presently, the fast Fourier transform (FFT) is executed using electronic means within prevailing architectures. However, this electronic approach faces limitations in terms of both speed and power consumption. Concurrently, existing OFFT systems struggle to balance the demands of large-scale processing and high precision simultaneously. In response, we introduce a novel, to the best of our knowledge, solution: a complex-valued matrix-vector system harnessed through wavelength selective switches (WSSs) for the realization of a 24-input optical FFT, achieving a high-accuracy level of 5.4 bits. This study capitalizes on the abundant wavelength resources available to present a feasible solution for an optical FFT system with a large N.

Auto-calibrating universal programmable photonic circuits: hardware error-correction and defect resilience

It is recently shown that discrete N × N linear unitary operators can be represented by interlacing N + 1 phase shift layers with a fixed intervening operator such as discrete fractional Fourier transform (DFrFT). Here, we show that introducing perturbations to the intervening operations does not compromise the universality of this architecture. Furthermore, we show that this architecture is resilient to defects in the phase shifters as long as no more than one faulty phase shifter is present in each layer. These properties enable post-fabrication auto-calibration of such universal photonic circuits, effectively compensating for fabrication errors and defects in phase components.

All-analog photoelectronic chip for high-speed vision tasks

Photonic computing enables faster and more energy-efficient processing of vision data1-5. However, experimental superiority of deployable systems remains a challenge because of complicated optical nonlinearities, considerable power consumption of analog-to-digital converters (ADCs) for downstream digital processing and vulnerability to noises and system errors1,6-8. Here we propose an all-analog chip combining electronic and light computing (ACCEL). It has a systemic energy efficiency of 74.8 peta-operations per second per watt and a computing speed of 4.6 peta-operations per second (more than 99% implemented by optics), corresponding to more than three and one order of magnitude higher than state-of-the-art computing processors, respectively. After applying diffractive optical computing as an optical encoder for feature extraction, the light-induced photocurrents are directly used for further calculation in an integrated analog computing chip without the requirement of analog-to-digital converters, leading to a low computing latency of 72 ns for each frame. With joint optimizations of optoelectronic computing and adaptive training, ACCEL achieves competitive classification accuracies of 85.5%, 82.0% and 92.6%, respectively, for Fashion-MNIST, 3-class ImageNet classification and time-lapse video recognition task experimentally, while showing superior system robustness in low-light conditions (0.14 fJ μm-2 each frame). ACCEL can be used across a broad range of applications such as wearable devices, autonomous driving and industrial inspections.

Graphene/silicon heterojunction for reconfigurable phase-relevant activation function in coherent optical neural networks

Optical neural networks (ONNs) herald a new era in information and communication technologies and have implemented various intelligent applications. In an ONN, the activation function (AF) is a crucial component determining the network performances and on-chip AF devices are still in development. Here, we first demonstrate on-chip reconfigurable AF devices with phase activation fulfilled by dual-functional graphene/silicon (Gra/Si) heterojunctions. With optical modulation and detection in one device, time delays are shorter, energy consumption is lower, reconfigurability is higher and the device footprint is smaller than other on-chip AF strategies. The experimental modulation voltage (power) of our Gra/Si heterojunction achieves as low as 1 V (0.5 mW), superior to many pure silicon counterparts. In the photodetection aspect, a high responsivity of over 200 mA/W is realized. Special nonlinear functions generated are fed into a complex-valued ONN to challenge handwritten letters and image recognition tasks, showing improved accuracy and potential of high-efficient, all-component-integration on-chip ONN. Our results offer new insights for on-chip ONN devices and pave the way to high-performance integrated optoelectronic computing circuits.

Demonstration of optically-assisted reconfigurable average of two 20-Gbaud 4-phase-encoded data channels using nonlinear wave mixing

Networks can play a key role in high-speed and reconfigurable arithmetic computing. However, two performance bottlenecks may arise when: (i) relying solely on electronics to handle computation for multiple data channels at high data rates, and (ii) the data streams input to a processing node (PN) are transmitted as phase-encoded signals over an optical network. We experimentally demonstrate the operation of optically-assisted reconfigurable average of two 4-phase-encoded data channels at 10- and 20-Gbaud rates. Our input signals are two streams of 2-bit numbers representing a binary floating-point format, and the operation results in 7-phase-encoded output signals represented by 3-bit numbers. The average operation is achieved in three stages: (1) phase encoding and division-using an optical modulator to encode the data streams; (2) summation-using a highly nonlinear fiber (HNLF); and (3) multicast-using a periodically poled lithium niobate (PPLN) waveguide to multicast back the result into the original signal wavelengths. The experimental results validate the concept, and the measured penalties indicate that: (i) the error vector magnitudes (EVMs) of optical signals increase at each stage and reach ∼18-21% for the final multicast results, and (ii) compared to the inputs, the optical signal-to-noise ratio (OSNR) penalty of output is ∼6.7 dB for the 10-Gbaud rate and ∼6.9 dB for the 20-Gbaud rate at a bit error rate (BER) of 3.8e-3.

Single-pixel image reconstruction using coherent nonlinear optics

We propose and experimentally demonstrate a novel, to the best of our knowledge, hybrid optoelectronic system that utilizes mode-selective frequency upconversion, single-pixel detection, and a deep neural network to achieve the reliable reconstruction of two-dimensional (2D) images from a noise-contaminated database of handwritten digits. Our system is designed to maximize the multi-scale structural similarity index measure (MS-SSIM) and minimize the mean absolute error (MAE) during the training process. Through extensive evaluation, we have observed that the reconstructed images exhibit high-quality results, with a peak signal-to-noise ratio (PSNR) reaching approximately 20 dB and a structural similarity index measure (SSIM) of around 0.85. These impressive metrics demonstrate the effectiveness and fidelity of our image reconstruction technique. The versatility of our approach allows its application in various fields, including Lidar, compressive imaging, volumetric reconstruction, and so on.

Acceleration of Approximate Matrix Multiplications on GPUs

Matrix multiplication is important in various information-processing applications, including the computation of eigenvalues and eigenvectors, and in combinatorial optimization algorithms. Therefore, reducing the computation time of matrix products is essential to speed up scientific and practical calculations. Several approaches have been proposed to speed up this process, including GPUs, fast matrix multiplication libraries, custom hardware, and efficient approximate matrix multiplication (AMM) algorithms. However, research to date has yet to focus on accelerating AMMs for general matrices on GPUs, despite the potential of GPUs to perform fast and accurate matrix product calculations. In this paper, we propose a method for improving Monte Carlo AMMs. We also give an analytical solution for the optimal values of the hyperparameters in the proposed method. The proposed method improves the approximation of the matrix product without increasing the computation time compared to the conventional AMMs. It is also designed to work well with parallel operations on GPUs and can be incorporated into various algorithms. Finally, the proposed method is applied to a power method used for eigenvalue computation. We demonstrate that, on an NVIDIA A100 GPU, the computation time can be halved compared to the conventional power method using cuBLAS.

A Multi-Channel Frequency Router Based on an Optimization Algorithm and Dispersion Engineering

Integrated frequency routers, which can guide light with different frequencies to different output ports, are an important kind of nanophotonic device. However, frequency routers with both a compact size and multiple channels are difficult to realize, which limits the application of these frequency routers in nanophotonics. Here, a kind of bandgap optimization algorithm, which consists of the finite element method and topology optimization, is proposed to design a multi-channel frequency router. Channels supporting photonic edge states with different frequencies are built through the synthetic dimension of translational deformation. Due to the help of the developed optimization algorithms, the number of channels and output ports can be increased up to nine while maintaining ultracompact device size. The device operates within a working band of 0.585-0.665 c/a, corresponding to 1.504-1.709 μm when the lattice constant is set as 1 μm, covering the telecom wavelength of 1.55 μm. The average crosstalk is about -11.49 dB. The average extinction ratio is around 16.18 dB. Because the bus of the device can be regarded as a part of a topological rainbow, the results show that the structure is robust to fabrication errors. This method is general, which can be used for different materials and different frequency ranges. The all-dielectric planar configuration of our router is compact, robust, and easy to integrate, providing a new method for on-chip multi-channel broadband information processing.

Compact and fast-response optical switch based on complex refractive index engineering

The optical switch is a crucial device in integrated photonic circuits. Among the various types of optical switches available, the on-off Mach-Zehnder interferometer is one of the most widely used structures. However, compared with other structures, such as a microring, the large footprint of a Mach-Zehnder interferometer significantly restricts the integration density. In this paper, we propose a compact Mach-Zehnder interferometer based on complex refractive index engineering. By manipulating the complex index of the material in the structure, the lateral size of the device can be compressed down to only 3.25 µm. Moreover, the reducing of the space between heaters and waveguides leads to a fast response of only 1.9 µs. Our work offers a new, to the best of our knowledge, approach of a compact integrated optical switch, and opens a new avenue for application of absorbing materials.

Artificial Intelligence and Advanced Materials

Artificial intelligence (AI) is gaining strength, and materials science can both contribute to and profit from it. In a simultaneous progress race, new materials, systems, and processes can be devised and optimized thanks to machine learning (ML) techniques, and such progress can be turned into innovative computing platforms. Future materials scientists will profit from understanding how ML can boost the conception of advanced materials. This review covers aspects of computation from the fundamentals to directions taken and repercussions produced by computation to account for the origins, procedures, and applications of AI. ML and its methods are reviewed to provide basic knowledge of its implementation and its potential. The materials and systems used to implement AI with electric charges are finding serious competition from other information-carrying and processing agents. The impact these techniques have on the inception of new advanced materials is so deep that a new paradigm is developing where implicit knowledge is being mined to conceive materials and systems for functions instead of finding applications to found materials. How far this trend can be carried is hard to fathom, as exemplified by the power to discover unheard of materials or physical laws buried in data.

Compact optical convolution processing unit based on multimode interference

Convolutional neural networks are an important category of deep learning, currently facing the limitations of electrical frequency and memory access time in massive data processing. Optical computing has been demonstrated to enable significant improvements in terms of processing speeds and energy efficiency. However, most present optical computing schemes are hardly scalable since the number of optical elements typically increases quadratically with the computational matrix size. Here, a compact on-chip optical convolutional processing unit is fabricated on a low-loss silicon nitride platform to demonstrate its capability for large-scale integration. Three 2 × 2 correlated real-valued kernels are made of two multimode interference cells and four phase shifters to perform parallel convolution operations. Although the convolution kernels are interrelated, ten-class classification of handwritten digits from the MNIST database is experimentally demonstrated. The linear scalability of the proposed design with respect to computational size translates into a solid potential for large-scale integration.

Dammann gratings-based truly parallel optical matrix multiplication accelerator

Matrix multiplication (MM) is a fundamental operation in various scientific and engineering computations, as well as in artificial intelligence algorithms. Efficient implementation of MM is crucial for speeding up numerous applications. Photonics presents an opportunity for efficient acceleration of dense matrix computation, owing to its intrinsic advantages, such as huge parallelism, low latency, and low power consumption. However, most optical matrix computing architectures have been limited to realizing single-channel vector-matrix multiplication or using complex configurations to expand the number of channels, which does not fully exploit the parallelism of optics. In this study, we propose a novel, to the best of our knowledge, scheme for the implementation of large-scale two-dimensional optical MM with truly massive parallelism based on a specially designed Dammann grating. We demonstrate a sequence of MMs of 50 pairs of randomly generated 4 × 8 and 8 × 4 matrices in our proof-of-principle experiment. The results indicate that the mean relative error is approximately 0.048, thereby demonstrating optical robustness and high accuracy.

Hardware Error Correction for MZI-Based Matrix Computation

With the rapid development of artificial intelligence, the electronic system has fallen short of providing the needed computation speed. It is believed that silicon-based optoelectronic computation may be a solution, where Mach-Zehnder interferometer (MZI)-based matrix computation is the key due to its advantages of simple implementation and easy integration on a silicon wafer, but one of the concerns is the precision of the MZI method in the actual computation. This paper will identify the main hardware error sources of MZI-based matrix computation, summarize the available hardware error correction methods from the perspective of the entire MZI meshes and a single MZI device, and propose a new architecture that will largely improve the precision of MZI-based matrix computation without increasing the size of the MZI's mesh, which may lead to a fast and accurate optoelectronic computing system.

Heavy tails and pruning in programmable photonic circuits for universal unitaries

Developing hardware for high-dimensional unitary operators plays a vital role in implementing quantum computations and deep learning accelerations. Programmable photonic circuits are singularly promising candidates for universal unitaries owing to intrinsic unitarity, ultrafast tunability and energy efficiency of photonic platforms. Nonetheless, when the scale of a photonic circuit increases, the effects of noise on the fidelity of quantum operators and deep learning weight matrices become more severe. Here we demonstrate a nontrivial stochastic nature of large-scale programmable photonic circuits-heavy-tailed distributions of rotation operators-that enables the development of high-fidelity universal unitaries through designed pruning of superfluous rotations. The power law and the Pareto principle for the conventional architecture of programmable photonic circuits are revealed with the presence of hub phase shifters, allowing for the application of network pruning to the design of photonic hardware. For the Clements design of programmable photonic circuits, we extract a universal architecture for pruning random unitary matrices and prove that "the bad is sometimes better to be removed" to achieve high fidelity and energy efficiency. This result lowers the hurdle for high fidelity in large-scale quantum computing and photonic deep learning accelerators.

Three-dimensional on-chip mode converter

The implementation of transverse mode, polarization, frequency, and other degrees of freedom (d.o.f.s) of photons is an important way to improve the capability of photonic circuits. Here, a three-dimensional (3D) linear polarized (LP) LP₁₁ mode converter was designed and fabricated using a femtosecond laser direct writing (FsLDW) technique. The converter included multi-mode waveguides, symmetric Y splitters, and phase delaying waveguides, which were constructed as different numbers and arrangements of circular cross section waveguides. Finally, the modes (LP_11a and LP_11b) were generated on-chip with a relatively low insertion loss (IL). The mode converter lays a foundation for on-chip high-order mode generation and conversion between different modes, and will play a significant role in mode coding and decoding of 3D photonic circuits.

Low Power Coherent Ising Machine Based on Mechanical Kerr Nonlinearity

Finding a reliable Ising machine for solving nondeterministic polynomial-class problems has attracted great attention in recent years, where an authentic system can be expanded with polynomial-scaled resources to find the ground state Ising Hamiltonian. In this Letter, we propose an extremely low power optomechanical coherent Ising machine based on a new enhanced symmetry breaking mechanism and highly nonlinear mechanical Kerr effect. The mechanical movement of an optomechanical actuator induced by the optical gradient force greatly increases the nonlinearity by a few orders and significantly reduces the power threshold using conventional structures capable of fabrication via photonic integrated circuit platforms. With the simple but strong bifurcation mechanism and remarkably low power requirement, our optomechanical spin model opens a path for chip-scale integration of large-size Ising machine implementations with great stability.

Coherent optical neuron control based on reinforcement learning

Optical neural networks take optical neurons as the cornerstone to achieve complex functions. The coherent optical neuron has become one of the mainstream implementations because it can effectively perform natural and even complex number calculations. However, its state variability and requirement for reliability and effectiveness render traditional control methods no longer applicable. In this Letter, deep reinforcement coherent optical neuron control (DRCON) is proposed, and its effectiveness is experimentally demonstrated. Compared with the standard stochastic gradient descent, the average convergence rate of DRCON is 33% faster, while the effective number of bits increases from less than 2 bits to 5.5 bits. DRCON is a promising first step for large-scale optical neural network control.

Photonic parallel channel estimation of MIMO-OFDM wireless communication systems

Channel estimation is a key technology in MIMO-OFDM wireless communication systems. Increasingly extensive application scenarios and exponentially growing data volumes of MIMO-OFDM systems have imposed greater challenges on the speed, latency, and parallelism of channel estimation based on electronic processors. Here, we propose a photonic parallel channel estimation (PPCE) architecture which features radio-frequency direct processing. Proof-of-concept experiment is carried out to demonstrate the general feasibility of the proposed architecture at different frequency bands (100 MHz, 4 GHz, and 10 GHz). The mean square errors (MSEs) between the experimental channel estimation results and the theoretically simulated ones lie on the order of 10^-3. The bit error rates (BERs) are below the pre-forward error correction (pre-FEC) threshold. Besides, we analyze the performance of PPCE under different signal-to-noise ratios (SNRs), baseband symbol forms, and weight tuning precisions. The proposed PPCE architecture has the potential to achieve high-speed, highly parallel channel estimation in large-scale MIMO-OFDM systems after the photonic-electronic chip integration.

Microcomb-based integrated photonic processing unit

The emergence of parallel convolution-operation technology has substantially powered the complexity and functionality of optical neural networks (ONN) by harnessing the dimension of optical wavelength. However, this advanced architecture faces remarkable challenges in high-level integration and on-chip operation. In this work, convolution based on time-wavelength plane stretching approach is implemented on a microcomb-driven chip-based photonic processing unit (PPU). To support the operation of this processing unit, we develop a dedicated control and operation protocol, leading to a record high weight precision of 9 bits. Moreover, the compact architecture and high data loading speed enable a preeminent photonic-core compute density of over 1 trillion of operations per second per square millimeter (TOPS mm^-2). Two proof-of-concept experiments are demonstrated, including image edge detection and handwritten digit recognition, showing comparable processing capability compared to that of a digital computer. Due to the advanced performance and the great scalability, this parallel photonic processing unit can potentially revolutionize sophisticated artificial intelligence tasks including autonomous driving, video action recognition and image reconstruction.

Prospects and applications of on-chip lasers

Integrated silicon photonics has sparked a significant ramp-up of investment in both academia and industry as a scalable, power-efficient, and eco-friendly solution. At the heart of this platform is the light source, which in itself, has been the focus of research and development extensively. This paper sheds light and conveys our perspective on the current state-of-the-art in different aspects of application-driven on-chip silicon lasers. We tackle this from two perspectives: device-level and system-wide points of view. In the former, the different routes taken in integrating on-chip lasers are explored from different material systems to the chosen integration methodologies. Then, the discussion focus is shifted towards system-wide applications that show great prospects in incorporating photonic integrated circuits (PIC) with on-chip lasers and active devices, namely, optical communications and interconnects, optical phased array-based LiDAR, sensors for chemical and biological analysis, integrated quantum technologies, and finally, optical computing. By leveraging the myriad inherent attractive features of integrated silicon photonics, this paper aims to inspire further development in incorporating PICs with on-chip lasers in, but not limited to, these applications for substantial performance gains, green solutions, and mass production.

Programmable low-power consumption all-optical nonlinear activation functions using a micro-ring resonator with phase-change materials

A programmable hardware implementation of all-optical nonlinear activation functions for different scenarios and applications in all-optical neural networks is essential. We demonstrate a programmable, low-loss all-optical activation function device based on a silicon micro-ring resonator loaded with phase change materials. Four different nonlinear activation functions of Relu, ELU, Softplus and radial basis functions are implemented for incident signal light of the same wavelength. The maximum power consumption required to switch between the four different nonlinear activation functions in calculation is only 1.748 nJ. The simulation of classification of hand-written digit images also shows that they can perform well as alternative nonlinear activation functions. The device we design can serve as nonlinear units in photonic neural networks, while its nonlinear transfer function can be flexibly programmed to optimize the performance of different neuromorphic tasks.

Large-scale coherent Ising machine based on optoelectronic parametric oscillator

Ising machines based on analog systems have the potential to accelerate the solution of ubiquitous combinatorial optimization problems. Although some artificial spins to support large-scale Ising machines have been reported, e.g., superconducting qubits in quantum annealers and short optical pulses in coherent Ising machines, the spin stability is fragile due to the ultra-low equivalent temperature or optical phase sensitivity. In this paper, we propose to use short microwave pulses generated from an optoelectronic parametric oscillator as the spins to implement a large-scale Ising machine with high stability. The proposed machine supports 25,600 spins and can operate continuously and stably for hours. Moreover, the proposed Ising machine is highly compatible with high-speed electronic devices for programmability, paving a low-cost, accurate, and easy-to-implement way toward solving real-world optimization problems.

Reconfigurable matrix multiplier with on-site reinforcement learning

Matrix multiplication is a fundamental building block for modern information processing and artificial intelligence algorithms. Photonics-based matrix multipliers have recently attracted much attention due to their advantages of low energy and ultrafast speed. Conventionally, achieving matrix multiplication relies on bulky Fourier optical components, and the functionalities are unchangeable once the design is determined. Furthermore, the bottom-up design strategy cannot easily be generalized into concrete and practical guidelines. Here, we introduce a reconfigurable matrix multiplier driven by on-site reinforcement learning. The constituent transmissive metasurfaces incorporating varactor diodes serve as tunable dielectrics based on the effective medium theory. We validate the viability of tunable dielectrics and demonstrate the performance of matrix customization. This work represents a new avenue in realizing reconfigurable photonic matrix multipliers for on-site applications.

Parallel optical coherent dot-product architecture for large-scale matrix multiplication with compatibility for diverse phase shifters

Photonics physically promises high-speed and low-consumption computing of matrix multiplication. Nevertheless, conventional approaches are challenging to achieve large throughput, high precision, low power consumption, and high density simultaneously in a single architecture, because the integration scale of conventional approaches is strongly limited by the insertion loss of cascaded optical phase shifters. Here, we present a parallel optical coherent dot-product (P-OCD) architecture, which deploys phase shifters in a fully parallel way. The insertion loss of phase shifters does not accumulate at large integration scale. The architecture decouples the integration scale and phase shifter insertion loss, making it possible to achieve superior throughput, precision, energy-efficiency, and compactness simultaneously in a single architecture. As the architecture is compatible with diverse integration technologies, high-performance computing can be realized with various off-the-shelf photonic phase shifters. Simulations show that compared with conventional architectures, the parallel architecture can achieve near 100× higher throughput and near 10× higher energy efficiency especially with lossy phase shifters. The parallel architecture is expected to perform its unique advantage in computing-intense applications including AI, communications, and autonomous driving.

Two-layer integrated photonic architectures with multiport photodetectors for high-fidelity and energy-efficient matrix multiplications

Photonic integrated circuits (PICs) are emerging as a promising tool for accelerating matrix multiplications in deep learning. Previous PIC architectures, primarily focusing on the matrix-vector multiplication (MVM), have large hardware errors that increase with the device scale. In this work, we propose a novel PIC architecture for MVM, which features an intrinsically small hardware error that does not increase with the device scale. Moreover, we further develop this concept and propose a PIC architecture for the general matrix-matrix multiplication (GEMM), which allows the GEMM to be directly performed on a photonic chip with a high energy efficiency unattainable by parallel or sequential MVMs. This work provides a promising approach to realize a high fidelity and high energy efficiency optical computing platform.

Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible

Replacing electrons with photons is a compelling route toward high-speed, massively parallel, and low-power artificial intelligence computing. Recently, diffractive networks composed of phase surfaces were trained to perform machine learning tasks through linear optical transformations. However, the existing architectures often comprise bulky components and, most critically, they cannot mimic the human brain for multitasking. Here, we demonstrate a multi-skilled diffractive neural network based on a metasurface device, which can perform on-chip multi-channel sensing and multitasking in the visible. The polarization multiplexing scheme of the subwavelength nanostructures is applied to construct a multi-channel classifier framework for simultaneous recognition of digital and fashionable items. The areal density of the artificial neurons can reach up to 6.25 × 10⁶ mm^-2 multiplied by the number of channels. The metasurface is integrated with the mature complementary metal-oxide semiconductor imaging sensor, providing a chip-scale architecture to process information directly at physical layers for energy-efficient and ultra-fast image processing in machine vision, autonomous driving, and precision medicine.

Polarization multiplexed diffractive computing: all-optical implementation of a group of linear transformations through a polarization-encoded diffractive network

Research on optical computing has recently attracted significant attention due to the transformative advances in machine learning. Among different approaches, diffractive optical networks composed of spatially-engineered transmissive surfaces have been demonstrated for all-optical statistical inference and performing arbitrary linear transformations using passive, free-space optical layers. Here, we introduce a polarization-multiplexed diffractive processor to all-optically perform multiple, arbitrarily-selected linear transformations through a single diffractive network trained using deep learning. In this framework, an array of pre-selected linear polarizers is positioned between trainable transmissive diffractive materials that are isotropic, and different target linear transformations (complex-valued) are uniquely assigned to different combinations of input/output polarization states. The transmission layers of this polarization-multiplexed diffractive network are trained and optimized via deep learning and error-backpropagation by using thousands of examples of the input/output fields corresponding to each one of the complex-valued linear transformations assigned to different input/output polarization combinations. Our results and analysis reveal that a single diffractive network can successfully approximate and all-optically implement a group of arbitrarily-selected target transformations with a negligible error when the number of trainable diffractive features/neurons (N) approaches [Formula: see text], where N_i and N_o represent the number of pixels at the input and output fields-of-view, respectively, and N_p refers to the number of unique linear transformations assigned to different input/output polarization combinations. This polarization-multiplexed all-optical diffractive processor can find various applications in optical computing and polarization-based machine vision tasks.