All-analog photoelectronic chip for high-speed vision tasks

Microfluidics for morpholomics and spatial omics applications

Creative designs, precise fluidic manipulation, and automation have supported the development of microfluidics for single-cell applications. Together with the advancements in detection technologies and artificial intelligence (AI), microfluidic-assisted platforms have been increasingly used for new modalities of single-cell investigations and in spatial omics applications. This review explores the use of microfluidic technologies for morpholomics and spatial omics with a focus on single-cell and tissue characterization. We emphasize how various fluid dynamic principles and unique design integrations enable highly precise fluid manipulation, enhancing sample handling in morpholomics. Additionally, we examine the use of microfluidics-assisted spatial barcoding with micrometer resolutions for the spatial profiling of tissue specimens. Finally, we discuss how microfluidics can serve as a bridge for integrating multiple unique fields in omics research and outline key challenges that these technologies may face in practical translation.

Critical band-to-band-tunnelling based optoelectronic memory

Neuromorphic vision hardware, embedded with multiple functions, has recently emerged as a potent platform for machine vision. To realize memory in sensor functions, reconfigurable and non-volatile manipulation of photocarriers is highly desirable. However, previous technologies bear mechanism challenges, such as the ambiguous optoelectronic memory mechanism and high potential barrier, resulting in a limited response speed and a high operating voltage. Here, for the first time, we propose a critical band-to-band tunnelling (BTBT) based device that combines sensing, integration and memory functions. The nearly infinitesimal barrier facilitates the tunnelling process, resulting in a broadband application range (940 nm). Furthermore, the observation of dual negative differential resistance (NDR) points confirms that the critical BTBT of photocarriers contributes to the sub-microsecond photomemory speed. Since the photomemory speed, with no motion blur, is important for motion detection, the critical BTBT memory is expected to enable moving target tracking and recognition, underscoring its superiority in intelligent perception.

Reversible Carrier Modulation in InP Nanolasers by Ionic Liquid Gating with Low Energy Consumption

Nanoscale light sources are demanded vigorously due to rapid development in photonic integrated circuits (PICs). III-V semiconductor nanowire (NW) lasers have manifested themselves as indispensable components in this field, associated with their extremely compact footprint and ultra-high optical gain within the 1D cavity. In this study, the carrier concentrations of indium phosphide (InP) NWs are actively controlled to modify their emissive properties at room temperature. The InP NW lasers can achieve repetitive switching between photoluminescence (PL) and lasing with an extinction ratio of 22-fold by applying a gate voltage of 3 V using ionic liquid (IL) as a dielectric layer. IL brings forth ultra-high capacitance due to the nanometer-wide electric double layer (EDL) between interfaces, mapping out gating efficiency of ≈100-fold compared to the conventional bottom gate configurations. This IL-embedded nanolaser device can be a promising platform for the advanced integrated nanophotonic system.

Femtojoule optical nonlinearity for deep learning with incoherent illumination

Optical neural networks (ONNs) are a promising computational alternative for deep learning due to their inherent massive parallelism for linear operations. However, the development of energy-efficient and highly parallel optical nonlinearities, a critical component in ONNs, remains an outstanding challenge. Here, we introduce a nonlinear optical microdevice array (NOMA) compatible with incoherent illumination by integrating the liquid crystal cell with silicon photodiodes at the single-pixel level. We fabricate NOMA with more than half a million pixels, each functioning as an optical analog of the rectified linear unit at ultralow switching energy down to 100 femtojoules per pixel. With NOMA, we demonstrate an optical multilayer neural network. Our work holds promise for large-scale and low-power deep ONNs, computer vision, and real-time optical image processing.

Self-Organized Protonic Conductive Nanochannel Arrays for Ultra-High-Density Data Storage

While the highest-performing memristors currently available offer superior storage density and energy efficiency, their large-scale integration is hindered by the random distribution of filaments and nonuniform resistive switching in memory cells. Here, we demonstrate the self-organized synthesis of a type of two-dimensional protonic coordination polymers with high crystallinity and porosity. Hydrogen-bond networks containing proton carriers along its nanochannels enable uniform resistive switching down to the subnanoscale range. Leveraging such nanochannel arrays, we achieve logic operations of graphical gate circuits with negligible leakage and sneak path currents over areas ranging from 0.5 μm × 0.5 μm to 20 nm × 20 nm, providing the smallest building blocks to date for large-scale integration. The nonvolatile resistive switching exhibits high mobility (∼0.309 cm2 V-1 s-1), a large on/off ratio (∼103), and ultrahigh-density data storage (∼645 Tbit/in2), even within a trilayer (∼4.01 nm). An ultrahigh-precision artificial retina with integrated convolutional neural network calculations is demonstrated, enabling facial and color recognition capabilities.

All-optical perception based on partially coherent optical neural networks

In the field of image processing, optical neural networks offer advantages such as high speed, high throughput, and low energy consumption. However, most existing coherent optical neural networks (CONN) rely on coherent light sources to establish transmission models. The use of laser inputs and electro-optic modulation devices at the front end of these neural networks diminishes their computational capability and energy efficiency, thereby limiting their practical applications in object detection tasks. This paper proposes a partially coherent optical neural network (PCONN) transmission model based on mutual intensity modulation. This model does not depend on coherent light source inputs or active electro-optic modulation devices, allowing it to directly compute and infer using natural light after simple filtering, thus achieving full optical perception from light signal acquisition to computation and inference. Simulation results indicate that the model achieves a highest classification accuracy of 96.80% and 86.77% on the MNIST and Fashion-MNIST datasets, respectively. In a binary classification simulation test based on the ISDD segmentation dataset, the model attained an accuracy of 94.69%. It is estimated that this system's computational inference speed for object detection tasks is 100 times faster than that of traditional CONN, with energy efficiency approximately 50 times greater. In summary, our proposed PCONN model addresses the limitations of conventional optical neural networks in coherent light environments and is anticipated to find applications in practical object detection scenarios.

High-integrated photonic tensor core utilizing high-dimensional lightwave and microwave multidomain multiplexing

The burgeoning volume of parameters in artificial neural network models has posed substantial challenges to conventional tensor computing hardware. Benefiting from the available optical multidimensional information entropy, optical intelligent computing is used as an alternative solution to address the emerging challenges of electrical computing. These limitations, in terms of device size and photonic integration scale, have hindered the performance of optical chips. Herein, an ultrahigh computing density optical tensor processing unit (OTPU), which is grounded in an individual microring resonator (MRR), is introduced to respond to these challenges. Through the independent tuning of multiwavelength lasers, the operational capabilities of an MRR are orchestrated, culminating in the formation of an optical tensor core. This design facilitates the execution of tensor convolution operations via the lightwave and microwave multidomain hybrid multiplexing in terms of the time, wavelength, and frequency of microwaves. The experimental results for the MRR-based OTPU show an extraordinary computing density of 34.04 TOPS/mm2. Additionally, the achieved accuracy rate in recognizing MNIST handwritten digits was 96.41%. These outcomes signify a significant advancement toward the realization of high-performance optical tensor processing chips.

TOPS-speed complex-valued convolutional accelerator for feature extraction and inference

Complex-valued neural networks process both amplitude and phase information, in contrast to conventional artificial neural networks, achieving additive capabilities in recognizing phase-sensitive data inherent in wave-related phenomena. The ever-increasing data capacity and network scale place substantial demands on underlying computing hardware. In parallel with the successes and extensive efforts made in electronics, optical neuromorphic hardware is promising to achieve ultra-high computing performances due to its inherent analog architecture and wide bandwidth. Here, we report a complex-valued optical convolution accelerator operating at over 2 Tera operations per second (TOPS). With appropriately designed phasors we demonstrate its performance in the recognition of synthetic aperture radar (SAR) images captured by the Sentinel-1 satellite, which are inherently complex-valued and more intricate than what optical neural networks have previously processed. Experimental tests with 500 images yield an 83.8% accuracy, close to in-silico results. This approach facilitates feature extraction of phase-sensitive information, and represents a pivotal advance in artificial intelligence towards real-time, high-dimensional data analysis of complex and dynamic environments.

Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light

Optical neural networks are considered next-generation physical implementations of artificial neural networks, but their capabilities are limited by on-chip integration scale and requirement for coherent light sources. This study proposes a spectral convolutional neural network (SCNN) with matter meta-imaging. The optical convolutional layer is implemented by integrating very large-scale and pixel-aligned spectral filters on CMOS image sensor. It facilitates highly parallel spectral vector-inner products of incident incoherent natural light i.e., the direct information carrier, which empowers in-sensor optical analog computing at extremely high energy efficiency. To the best of our knowledge, this is the first integrated optical computing utilizing natural light. We employ the same SCNN chip for completely different real-world complex tasks and achieve accuracies of over 96% for pathological diagnosis and almost 100% for face anti-spoofing at video rates. These results indicate a feasible and scalable in-sensor edge computing chip of natural light for various portable terminals.

Photonics for Neuromorphic Computing: Fundamentals, Devices, and Opportunities

In the dynamic landscape of Artificial Intelligence (AI), two notable phenomena are becoming predominant: the exponential growth of large AI model sizes and the explosion of massive amount of data. Meanwhile, scientific research such as quantum computing and protein synthesis increasingly demand higher computing capacities. As the Moore's Law approaches its terminus, there is an urgent need for alternative computing paradigms that satisfy this growing computing demand and break through the barrier of the von Neumann model. Neuromorphic computing, inspired by the mechanism and functionality of human brains, uses physical artificial neurons to do computations and is drawing widespread attention. This review studies the expansion of optoelectronic devices on photonic integration platforms that has led to significant growth in photonic computing, where photonic integrated circuits (PICs) have enabled ultrafast artificial neural networks (ANN) with sub-nanosecond latencies, low heat dissipation, and high parallelism. In particular, various technologies and devices employed in neuromorphic photonic AI accelerators, spanning from traditional optics to PCSEL lasers are examined. Lastly, it is recognized that existing neuromorphic technologies encounter obstacles in meeting the peta-level computing speed and energy efficiency threshold, and potential approaches in new devices, fabrication, materials, and integration to drive innovation are also explored. As the current challenges and barriers in cost, scalability, footprint, and computing capacity are resolved one-by-one, photonic neuromorphic systems are bound to co-exist with, if not replace, conventional electronic computers and transform the landscape of AI and scientific computing in the foreseeable future.

Ultrafast silicon photonic reservoir computing engine delivering over 200 TOPS

Reservoir computing (RC) is a powerful machine learning algorithm for information processing. Despite numerous optical implementations, its speed and scalability remain limited by the need to establish recurrent connections and achieve efficient optical nonlinearities. This work proposes a streamlined photonic RC design based on a new paradigm, called next-generation RC, which overcomes these limitations. Our design leads to a compact silicon photonic computing engine with an experimentally demonstrated processing speed of over 60 GHz. Experimental results demonstrate state-of-the-art performance in prediction, emulation, and classification tasks across various machine learning applications. Compared to traditional RC systems, our silicon photonic RC engine offers several key advantages, including no speed limitations, a compact footprint, and a high tolerance to fabrication errors. This work lays the foundation for ultrafast on-chip photonic RC, representing significant progress toward developing next-generation high-speed photonic computing and signal processing.

Unveiling van Hove Singularity-Boosted Photothermoelectric Response for Wearable Human-Radiation Detection

Van Hove singularity (vHs), the singularity point of density of states (DOS) in crystalline solids, is a research hotspot in emerging phenomena such as light-matter interaction, superconducting, and quantum anomalous Hall effect. Although the significance of vHs in photothermoelectric (PTE) effect has been recognized, its integral role in electron excitation and thermoelectric effect is still unclear, particularly in the mid-infrared band that suffers from Pauli blockade in semimetals. Here, we unveil the Fermi-level-modulated PTE behavior in the vicinity of vHs in carbon nanotubes, employing ionic-liquid gating. The concurrent enhancement of optical absorption and thermoelectric effect effectively improves the overall photoresponse by tens of folds at the vHs point. Generally applicable to strongly correlated systems such as metallic 1D nanomaterials and 2D Moiré systems, a quantitative correlation between PTE photodetectivity and electronic DOS is derived in the vicinity of the vHs point. Finally, chemically doped PTE mid-infrared detectors with graded doping levels are demonstrated to exhibit human-radiation sensitivity, high flexibility, and high transparency, paving the way for wearable sensor networks in healthcare systems and the Internet of Things.

Photonic diffractive generators through sampling noises from scattering media

Photonic computing, with potentials of high parallelism, low latency and high energy efficiency, have gained progressive interest at the forefront of neural network (NN) accelerators. However, most existing photonic computing accelerators concentrate on discriminative NNs. Large-scale generative photonic computing machines remain largely unexplored, partly due to poor data accessibility, accuracy and hardware feasibility. Here, we harness random light scattering in disordered media as a native noise source and leverage large-scale diffractive optical computing to generate images from above noise, thereby achieving hardware consistency by solely pursuing the spatial parallelism of light. To realize experimental data accessibility, we design two encoding strategies between images and optical noise latent space that effectively solves the training problem. Furthermore, we utilize advanced photonic NN architectures including cascaded and parallel configurations of diffraction layers to enhance the image generation performance. Our results show that the photonic generator is capable of producing clear and meaningful synthesized images across several standard public datasets. As a photonic generative machine, this work makes an important contribution to photonic computing and paves the way for more sophisticated applications such as real world data augmentation and multi modal generation.

Multimodal In-Sensor Computing System Using Integrated Silicon Photonic Convolutional Processor

Photonic integrated circuits offer miniaturized solutions for multimodal spectroscopic sensory systems by leveraging the simultaneous interaction of light with temperature, chemicals, and biomolecules, among others. The multimodal spectroscopic sensory data is complex and has huge data volume with high redundancy, thus requiring high communication bandwidth associated with high communication power consumption to transfer the sensory data. To circumvent this high communication cost, the photonic sensor and processor are brought into intimacy and propose a photonic multimodal in-sensor computing system using an integrated silicon photonic convolutional processor. A microring resonator crossbar array is used as the photonic processor to implement convolutional operation with 5-bit accuracy, validated through image edge detection tasks. Further integrating the processor with a photonic spectroscopic sensor, the in situ processing of multimodal spectroscopic sensory data is demonstrated, achieving the classification of protein species of different types and concentrations at various temperatures. A classification accuracy of 97.58% across 45 different classes is achieved. The multimodal in-sensor computing system demonstrates the feasibility of integrating photonic processors and photonic sensors to enhance the data processing capability of photonic devices at the edge.

Metasurface-Based Image Classification Using Diffractive Deep Neural Network

The computer-assisted inverse design of photonic computing, especially by leveraging artificial intelligence algorithms, offers great convenience to accelerate the speed of development and improve calculation accuracy. However, traditional thickness-based modulation methods are hindered by large volume and difficult fabrication process, making it hard to meet the data-driven requirements of flexible light modulation. Here, we propose a diffractive deep neural network (D2NN) framework based on a three-layer all-dielectric phased transmitarray as hidden layers, which can perform the classification of handwritten digits. By tailoring the radius of a silicon nanodisk of a meta-atom, the metasurface can realize the phase profile calculated by D2NN and maintain a relative high transmittance of 0.9 at a wavelength of 600 nm. The designed image classifier consists of three layers of phase-only metasurfaces, each of which contains 1024 units, mimicking a fully connected neural network through the diffraction of light fields. The classification task of handwriting digits from the '0' to '5' dataset is verified, with an accuracy of over 90% on the blind test dataset, as well as demonstrated by the full-wave simulation. Furthermore, the performance of the more complex animal image classification task is also validated by increasing the number of neurons to enhance the connectivity of the neural network. This study may provide a possible solution for practical applications such as biomedical detection, image processing, and machine vision based on all-optical computing.

Spatially varying nanophotonic neural networks

The explosive growth in computation and energy cost of artificial intelligence has spurred interest in alternative computing modalities to conventional electronic processors. Photonic processors, which use photons instead of electrons, promise optical neural networks with ultralow latency and power consumption. However, existing optical neural networks, limited by their designs, have not achieved the recognition accuracy of modern electronic neural networks. In this work, we bridge this gap by embedding parallelized optical computation into flat camera optics that perform neural network computations during capture, before recording on the sensor. We leverage large kernels and propose a spatially varying convolutional network learned through a low-dimensional reparameterization. We instantiate this network inside the camera lens with a nanophotonic array with angle-dependent responses. Combined with a lightweight electronic back-end of about 2K parameters, our reconfigurable nanophotonic neural network achieves 72.76% accuracy on CIFAR-10, surpassing AlexNet (72.64%), and advancing optical neural networks into the deep learning era.

An Active-Matrix Synaptic Phototransistor Array for In-Sensor Spectral Processing

The human retina perceives and preprocesses the spectral information of incident light, enabling fast image recognition and efficient chromatic adaptation. In comparison, it is reluctant to implement parallel spectral preprocessing and temporal information fusion in current complementary metal-oxide-semiconductor (CMOS) image sensors, requiring intricate circuitry, frequent data transmission, and color filters. Herein, an active-matrix synaptic phototransistor array (AMSPA) is developed based on organic/inorganic semiconductor heterostructures. The AMSPA provides wavelength-dependent, bidirectional photoresponses, enabling dynamic imaging and in-sensor spectral preprocessing functions. Specifically, near-infrared light induces inhibitory photoresponse while UV light results in exhibitory photoresponse. With rational structural design of the organic/inorganic hybrid heterostructures, the current dynamic range of phototransistor is improved to over 90 dB. Finally, a 32 × 64 AMSPA (128 pixels per inch) is demonstrated with one-switch-transistor and one-synaptic phototransistor (1-T-1-PT) structure, achieving spatial chromatic enhancement and temporal trajectory imaging. These results reveal the feasibility of AMSPA for constructing artificial vision systems.

Optical neural networks: progress and challenges

Artificial intelligence has prevailed in all trades and professions due to the assistance of big data resources, advanced algorithms, and high-performance electronic hardware. However, conventional computing hardware is inefficient at implementing complex tasks, in large part because the memory and processor in its computing architecture are separated, performing insufficiently in computing speed and energy consumption. In recent years, optical neural networks (ONNs) have made a range of research progress in optical computing due to advantages such as sub-nanosecond latency, low heat dissipation, and high parallelism. ONNs are in prospect to provide support regarding computing speed and energy consumption for the further development of artificial intelligence with a novel computing paradigm. Herein, we first introduce the design method and principle of ONNs based on various optical elements. Then, we successively review the non-integrated ONNs consisting of volume optical components and the integrated ONNs composed of on-chip components. Finally, we summarize and discuss the computational density, nonlinearity, scalability, and practical applications of ONNs, and comment on the challenges and perspectives of the ONNs in the future development trends.

Harnessing the capabilities of VCSELs: unlocking the potential for advanced integrated photonic devices and systems

Vertical cavity surface emitting lasers (VCSELs) have emerged as a versatile and promising platform for developing advanced integrated photonic devices and systems due to their low power consumption, high modulation bandwidth, small footprint, excellent scalability, and compatibility with monolithic integration. By combining these unique capabilities of VCSELs with the functionalities offered by micro/nano optical structures (e.g. metasurfaces), it enables various versatile energy-efficient integrated photonic devices and systems with compact size, enhanced performance, and improved reliability and functionality. This review provides a comprehensive overview of the state-of-the-art versatile integrated photonic devices/systems based on VCSELs, including photonic neural networks, vortex beam emitters, holographic devices, beam deflectors, atomic sensors, and biosensors. By leveraging the capabilities of VCSELs, these integrated photonic devices/systems open up new opportunities in various fields, including artificial intelligence, large-capacity optical communication, imaging, biosensing, and so on. Through this comprehensive review, we aim to provide a detailed understanding of the pivotal role played by VCSELs in integrated photonics and highlight their significance in advancing the field towards efficient, compact, and versatile photonic solutions.

Terahertz spoof plasmonic neural network for diffractive information recognition and processing

All-optical diffractive neural networks, as analog artificial intelligence accelerators, leverage parallelism and analog computation for complex data processing. However, their low space transmission efficiency or large spatial dimensions hinder miniaturization and broader application. Here, we propose a terahertz spoof plasmonic neural network on a planar diffractive platform for direct multi-target recognition. Our approach employs a spoof surface plasmon polariton coupler array to construct a diffractive network layer, resulting in a compact, efficient, and easily integrable architecture. We designed three schemes: basis vector classification, multi-user recognition, and MNIST handwritten digit classification. Experimental results reveal that the terahertz spoof plasmonic neural network successfully classifies basis vectors, recognizes multi-user orientation information, and directly processes handwritten digits using a designed input framework comprising a metal grating array, transmitters, and receivers. This work broadens the application of terahertz plasmonic metamaterials, paving the way for terahertz on-chip integration, intelligent communication, and advanced computing systems.

Femtosecond laser direct nanolithography of perovskite hydration for temporally programmable holograms

Modern nanofabrication technologies have propelled significant advancement of high-resolution and optically thin holograms. However, it remains a long-standing challenge to tune the complex hologram patterns at the nanoscale for temporal light field control. Here, we report femtosecond laser direct lithography of perovskites with nanoscale feature size and pixel-level temporal dynamics control for temporally programmable holograms. Specifically, under tightly focused laser irradiation, the organic molecules of layered perovskites (PEA)2PbI4 can be exfoliated with nanometric thickness precision and subwavelength lateral size. This creates inorganic lead halide capping nanostructures that retard perovskite hydration, enabling tunable hydration time constant. Leveraging advanced inverse design methods, temporal holograms in which multiple independent images are multiplexed with low cross talk are demonstrated. Furthermore, cascaded holograms are constructed to form temporally holographic neural networks with programmable optical inference functionality. Our work opens up new opportunities for tunable photonic devices with broad impacts on holography display and storage, high-dimensional optical encryption and artificial intelligence.

Pre-sensor computing with compact multilayer optical neural network

Moving computation units closer to sensors is becoming a promising approach to addressing bottlenecks in computing speed, power consumption, and data storage. Pre-sensor computing with optical neural networks (ONNs) allows extensive processing. However, the lack of nonlinear activation and dependence on laser input limits the computational capacity, practicality, and scalability. A compact and passive multilayer ONN (MONN) is proposed, which has two convolution layers and an inserted nonlinear layer, performing pre-sensor computations with designed passive masks and a quantum dot film for incoherent light. MONN has an optical length as short as 5 millimeters, two orders of magnitude smaller than state-of-the-art lens-based ONNs. MONN outperforms linear single-layer ONN across various vision tasks, off-loading up to 95% of computationally expensive operations into optics from electronics. Motivated by MONN, a paradigm is emerging for mobile vision, fulfilling the demands for practicality, miniaturization, and low power consumption.

Image authentication method based on Fourier zero-frequency replacement and single-pixel self-calibration imaging by diffractive deep neural network

The diffractive deep neural network is a novel network model that applies the principles of diffraction to neural networks, enabling machine learning tasks to be performed through optical principles. In this paper, a fully optical authentication model is developed using the diffractive deep neural network. The model utilizes terahertz light for propagation and combines it with a self-calibration single-pixel imaging model to construct a comprehensive optical authentication system with faster authentication speed. The proposed system filters the authentication images, establishes an optical connection with the Fourier zero-frequency response of the illumination pattern, and introduces the signal-to-noise ratio as a criterion for batch image authentication. Computer simulations demonstrate the fast speed and strong automation performance of the proposed optical authentication system, suggesting broad prospects for the combined application of diffractive deep neural networks and optical systems.

Multidimensional vision sensors for information processing

The visual scene in the physical world integrates multidimensional information (spatial, temporal, polarization, spectrum and so on) and typically shows unstructured characteristics. Conventional image sensors cannot process this multidimensional vision data, creating a need for vision sensors that can efficiently extract features from substantial multidimensional vision data. Vision sensors are able to transform the unstructured visual scene into featured information without relying on sophisticated algorithms and complex hardware. The response characteristics of sensors can be abstracted into operators with specific functionalities, allowing for the efficient processing of perceptual information. In this Review, we delve into the hardware implementation of multidimensional vision sensors, exploring their working mechanisms and design principles. We exemplify multidimensional vision sensors built on emerging devices and silicon-based system integration. We further provide benchmarking metrics for multidimensional vision sensors and conclude with the principle of device-system co-design and co-optimization.

Memory-less scattering imaging with ultrafast convolutional optical neural networks

The optical memory effect in complex scattering media including turbid tissue and speckle layers has been a critical foundation for macroscopic and microscopic imaging methods. However, image reconstruction from strong scattering media without the optical memory effect has not been achieved. Here, we demonstrate image reconstruction through scattering layers where no optical memory effect exists, by developing a multistage convolutional optical neural network (ONN) integrated with multiple parallel kernels operating at the speed of light. Training this Fourier optics-based, parallel, one-step convolutional ONN with the strong scattering process for direct feature extraction, we achieve memory-less image reconstruction with a field of view enlarged by a factor up to 271. This device is dynamically reconfigurable for ultrafast multitask image reconstruction with a computational power of 1.57 peta-operations per second (POPS). Our achievement establishes an ultrafast and high energy-efficient optical machine learning platform for graphic processing.

Integrated photonic encoder for low power and high-speed image processing

Modern lens designs are capable of resolving greater than 10 gigapixels, while advances in camera frame-rate and hyperspectral imaging have made data acquisition rates of Terapixel/second a real possibility. The main bottlenecks preventing such high data-rate systems are power consumption and data storage. In this work, we show that analog photonic encoders could address this challenge, enabling high-speed image compression using orders-of-magnitude lower power than digital electronics. Our approach relies on a silicon-photonics front-end to compress raw image data, foregoing energy-intensive image conditioning and reducing data storage requirements. The compression scheme uses a passive disordered photonic structure to perform kernel-type random projections of the raw image data with minimal power consumption and low latency. A back-end neural network can then reconstruct the original images with structural similarity exceeding 90%. This scheme has the potential to process data streams exceeding Terapixel/second using less than 100 fJ/pixel, providing a path to ultra-high-resolution data and image acquisition systems.

Experimental reservoir computing with diffractively coupled VCSELs

We present experiments on reservoir computing (RC) using a network of vertical-cavity surface-emitting lasers (VCSELs) that we diffractively couple via an external cavity. Our optical reservoir computer consists of 24 physical VCSEL nodes. We evaluate the system's memory and solve the 2-bit XOR task and the 3-bit header recognition (HR) task with bit error ratios (BERs) below 1% and the 2-bit digital-to-analog conversion (DAC) task with a root mean square error (RMSE) of 0.067.

Large-scale photonic chiplet Taichi empowers 160-TOPS/W artificial general intelligence

The pursuit of artificial general intelligence (AGI) continuously demands higher computing performance. Despite the superior processing speed and efficiency of integrated photonic circuits, their capacity and scalability are restricted by unavoidable errors, such that only simple tasks and shallow models are realized. To support modern AGIs, we designed Taichi-large-scale photonic chiplets based on an integrated diffractive-interference hybrid design and a general distributed computing architecture that has millions-of-neurons capability with 160-tera-operations per second per watt (TOPS/W) energy efficiency. Taichi experimentally achieved on-chip 1000-category-level classification (testing at 91.89% accuracy in the 1623-category Omniglot dataset) and high-fidelity artificial intelligence-generated content with up to two orders of magnitude of improvement in efficiency. Taichi paves the way for large-scale photonic computing and advanced tasks, further exploiting the flexibility and potential of photonics for modern AGI.

Cross-layer transmission realized by light-emitting memristor for constructing ultra-deep neural network with transfer learning ability

Deep neural networks have revolutionized several domains, including autonomous driving, cancer detection, and drug design, and are the foundation for massive artificial intelligence models. However, hardware neural network reports still mainly focus on shallow networks (2 to 5 layers). Implementing deep neural networks in hardware is challenging due to the layer-by-layer structure, resulting in long training times, signal interference, and low accuracy due to gradient explosion/vanishing. Here, we utilize negative ultraviolet photoconductive light-emitting memristors with intrinsic parallelism and hardware-software co-design to achieve electrical information's optical cross-layer transmission. We propose a hybrid ultra-deep photoelectric neural network and an ultra-deep super-resolution reconstruction neural network using light-emitting memristors and cross-layer block, expanding the networks to 54 and 135 layers, respectively. Further, two networks enable transfer learning, approaching or surpassing software-designed networks in multi-dataset recognition and high-resolution restoration tasks. These proposed strategies show great potential for high-precision multifunctional hardware neural networks and edge artificial intelligence.

Parallel edge extraction operators on chip speed up photonic convolutional neural networks

We experimentally establish a 3 × 3 cross-shaped micro-ring resonator (MRR) array-based photonic multiplexing architecture relying on silicon photonics to achieve parallel edge extraction operations in images for photonic convolution neural networks. The main mathematical operations involved are convolution. Precisely, a faster convolutional calculation speed of up to four times is achieved by extracting four feature maps simultaneously with the same photonic hardware's structure and power consumption, where a maximum computility of 0.742 TOPS at an energy cost of 48.6 mW and a convolution accuracy of 95.1% is achieved in an MRR array chip. In particular, our experimental results reveal that this system using parallel edge extraction operators instead of universal operators can improve the imaging recognition accuracy for CIFAR-10 dataset by 6.2% within the same computing time, reaching a maximum of 78.7%. This work presents high scalability and efficiency of parallel edge extraction chips, furnishing a novel, to the best of our knowledge, approach to boost photonic computing speed.

Optical computing metasurfaces: applications and advances

Integrated photonic devices and artificial intelligence have presented a significant opportunity for the advancement of optical computing in practical applications. Optical computing technology is a unique computing system based on optical devices and computing functions, which significantly differs from the traditional electronic computing technology. On the other hand, optical computing technology offers the advantages such as fast speed, low energy consumption, and high parallelism. Yet there are still challenges such as device integration and portability. In the burgeoning development of micro-nano optics technology, especially the deeply ingrained concept of metasurface technique, it provides an advanced platform for optical computing applications, including edge detection, image or motion recognition, logic computation, and on-chip optical computing. With the aim of providing a comprehensive introduction and perspective for optical computing metasurface applications, we review the recent research advances of optical computing, from nanostructure and computing methods to practical applications. In this work, we review the challenges and analysis of optical computing metasurfaces in engineering field and look forward to the future development trends of optical computing.

Memristor-Based Artificial Chips

Memristors, promising nanoelectronic devices with in-memory resistive switching behavior that is assembled with a physically integrated core processing unit (CPU) and memory unit and even possesses highly possible multistate electrical behavior, could avoid the von Neumann bottleneck of traditional computing devices and show a highly efficient ability of parallel computation and high information storage. These advantages position them as potential candidates for future data-centric computing requirements and add remarkable vigor to the research of next-generation artificial intelligence (AI) systems, particularly those that involve brain-like intelligence applications. This work provides an overview of the evolution of memristor-based devices, from their initial use in creating artificial synapses and neural networks to their application in developing advanced AI systems and brain-like chips. It offers a broad perspective of the key device primitives enabling their special applications from the view of materials, nanostructure, and mechanism models. We highlight these demonstrations of memristor-based nanoelectronic devices that have potential for use in the field of brain-like AI, point out the existing challenges of memristor-based nanodevices toward brain-like chips, and propose the guiding principle and promising outlook for future device promotion and system optimization in the biomedical AI field.