In-Sensor Computing: Materials, Devices, and Integration Technologies

A Single-Pixel Event Photoactive Device for Real-Time, In-Sensor Spatiotemporal Optical Information Processing

The increasing demand for energy-efficient, sophisticated optical sensing technologies in various applications, from machine vision to optical communication, highlights the necessity for innovations in spatiotemporal information sensing and processing at a nearly single-pixel scale. Traditional methods, including multi-pixel photodetector arrays and event-based camera systems, often fail to provide rapid, real-time detection and processing of dynamic events within the sensor. This shortcoming is particularly notable in handling high-dimensional spatiotemporal data, where the dependency on sequential data input and external processing tools leads to latency, reduced throughput, and heightened energy consumption, thereby impeding real-time parallel data processing capabilities. Here, a carrier-selective, single-pixel, position-sensitive planar photoactive device that integrates spatiotemporal event sensing with inherent short-term memory capabilities is introduced. The proof-of-concept single-pixel event photoactive device enables in-sensor spatiotemporal parallel optical information processing, efficiently managing multibit (>4 bit) data simultaneously and facilitating ultrafast (≈0.4 µs) recognition of input patterns with low energy consumption (25 femtojoules). Additionally, by adjusting the operating speed from continuous to pulsed light illumination, the sensor array can detect trajectories and absolute position of events, offering in-sensor optical flow detection. This single-pixel event photodetector marks significant advancement toward developing compact, energy-efficient, ultrafast sensors suitable for a wide range of in sensor-based photonic applications.

Harnessing a silicon carbide nanowire photoelectric synaptic device for novel visual adaptation spiking neural networks

Visual adaptation is essential for optimizing the image quality and sensitivity of artificial vision systems in real-world lighting conditions. However, additional modules, leading to time delays and potentially increasing power consumption, are needed for traditional artificial vision systems to implement visual adaptation. Here, an ITO/PMMA/SiC-NWs/ITO photoelectric synaptic device is developed for compact artificial vision systems with the visual adaption function. The theoretical calculation and experimental results demonstrated that the heating effect, induced by the increment light intensity, leads to the photoelectric synaptic device enabling the visual adaption function. Additionally, a visual adaptation artificial neuron (VAAN) circuit was implemented by incorporating the photoelectric synaptic device into a LIF neuron circuit. The output frequency of this VAAN circuit initially increases and then decreases with gradual light intensification, reflecting the dynamic process of visual adaptation. Furthermore, a visual adaptation spiking neural network (VASNN) was constructed to evaluate the photoelectric synaptic device based visual system for perception tasks. The results indicate that, in the task of traffic sign detection under extreme weather conditions, an accuracy of 97% was achieved (which is approximately 12% higher than that without a visual adaptation function). Our research provides a biologically plausible hardware solution for visual adaptation in neuromorphic computing.

A Plasmonic Optoelectronic Resistive Random-Access Memory for In-Sensor Color Image Cryptography

The optoelectronic resistive random-access memory (RRAM) with the integrated function of perception, storage and intrinsic randomness displays promising applications in the hardware level in-sensor image cryptography. In this work, 2D hexagonal boron nitride based optoelectronic RRAM is fabricated with semitransparent noble metal (Ag or Au) as top electrodes, which can simultaneous capture color image and generate physically unclonable function (PUF) key for in-sensor color image cryptography. Surface plasmons of noble metals enable the strong light absorption to realize an efficient modulation of filament growth at nanoscale. Resistive switching curves show that the optical stimuli can impede the filament aggregation and promote the filament annihilation, which originates from photothermal effects and photogenerated hot electrons in localized surface plasmon resonance of noble metals. By selecting noble metals, the optoelectronic RRAM array can respond to distinct wavelengths and mimic the biological dichromatic cone cells to perform the color perception. Due to the intrinsic and high-quality randomness, the optoelectronic RRAM can produce a PUF key in every exposure cycle, which can be applied in the reconfigurable cryptography. The findings demonstrate an effective strategy to build optoelectronic RRAM for in-sensor color image cryptography applications.

Dimension-Controlled VO2 Film for Optoelectronic Logic Gates and Information Encryption

As the fields of photonics and information technology develop, a lot of novel applications based on VO2 material, such as optoelectronic computing and information encryption, have been developed. While the performance of these devices was not only closely associated with the VO2 phase transition properties but also depended on their dimensional characteristics. In the current study, we conducted the dimension-controlled vanadium dioxide (VO2) film growth, resulting in the epitaxial 2-dimensional (2D) VO2 film and well-distributed 3-dimensional (3D) VO2 crystal film deposition, respectively. It was revealed that, unlike the 2D film, the pronounced localized surface plasmon resonance dominated the near-infrared spectrum across the phase transition for the 3D VO2 film due to the naturally formed meta-surface structure, which showed a transmittance valley in the infrared spectrum after metallization. Based on this distinct infrared spectrum feature in the 3D VO2 film, we proposed an optoelectronic logic gate controlled by the input voltage and the probing Vis/IR light. By detecting the transmittance states of the probing light with different wavelengths, we achieved multistate encoding functions and demonstrated the information encryption application. This new conception device also showed great potential for some other applications such as optoelectronic coupled computing, information encryption, and optical near-field sensing computing.

Optoelectronic Devices for In-Sensor Computing

The demand for accurate perception of the physical world leads to a dramatic increase in sensory nodes. However, the transmission of massive and unstructured sensory data from sensors to computing units poses great challenges in terms of power-efficiency, transmission bandwidth, data storage, time latency, and security. To efficiently process massive sensory data, it is crucial to achieve data compression and structuring at the sensory terminals. In-sensor computing integrates perception, memory, and processing functions within sensors, enabling sensory terminals to perform data compression and data structuring. Here, vision sensors are adopted as an example and discuss the functions of electronic, optical, and optoelectronic hardware for visual processing. Particularly, hardware implementations of optoelectronic devices for in-sensor visual processing that can compress and structure multidimensional vision information are examined. The underlying resistive switching mechanisms of volatile/nonvolatile optoelectronic devices and their processing operations are explored. Finally, a perspective on the future development of optoelectronic devices for in-sensor computing is provided.

Spike-Based Neuromorphic Hardware for Dynamic Tactile Perception with a Self-Powered Mechanoreceptor Array

A self-powered mechanoreceptor array is demonstrated using four mechanoreceptor cells for recognition of dynamic touch gestures. Each cell consists of a triboelectric nanogenerator (TENG) for touch sensing and a bi-stable resistor (biristor) for spike encoding. It produces informative spike signals by sensing a force of an external touch and encoding the force into the number of spikes. An array of the mechanoreceptor cells is utilized to monitor various touch gestures and it successfully generated spike signals corresponding to all the gestures. To validate the practicality of the mechanoreceptor array, a spiking neural network (SNN), highly attractive for power consumption compared to the conventional von Neumann architecture, is used for the identification of touch gestures. The measured spiking signals are reflected as inputs for the SNN simulations. Consequently, touch gestures are classified with a high accuracy rate of 92.5%. The proposed mechanoreceptor array emerges as a promising candidate for a building block of tactile in-sensor computing in the era of the Internet of Things (IoT), due to the low cost and high manufacturability of the TENG. This eliminates the need for a power supply, coupled with the intrinsic high throughput of the Si-based biristor employing complementary metal-oxide-semiconductor (CMOS) technology.

Multidimensional vision sensors for information processing

The visual scene in the physical world integrates multidimensional information (spatial, temporal, polarization, spectrum and so on) and typically shows unstructured characteristics. Conventional image sensors cannot process this multidimensional vision data, creating a need for vision sensors that can efficiently extract features from substantial multidimensional vision data. Vision sensors are able to transform the unstructured visual scene into featured information without relying on sophisticated algorithms and complex hardware. The response characteristics of sensors can be abstracted into operators with specific functionalities, allowing for the efficient processing of perceptual information. In this Review, we delve into the hardware implementation of multidimensional vision sensors, exploring their working mechanisms and design principles. We exemplify multidimensional vision sensors built on emerging devices and silicon-based system integration. We further provide benchmarking metrics for multidimensional vision sensors and conclude with the principle of device-system co-design and co-optimization.

Autonomous Artificial Olfactory Sensor Systems with Homeostasis Recovery via a Seamless Neuromorphic Architecture

Neuromorphic olfactory systems have been actively studied in recent years owing to their considerable potential in electronic noses, robotics, and neuromorphic data processing systems. However, conventional gas sensors typically have the ability to detect hazardous gas levels but lack synaptic functions such as memory and recognition of gas accumulation, which are essential for realizing human-like neuromorphic sensory system. In this study, a seamless architecture for a neuromorphic olfactory system capable of detecting and memorizing the present level and accumulation status of nitrogen dioxide (NO2) during continuous gas exposure, regulating a self-alarm implementation triggered after 147 and 85 s at a continuous gas exposure of 20 and 40 ppm, respectively. Thin-film-transistor type gas sensors utilizing carbon nanotube semiconductors detect NO2 gas molecules through carrier trapping and exhibit long-term retention properties, which are compatible with neuromorphic excitatory applications. Additionally, the neuromorphic inhibitory performance is also characterized via gas desorption with programmable ultraviolet light exposure, demonstrating homeostasis recovery. These results provide a promising strategy for developing a facile artificial olfactory system that demonstrates complicated biological synaptic functions with a seamless and simplified system architecture.

Gate-Tunable Positive and Negative Photoconductance in Near-Infrared Organic Heterostructures for In-Sensor Computing

The rapid growth of sensor data in the artificial intelligence often causes significant reductions in processing speed and power efficiency. Addressing this challenge, in-sensor computing is introduced as an advanced sensor architecture that simultaneously senses, memorizes, and processes images at the sensor level. However, this is rarely reported for organic semiconductors that possess inherent flexibility and tunable bandgap. Herein, an organic heterostructure that exhibits a robust photoresponse to near-infrared (NIR) light is introduced, making it ideal for in-sensor computing applications. This heterostructure, consisting of partially overlapping p-type and n-type organic thin films, is compatible with conventional photolithography techniques, allowing for high integration density of up to 520 devices cm-2 with a 5 µm channel length. Importantly, by modulating gate voltage, both positive and negative photoresponses to NIR light (1050 nm) are attained, which establishes a linear correlation between responsivity and gate voltage and consequently enables real-time matrix multiplication within the sensor. As a result, this organic heterostructure facilitates efficient and precise NIR in-sensor computing, including image processing and nondestructive reading and classification, achieving a recognition accuracy of 97.06%. This work serves as a foundation for the development of reconfigurable and multifunctional NIR neuromorphic vision systems.

Coexistance of the Negative Photoconductance Effect and Analogue Switching Memory in the CuPc Organic Memristor for Neuromorphic Vision Computing

A bioinspired in-sensing computing paradigm using emerging photoelectronic memristors pursues multifunctionality with low power consumption and high efficiency for processing large amounts of sensing information. An organic semiconductor memristor strategy based on the CuPc functional layer integrates a negative photoconductance (NPC) effect and an analogue switching memory (ASM) effect in the same pixel. The NPC effect, present in the pure capacitance state at low bias voltage, provides high-performance short/long-term synaptic plasticity modulable by light pulse parameters. The interface charge effect along with defeat site trapping and detrapping is responsible for the pure capacitance effect and the NPC effect, with electron tunneling and electric-field-driven band dynamics responsible for ASM. This work reveals an organic memristor approach for hardware implementation of a neuromorphic vision computing system, emulating retinal bipolar cells via light-dominated NPC and electrically induced ASM with stable, tunable conductance states.

Transformative Multifunction Deep Ultraviolet Photodetectors for On-Demand Applications: From Fast Optical Communication to Tunable In-Sensor Photocurrent Integration

The strategic utilization of photodetectors' transient response could open new frontiers from free-space optical communication to the emerging field of neuromorphic optoelectronics. Contrarily, while communication requires a fast response, neuromorphic applications benefit from a slow and integrative transient photocurrent. By integrating these functionalities in a single device, this study unveils a photodetector with tunable responses, bridging the gap between optical communication and neuromorphic sensing and creating a versatile platform with on-demand applications. Particularly, a Ga2O3-based photodetector was designed, exhibiting a photocurrent on/off ratio close to 104, high responsivity of 0.43 A/W, and detectivity 1.22 × 1013 Jones under deep ultraviolet illumination (λ ∼ 260 nm). The photodetector demonstrates transient time-dependent on operational voltage, ranging from 10-4 to 0.2 s. The underlying mechanism is attributed to the voltage-dependent balance between photocarrier generation and defect-related recombination, as revealed by electrostatic force microscopy. Additionally, we have demonstrated potential applications, including digital Morse code interpretation, tunable integration of optical inputs within the sensor, one-time readouts, and effective analog Morse code reading. Furthermore, the effectiveness of input information recognition using analog integration, even with anomalies, was demonstrated. This work establishes a versatile approach for tunable in-sensor optical processing, potentially useful for a wide range of applications, from free-space optical communication to neuromorphic sensing.

AI-Driven Sensing Technology: Review

Machine learning and deep learning technologies are rapidly advancing the capabilities of sensing technologies, bringing about significant improvements in accuracy, sensitivity, and adaptability. These advancements are making a notable impact across a broad spectrum of fields, including industrial automation, robotics, biomedical engineering, and civil infrastructure monitoring. The core of this transformative shift lies in the integration of artificial intelligence (AI) with sensor technology, focusing on the development of efficient algorithms that drive both device performance enhancements and novel applications in various biomedical and engineering fields. This review delves into the fusion of ML/DL algorithms with sensor technologies, shedding light on their profound impact on sensor design, calibration and compensation, object recognition, and behavior prediction. Through a series of exemplary applications, the review showcases the potential of AI algorithms to significantly upgrade sensor functionalities and widen their application range. Moreover, it addresses the challenges encountered in exploiting these technologies for sensing applications and offers insights into future trends and potential advancements.

A Spiking Artificial Vision Architecture Based on Fully Emulating the Human Vision

Intelligent vision necessitates the deployment of detectors that are always-on and low-power, mirroring the continuous and uninterrupted responsiveness characteristic of human vision. Nonetheless, contemporary artificial vision systems attain this goal by the continuous processing of massive image frames and executing intricate algorithms, thereby expending substantial computational power and energy. In contrast, biological data processing, based on event-triggered spiking, has higher efficiency and lower energy consumption. Here, this work proposes an artificial vision architecture consisting of spiking photodetectors and artificial synapses, closely mirroring the intricacies of the human visual system. Distinct from previously reported techniques, the photodetector is self-powered and event-triggered, outputting light-modulated spiking signals directly, thereby fulfilling the imperative for always-on with low-power consumption. With the spiking signals processing through the integrated synapse units, recognition of graphics, gestures, and human action has been implemented, illustrating the potent image processing capabilities inherent within this architecture. The results prove the 90% accuracy rate in human action recognition within a mere five epochs utilizing a rudimentary artificial neural network. This novel architecture, grounded in spiking photodetectors, offers a viable alternative to the extant models of always-on low-power artificial vision system.

Visualized in-sensor computing

In artificial nervous systems, conductivity changes indicate synaptic weight updates, but they provide limited information compared to living organisms. We present the pioneering design and production of an electrochromic neuromorphic transistor employing color updates to represent synaptic weight for in-sensor computing. Here, we engineer a specialized mechanism for adaptively regulating ion doping through an ion-exchange membrane, enabling precise control over color-coded synaptic weight, an unprecedented achievement. The electrochromic neuromorphic transistor not only enhances electrochromatic capabilities for hardware coding but also establishes a visualized pattern-recognition network. Integrating the electrochromic neuromorphic transistor with an artificial whisker, we simulate a bionic reflex system inspired by the longicorn beetle, achieving real-time visualization of signal flow within the reflex arc in response to environmental stimuli. This research holds promise in extending the biomimetic coding paradigm and advancing the development of bio-hybrid interfaces, particularly in incorporating color-based expressions.

In-Memory Sensing and Computing for Cancer Diagnostics: A Perspective Paper

During the past two decades, a number of two-terminal switching devices have been demonstrated in the literature. They typically exhibit hysteric behavior in the current-to-voltage characteristics. These devices have often been also referred to as memristive devices. Their capacity to switch and exhibit electrical hysteresis has made them well-suited for applications such as data storage, in-memory computing, and in-sensor computing or in-memory sensing. The aim of this perspective paper is to is twofold. Firstly, it seeks to provide a comprehensive examination of the existing research findings in the field and engage in a critical discussion regarding the potential for the development of new non-Von-Neumann computing machines that can seamlessly integrate sensing and computing within memory units. Secondly, this paper aims to demonstrate the practical application of such an innovative approach in the realm of cancer medicine. Specifically, it explores the modern concept of employing multiple cancer markers simultaneously to enhance the efficiency of diagnostic processes in cancer medicine.

Coupled Ferroelectric-Photonic Memory in a Retinomorphic Hardware for In-Sensor Computing

The development of all-in-one devices for artificial visual systems offers an attractive solution in terms of energy efficiency and real-time processing speed. In recent years, the proliferation of smart sensors in the growth of Internet-of-Things (IoT) has led to the increasing importance of in-sensor computing technology, which places computational power at the edge of the data-flow architecture. In this study, a prototype visual sensor inspired by the human retina is proposed, which integrates ferroelectricity and photosensitivity in two-dimensional (2D) α-In2Se3 material. This device mimics the functions of photoreceptors and amacrine cells in the retina, performing optical reception and memory computation functions through the use of electrical switching polarization in the channel. The gate-tunable linearity of excitatory and inhibitory functions in photon-induced short-term plasticity enables to encode and classify 12 000 images in the Mixed National Institute of Standards and Technology (MNIST) dataset with remarkable accuracy, achieving ≈94%. Additionally, in-sensor convolution image processing through a network of phototransistors, with five convolutional kernels electrically pre-programmed into the transistors is demonstrated. The convoluted photocurrent matrices undergo straightforward arithmetic calculations to produce edge and feature-enhanced scenarios. The findings demonstrate the potential of ferroelectric α-In2Se3 for highly compact and efficient retinomorphic hardware implementation, regardless of ambipolar transport in the channel.

Neuromorphic Nanoionics for Human-Machine Interaction: From Materials to Applications

Human-machine interaction (HMI) technology has undergone significant advancements in recent years, enabling seamless communication between humans and machines. Its expansion has extended into various emerging domains, including human healthcare, machine perception, and biointerfaces, thereby magnifying the demand for advanced intelligent technologies. Neuromorphic computing, a paradigm rooted in nanoionic devices that emulate the operations and architecture of the human brain, has emerged as a powerful tool for highly efficient information processing. This paper delivers a comprehensive review of recent developments in nanoionic device-based neuromorphic computing technologies and their pivotal role in shaping the next-generation of HMI. Through a detailed examination of fundamental mechanisms and behaviors, the paper explores the ability of nanoionic memristors and ion-gated transistors to emulate the intricate functions of neurons and synapses. Crucial performance metrics, such as reliability, energy efficiency, flexibility, and biocompatibility, are rigorously evaluated. Potential applications, challenges, and opportunities of using the neuromorphic computing technologies in emerging HMI technologies, are discussed and outlooked, shedding light on the fusion of humans with machines.

An in-sensor humidity computing system for contactless human-computer interaction

Being capable of processing large amounts of redundant data and decreasing power consumption, in-sensor computing approaches play significant roles in neuromorphic computing and are attracting increasing interest in perceptual information processing. Herein, we proposed a high performance humidity-sensitive memristor based on a Ti/graphene oxide (GO)/HfOx/Pt structure and verified its potential for application in remote health management and contactless human-machine interfaces. Since GO possesses abundant hydrophilic groups (carbonyl, epoxide, and hydroxyl), the memristor shows a high humidity sensitivity, fast response, and wide response range. By utilizing the proton-modulated redox reaction, humidity exposure to the memristor induces a dynamic change in the switching between high and low resistance states, ensuring essential synaptic learning functions, such as paired-pulse facilitation, spike number-dependent plasticity, and spike amplitude-dependent plasticity. More importantly, based on the humidity-induced salient features originating from the abundant hydrophilic functional groups in GO, we have implemented a noncontact human-machine interface utilizing the respiratory mode in humans, demonstrating the potential of promoting health monitoring applications and effectively blocking virus transmission. In addition, the high recognition accuracy of contactless handwriting in a 5 × 5 array artificial neural network was successfully achieved, which is attributed to the excellent emulated synaptic behaviors. This study provides a feasible method to develop an excellent humidity-sensitive memristor for constructing efficient in-sensor computing for application in health management and contactless human-computer interaction.

A Diffusive Artificial Synapse Based on Charged Metal Nanoparticles

We show that a diffusive memristor with analogue switching characteristics can be achieved in a layer of gold nanoparticles (AuNPs) functionalized with charged self-assembled monolayers (deprotonated 11-mercaptoundecanoic acid). The nanoparticle core and the anchored stationary charges are jammed within the layer while the mobile counterions [N(CH3)4+] can respond to the electric field and spontaneously diffuse back to the initial positions upon removal of the field. This metal nanoparticle device is set-step free, energy consumption efficient, mechanically flexible, and analogous to bio-Ca2+ dynamics and has tunable conductance modulation capabilities at the counterion concentrations. The gradual resistive switching behavior enables us to implement several important synaptic functions such as potentiation/depression, spike voltage-dependent plasticity, spike duration-dependent plasticity, spike frequency-dependent plasticity, and paired-pulse facilitation. Finally, on the basis of the paired-pulse facilitation characteristics, the metal nanoparticle diffusive artificial synapse is used for edge extraction with exhibits excellent performance.

Toward a Brain-Neuromorphics Interface

Brain-computer interfaces (BCIs) that enable human-machine interaction have immense potential in restoring or augmenting human capabilities. Traditional BCIs are realized based on complementary metal-oxide-semiconductor (CMOS) technologies with complex, bulky, and low biocompatible circuits, and suffer with the low energy efficiency of the von Neumann architecture. The brain-neuromorphics interface (BNI) would offer a promising solution to advance the BCI technologies and shape the interactions with machineries. Neuromorphic devices and systems are able to provide substantial computation power with extremely high energy-efficiency by implementing in-materia computing such as in situ vector-matrix multiplication (VMM) and physical reservoir computing. Recent progresses on integrating neuromorphic components with sensing and/or actuating modules, give birth to the neuromorphic afferent nerve, efferent nerve, sensorimotor loop, and so on, which has advanced the technologies for future neurorobotics by achieving sophisticated sensorimotor capabilities as the biological system. With the development on the compact artificial spiking neuron and bioelectronic interfaces, the seamless communication between a BNI and a bioentity is reasonably expectable. In this review, the upcoming BNIs are profiled by introducing the brief history of neuromorphics, reviewing the recent progresses on related areas, and discussing the future advances and challenges that lie ahead.

Low-dimensional nanostructures for monolithic 3D-integrated flexible and stretchable electronics

Flexible/stretchable electronics, which are characterized by their ultrathin design, lightweight structure, and excellent mechanical robustness and conformability, have garnered significant attention due to their unprecedented potential in healthcare, advanced robotics, and human-machine interface technologies. An increasing number of low-dimensional nanostructures with exceptional mechanical, electronic, and/or optical properties are being developed for flexible/stretchable electronics to fulfill the functional and application requirements of information sensing, processing, and interactive loops. Compared to the traditional single-layer format, which has a restricted design space, a monolithic three-dimensional (M3D) integrated device architecture offers greater flexibility and stretchability for electronic devices, achieving a high-level of integration to accommodate the state-of-the-art design targets, such as skin-comfort, miniaturization, and multi-functionality. Low-dimensional nanostructures possess small size, unique characteristics, flexible/elastic adaptability, and effective vertical stacking capability, boosting the advancement of M3D-integrated flexible/stretchable systems. In this review, we provide a summary of the typical low-dimensional nanostructures found in semiconductor, interconnect, and substrate materials, and discuss the design rules of flexible/stretchable devices for intelligent sensing and data processing. Furthermore, artificial sensory systems in 3D integration have been reviewed, highlighting the advancements in flexible/stretchable electronics that are deployed with high-density, energy-efficiency, and multi-functionalities. Finally, we discuss the technical challenges and advanced methodologies involved in the design and optimization of low-dimensional nanostructures, to achieve monolithic 3D-integrated flexible/stretchable multi-sensory systems.

Nanoelectronics Using Metal-Insulator Transition

Metal-insulator transition (MIT) coupled with an ultrafast, significant, and reversible resistive change in Mott insulators has attracted tremendous interest for investigation into next-generation electronic and optoelectronic devices, as well as a fundamental understanding of condensed matter systems. Although the mechanism of MIT in Mott insulators is still controversial, great efforts have been made to understand and modulate MIT behavior for various electronic and optoelectronic applications. In this review, recent progress in the field of nanoelectronics utilizing MIT is highlighted. A brief introduction to the physics of MIT and its underlying mechanisms is begun. After discussing the MIT behaviors of various Mott insulators, recent advances in the design and fabrication of nanoelectronics devices based on MIT, including memories, gas sensors, photodetectors, logic circuits, and artificial neural networks are described. Finally, an outlook on the development and future applications of nanoelectronics utilizing MIT is provided. This review can serve as an overview and a comprehensive understanding of the design of MIT-based nanoelectronics for future electronic and optoelectronic devices.

Nanomaterial-Based Artificial Vision Systems: From Bioinspired Electronic Eyes to In-Sensor Processing Devices

High-performance robotic vision empowers mobile and humanoid robots to detect and identify their surrounding objects efficiently, which enables them to cooperate with humans and assist human activities. For error-free execution of these robots' tasks, efficient imaging and data processing capabilities are essential, even under diverse and complex environments. However, conventional technologies fall short of meeting the high-standard requirements of robotic vision under such circumstances. Here, we discuss recent progress in artificial vision systems with high-performance imaging and data processing capabilities enabled by distinctive electrical, optical, and mechanical characteristics of nanomaterials surpassing the limitations of traditional silicon technologies. In particular, we focus on nanomaterial-based electronic eyes and in-sensor processing devices inspired by biological eyes and animal visual recognition systems, respectively. We provide perspectives on key nanomaterials, device components, and their functionalities, as well as explain the remaining challenges and future prospects of the artificial vision systems.

Full hardware implementation of neuromorphic visual system based on multimodal optoelectronic resistive memory arrays for versatile image processing

In-sensor and near-sensor computing are becoming the next-generation computing paradigm for high-density and low-power sensory processing. To fulfil a high-density and efficient neuromorphic visual system with fully hierarchical emulation of the retina and visual cortex, emerging multimodal neuromorphic devices for multi-stage processing and a fully hardware-implemented system with versatile image processing functions are still lacking and highly desirable. Here we demonstrate an emerging multimodal-multifunctional resistive random-access memory (RRAM) device array based on modified silk fibroin protein (MSFP), exhibiting both optoelectronic RRAM (ORRAM) mode featured by unique negative and positive photoconductance memory and electrical RRAM (ERRAM) mode featured by analogue resistive switching. A full hardware implementation of the artificial visual system with versatile image processing functions is realised for the first time, including ORRAM mode array for the in-sensor image pre-processing (contrast enhancement, background denoising, feature extraction) and ERRAM mode array for near-sensor high-level image recognition, which hugely improves the integration density, and simply the circuit design and the fabrication and integration complexity.

Porous crystalline materials for memories and neuromorphic computing systems

Porous crystalline materials usually include metal-organic frameworks (MOFs), covalent organic frameworks (COFs), hydrogen-bonded organic frameworks (HOFs) and zeolites, which exhibit exceptional porosity and structural/composition designability, promoting the increasing attention in memory and neuromorphic computing systems in the last decade. From both the perspective of materials and devices, it is crucial to provide a comprehensive and timely summary of the applications of porous crystalline materials in memory and neuromorphic computing systems to guide future research endeavors. Moreover, the utilization of porous crystalline materials in electronics necessitates a shift from powder synthesis to high-quality film preparation to ensure high device performance. This review highlights the strategies for preparing porous crystalline materials films and discusses their advancements in memory and neuromorphic electronics. It also provides a detailed comparative analysis and presents the existing challenges and future research directions, which can attract the experts from various fields (e.g., materials scientists, chemists, and engineers) with the aim of promoting the applications of porous crystalline materials in memory and neuromorphic computing systems.

Energy Efficient Artificial Olfactory System with Integrated Sensing and Computing Capabilities for Food Spoilage Detection

Artificial olfactory systems (AOSs) that mimic biological olfactory systems are of great interest. However, most existing AOSs suffer from high energy consumption levels and latency issues due to data conversion and transmission. In this work, an energy- and area-efficient AOS based on near-sensor computing is proposed. The AOS efficiently integrates an array of sensing units (merged field effect transistor (FET)-type gas sensors and amplifier circuits) and an AND-type nonvolatile memory (NVM) array. The signals of the sensing units are directly connected to the NVM array and are computed in memory, and the meaningful linear combinations of signals are output as bit line currents. The AOS is designed to detect food spoilage by employing thin zinc oxide films as gas-sensing materials, and it exhibits low detection limits for H2 S and NH3 gases (0.01 ppm), which are high-protein food spoilage markers. As a proof of concept, monitoring the entire spoilage process of chicken tenderloin is demonstrated. The system can continuously track freshness scores and food conditions throughout the spoilage process. The proposed AOS platform is applicable to various applications due to its ability to change the sensing temperature and programmable NVM cells.

Broadband sensory networks with locally stored responsivities for neuromorphic machine vision

As the most promising candidates for the implementation of in-sensor computing, retinomorphic vision sensors can constitute built-in neural networks and directly implement multiply-and-accumulation operations using responsivities as the weights. However, existing retinomorphic vision sensors mainly use a sustained gate bias to maintain the responsivity due to its volatile nature. Here, we propose an ion-induced localized-field strategy to develop retinomorphic vision sensors with nonvolatile tunable responsivity in both positive and negative regimes and construct a broadband and reconfigurable sensory network with locally stored weights to implement in-sensor convolutional processing in spectral range of 400 to 1800 nanometers. In addition to in-sensor computing, this retinomorphic device can implement in-memory computing benefiting from the nonvolatile tunable conductance, and a complete neuromorphic visual system involving front-end in-sensor computing and back-end in-memory computing architectures has been constructed, executing supervised and unsupervised learning tasks as demonstrations. This work paves the way for the development of high-speed and low-power neuromorphic machine vision for time-critical and data-intensive applications.

Emerging Iontronic Neural Devices for Neuromorphic Sensory Computing

Living organisms have a very mysterious and powerful sensory computing system based on ion activity. Interestingly, studies on iontronic devices in the past few years have proposed a promising platform for simulating the sensing and computing functions of living organisms, because: 1) iontronic devices can generate, store, and transmit a variety of signals by adjusting the concentration and spatiotemporal distribution of ions, which analogs to how the brain performs intelligent functions by alternating ion flux and polarization; 2) through ionic-electronic coupling, iontronic devices can bridge the biosystem with electronics and offer profound implications for soft electronics; 3) with the diversity of ions, iontronic devices can be designed to recognize specific ions or molecules by customizing the charge selectivity, and the ionic conductivity and capacitance can be adjusted to respond to external stimuli for a variety of sensing schemes, which can be more difficult for electron-based devices. This review provides a comprehensive overview of emerging neuromorphic sensory computing by iontronic devices, highlighting representative concepts of both low-level and high-level sensory computing and introducing important material and device breakthroughs. Moreover, iontronic devices as a means of neuromorphic sensing and computing are discussed regarding the pending challenges and future directions.

Electrochemical-Memristor-Based Artificial Neurons and Synapses-Fundamentals, Applications, and Challenges

Artificial neurons and synapses are considered essential for the progress of the future brain-inspired computing, based on beyond von Neumann architectures. Here, a discussion on the common electrochemical fundamentals of biological and artificial cells is provided, focusing on their similarities with the redox-based memristive devices. The driving forces behind the functionalities and the ways to control them by an electrochemical-materials approach are presented. Factors such as the chemical symmetry of the electrodes, doping of the solid electrolyte, concentration gradients, and excess surface energy are discussed as essential to understand, predict, and design artificial neurons and synapses. A variety of two- and three-terminal memristive devices and memristive architectures are presented and their application for solving various problems is shown. The work provides an overview of the current understandings on the complex processes of neural signal generation and transmission in both biological and artificial cells and presents the state-of-the-art applications, including signal transmission between biological and artificial cells. This example is showcasing the possibility for creating bioelectronic interfaces and integrating artificial circuits in biological systems. Prospectives and challenges of the modern technology toward low-power, high-information-density circuits are highlighted.

Reversible Diode with Tunable Band Alignment for Photoelectricity-Induced Artificial Synapse

The advent of big data era has put forward higher requirements for electronic nanodevices that have low energy consumption for their application in analog computing with memory and logic circuit to address attendant energy efficiency issues. Here, a miniaturized diode with a reversible switching state based on N-n MoS2 homojunction used a bandgap renormalization effect through the band alignment type regulated by both dielectric and polarization, controllably switched between type-I and type-II, which can be simulated as artificial synapse for sensing memory processing because of its rectification, nonvolatile characteristic and high optical responsiveness. The device demonstrates a rectification ratio of 103 . When served as memory retention time, it can attain at least 7000 s. For the synapse simulation, it has an ultralow-level energy consumption because of the pA-level operation current with 5 pJ for long-term potentiation and 7.8 fJ for long-term depression. Furthermore, the paired pulse facilitation index reaches up to 230%, and it realizes the function of optical storage that can be applied to simulate visual cells.

Optoelectronic graded neurons for bioinspired in-sensor motion perception

Motion processing has proven to be a computational challenge and demands considerable computational resources. Contrast this with the fact that flying insects can agilely perceive real-world motion with their tiny vision system. Here we show that phototransistor arrays can directly perceive different types of motion at sensory terminals, emulating the non-spiking graded neurons of insect vision systems. The charge dynamics of the shallow trapping centres in MoS2 phototransistors mimic the characteristics of graded neurons, showing an information transmission rate of 1,200 bit s-1 and effectively encoding temporal light information. We used a 20 × 20 photosensor array to detect trajectories in the visual field, allowing the efficient perception of the direction and vision saliency of moving objects and achieving 99.2% recognition accuracy with a four-layer neural network. By modulating the charge dynamics of the shallow trapping centres of MoS2, the sensor array can recognize motion with a temporal resolution ranging from 101 to 106 ms.

Highly Trustworthy In-Sensor Cryptography for Image Encryption and Authentication

The prevailing transmission of image information over the Internet of Things demands trustworthy cryptography for high security and privacy. State-of-the-art security modules are usually physically separated from the sensory terminals that capture images, which unavoidably exposes image information to various attacks during the transmission process. Here we develop in-sensor cryptography that enables capturing images and producing security keys in the same hardware devices. The generated key inherently binds to the captured images, which gives rise to highly trustworthy cryptography. Using the intrinsic electronic and optoelectronic characteristics of the 256 molybdenum disulfide phototransistor array, we can harvest electronic and optoelectronic binary keys with a physically unclonable function and further upgrade them into multiple-state ternary and double-binary keys, exhibiting high uniformity, uniqueness, randomness, and coding capacity. This in-sensor cryptography enables highly trustworthy image encryption to avoid passive attacks and image authentication to prevent unauthorized editions.

Near-Sensor Reservoir Computing for Gait Recognition via a Multi-Gate Electrolyte-Gated Transistor

The recent emergence of various smart wearable electronics has furnished the rapid development of human-computer interaction, medical health monitoring technologies, etc. Unfortunately, processing redundant motion and physiological data acquired by multiple wearable sensors using conventional off-site digital computers typically result in serious latency and energy consumption problems. In this work, a multi-gate electrolyte-gated transistor (EGT)-based reservoir device for efficient multi-channel near-sensor computing is reported. The EGT, exhibiting rich short-term dynamics under voltage modulation, can implement nonlinear parallel integration of the time-series signals thus extracting the temporal features such as the synchronization state and collective frequency in the inputs. The flexible EGT integrated with pressure sensors can perform on-site gait information analysis, enabling the identification of motion behaviors and Parkinson's disease. This near-sensor reservoir computing system offers a new route for rapid analysis of the motion and physiological signals with significantly improved efficiency and will lead to robust smart flexible wearable electronics.

Picoampere Dark Current and Electro-Opto-Coupled Sub-to-Super-linear Response from Mott-Transition Enabled Infrared Photodetector for Near-Sensor Vision Processing

Light-intensity selective superlinear photodetectors with ultralow dark current can provide an essential breakthrough for the development of high-performing near-sensor vision processing. However, the development of near-sensor vision processing is not only conceptually important for device operation (given that sensors naturally exhibit linear/sublinear responses), but also essential to get rid of the massive amount of data generated during object sensing and classification with noisy inputs. Therefore, achieving the giant superlinear photoresponse while maintaining the picoampere leakage current, irrespective of the measurement bias, is one of the most challenging tasks. Here, Mott material (vanadium dioxide) and silicon-based integrated infrared photodetectors are developed that show giant superlinear photoresponse (exponent >18) and ultralow dark current of 4.46 pA. Specifically, the device demonstrates an electro-opto-coupled insulator-to-metal transition, which leads to outstanding photocurrent on/off ratio (>10⁶ ), a high responsivity (>1 mA W^-1 ), and excellent detectivity (>10¹²  Jones), while maintaining response speed (τ_r  = 6 µs and τ_f  = 10 µs). Further, intensity-selective near-sensor processing is demonstrated and night vision pattern reorganization even with noisy inputs is exhibited. This research will pave the way for the creation of high-performance photodetectors with potential uses, such as in night vision, pattern recognition, and neuromorphic processing.

Nanomaterial-Based Synaptic Optoelectronic Devices for In-Sensor Preprocessing of Image Data

With the advance in information technologies involving machine vision applications, the demand for energy- and time-efficient acquisition, transfer, and processing of a large amount of image data has rapidly increased. However, current architectures of the machine vision system have inherent limitations in terms of power consumption and data latency owing to the physical isolation of image sensors and processors. Meanwhile, synaptic optoelectronic devices that exhibit photoresponse similar to the behaviors of the human synapse enable in-sensor preprocessing, which makes the front-end part of the image recognition process more efficient. Herein, we review recent progress in the development of synaptic optoelectronic devices using functional nanomaterials and their unique interfacial characteristics. First, we provide an overview of representative functional nanomaterials and device configurations for the synaptic optoelectronic devices. Then, we discuss the underlying physics of each nanomaterial in the synaptic optoelectronic device and explain related device characteristics that allow for the in-sensor preprocessing. We also discuss advantages achieved by the application of the synaptic optoelectronic devices to image preprocessing, such as contrast enhancement and image filtering. Finally, we conclude this review and present a short prospect.

Wearable in-sensor reservoir computing using optoelectronic polymers with through-space charge-transport characteristics for multi-task learning

In-sensor multi-task learning is not only the key merit of biological visions but also a primary goal of artificial-general-intelligence. However, traditional silicon-vision-chips suffer from large time/energy overheads. Further, training conventional deep-learning models is neither scalable nor affordable on edge-devices. Here, a material-algorithm co-design is proposed to emulate human retina and the affordable learning paradigm. Relying on a bottle-brush-shaped semiconducting p-NDI with efficient exciton-dissociations and through-space charge-transport characteristics, a wearable transistor-based dynamic in-sensor Reservoir-Computing system manifesting excellent separability, fading memory, and echo state property on different tasks is developed. Paired with a 'readout function' on memristive organic diodes, the RC recognizes handwritten letters and numbers, and classifies diverse costumes with accuracies of 98.04%, 88.18%, and 91.76%, respectively (higher than all reported organic semiconductors). In addition to 2D images, the spatiotemporal dynamics of RC naturally extract features of event-based videos, classifying 3 types of hand gestures at an accuracy of 98.62%. Further, the computing cost is significantly lower than that of the conventional artificial-neural-networks. This work provides a promising material-algorithm co-design for affordable and highly efficient photonic neuromorphic systems.

In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array

Detection and recognition of latent fingerprints play crucial roles in identification and security. However, the separation of sensor, memory, and processor in conventional ex-situ fingerprint recognition system seriously deteriorates the latency of decision-making and inevitably increases the overall computing power. In this work, a photoelectronic reservoir computing (RC) system, consisting of DUV photo-synapses and nonvolatile memristor array, is developed to detect and recognize the latent fingerprint with in-sensor and parallel in-memory computing. Through the Ga-rich design, we achieve amorphous GaO_x (a-GaO_x) photo-synapses with an enhanced persistent photoconductivity (PPC) effect. The PPC effect, which induces nonlinearly tunable conductivity, renders the a-GaO_x photo-synapses an ideal deep ultraviolet (DUV) photoelectronic reservoir, thus mapping the complex input vector into a dimensionality-reduced output vector. Connecting the reservoirs and a memristor array, we further construct an in-sensor RC system for latent fingerprint identification. The system maintains over 90% recognition accuracy for latent fingerprint within 15% stochastic noise level via the proposed dual-feature strategy. This work provides a subversive prototype system of DUV in-sensor RC for highly efficient recognition of latent fingerprints.

Progress of Materials and Devices for Neuromorphic Vision Sensors

The latest developments in bio-inspired neuromorphic vision sensors can be summarized in 3 keywords: smaller, faster, and smarter. (1) Smaller: Devices are becoming more compact by integrating previously separated components such as sensors, memory, and processing units. As a prime example, the transition from traditional sensory vision computing to in-sensor vision computing has shown clear benefits, such as simpler circuitry, lower power consumption, and less data redundancy. (2) Swifter: Owing to the nature of physics, smaller and more integrated devices can detect, process, and react to input more quickly. In addition, the methods for sensing and processing optical information using various materials (such as oxide semiconductors) are evolving. (3) Smarter: Owing to these two main research directions, we can expect advanced applications such as adaptive vision sensors, collision sensors, and nociceptive sensors. This review mainly focuses on the recent progress, working mechanisms, image pre-processing techniques, and advanced features of two types of neuromorphic vision sensors based on near-sensor and in-sensor vision computing methodologies.