Bioinspired multisensory neural network with crossmodal integration and recognition

SSTE: Syllable-Specific Temporal Encoding to FORCE-learn audio sequences with an associative memory approach

The circuitry and pathways in the brains of humans and other species have long inspired researchers and system designers to develop accurate and efficient systems capable of solving real-world problems and responding in real-time. We propose the Syllable-Specific Temporal Encoding (SSTE) to learn vocal sequences in a reservoir of Izhikevich neurons, by forming associations between exclusive input activities and their corresponding syllables in the sequence. Our model converts the audio signals to cochleograms using the CAR-FAC model to simulate a brain-like auditory learning and memorization process. The reservoir is trained using a hardware-friendly approach to FORCE learning. Reservoir computing could yield associative memory dynamics with far less computational complexity compared to RNNs. The SSTE-based learning enables competent accuracy and stable recall of spatiotemporal sequences with fewer reservoir inputs compared with existing encodings in the literature for similar purpose, offering resource savings. The encoding points to syllable onsets and allows recalling from a desired point in the sequence, making it particularly suitable for recalling subsets of long vocal sequences. The SSTE demonstrates the capability of learning new signals without forgetting previously memorized sequences and displays robustness against occasional noise, a characteristic of real-world scenarios. The components of this model are configured to improve resource consumption and computational intensity, addressing some of the cost-efficiency issues that might arise in future implementations aiming for compactness and real-time, low-power operation. Overall, this model proposes a brain-inspired pattern generation network for vocal sequences that can be extended with other bio-inspired computations to explore their potentials for brain-like auditory perception. Future designs could inspire from this model to implement embedded devices that learn vocal sequences and recall them as needed in real-time. Such systems could acquire language and speech, operate as artificial assistants, and transcribe text to speech, in the presence of natural noise and corruption on audio data.

Bioinspired In-Sensor Multimodal Fusion for Enhanced Spatial and Spatiotemporal Association

Multimodal perception can capture more precise and comprehensive information compared with unimodal approaches. However, current sensory systems typically merge multimodal signals at computing terminals following parallel processing and transmission, which results in the potential loss of spatial association information and requires time stamps to maintain temporal coherence for time-series data. Here we demonstrate bioinspired in-sensor multimodal fusion, which effectively enhances comprehensive perception and reduces the level of data transfer between sensory terminal and computation units. By adopting floating gate phototransistors with reconfigurable photoresponse plasticity, we realize the agile spatial and spatiotemporal fusion under nonvolatile and volatile photoresponse modes. To realize an optimal spatial estimation, we integrate spatial information from visual-tactile signals. For dynamic events, we capture and fuse in real time spatiotemporal information from visual-audio signals, realizing a dance-music synchronization recognition task without a time-stamping process. This in-sensor multimodal fusion approach provides the potential to simplify the multimodal integration system, extending the in-sensor computing paradigm.

Highly Promising 2D/1D BP-C/CNT Bionic Opto-Olfactory Co-Sensory Artificial Synapses for Multisensory Integration

The development of high-performance artificial synaptic neuromorphic devices poses a significant challenge in the creation of biomimetic sensing neural systems that seamlessly integrate both sensory and computational functionalities. In pursuit of this objective, promising bionic opto-olfactory co-sensory artificial synapse devices are constructed utilizing the BP-C/CNT (2D/1D) hybrid filter membrane as the resistive layer. Experimental results demonstrated that the devices seamlessly integrated the light modulation, gas detection, and biological synaptic functions into a single device while addressing the challenge with separating artificial synaptic devices from sensors. These devices offered the following advantages: 1) Simulating visual synapses, they can effectively replicate fundamental synaptic functions under both electrical and optical stimulation. 2) By emulating olfactory synapse responses to specific gases, they can achieve ultra-low detection limits and rapid identification of ethanol and acetone gases. 3) They enable photo-olfactory co-sensing simulations that mimic synaptic function under light-modulated pulse conditions in distinct gas environments, facilitating the study of synaptic learning rules and Pavlovian responses. This work provides a pioneering approach for exploring highly stable 2D BP-based optoelectronics and advancing the development of biomimetic neural systems.

Additive Manufacturing Provides Infinite Possibilities for Self-Sensing Technology

Integrating sensors and other functional parts in one device can enable a new generation of integrated intelligent devices that can perform self-sensing and monitoring autonomously. Applications include buildings that detect and repair damage, robots that monitor conditions and perform real-time correction and reconstruction, aircraft capable of real-time perception of the internal and external environment, and medical devices and prosthetics with a realistic sense of touch. Although integrating sensors and other functional parts into self-sensing intelligent devices has become increasingly common, additive manufacturing has only been marginally explored. This review focuses on additive manufacturing integrated design, printing equipment, and printable materials and stuctures. The importance of the material, structure, and function of integrated manufacturing are highlighted. The study summarizes current challenges to be addressed and provides suggestions for future development directions.

Advanced Neuromorphic Applications Enabled by Synaptic Ion-Gating Vertical Transistors

Bioinspired synaptic devices have shown great potential in artificial intelligence and neuromorphic electronics. Low energy consumption, multi-modal sensing and recording, and multifunctional integration are critical aspects limiting their applications. Recently, a new synaptic device architecture, the ion-gating vertical transistor (IGVT), has been successfully realized and timely applied to perform brain-like perception, such as artificial vision, touch, taste, and hearing. In this short time, IGVTs have already achieved faster data processing speeds and more promising memory capabilities than many conventional neuromorphic devices, even while operating at lower voltages and consuming less power. This work focuses on the cutting-edge progress of IGVT technology, from outstanding fabrication strategies to the design and realization of low-voltage multi-sensing IGVTs for artificial-synapse applications. The fundamental concepts of artificial synaptic IGVTs, such as signal processing, transduction, plasticity, and multi-stimulus perception are discussed comprehensively. The contribution draws special attention to the development and optimization of multi-modal flexible sensor technologies and presents a roadmap for future high-end theoretical and experimental advancements in neuromorphic research that are mostly achievable by the synaptic IGVTs.

A Reconfigurable Bipolar Image Sensor for High-Efficiency Dynamic Vision Recognition

Dynamic vision perception and processing (DVPP) is in high demand by booming edge artificial intelligence. However, existing imaging systems suffer from low efficiency or low compatibility with advanced machine vision techniques. Here, we propose a reconfigurable bipolar image sensor (RBIS) for in-sensor DVPP based on a two-dimensional WSe2/GeSe heterostructure device. Owing to the gate-tunable and reversible built-in electric field, its photoresponse shows bipolarity as being positive or negative. High-efficiency DVPP incorporating front-end RBIS and back-end CNN is then demonstrated. It shows a high recognition accuracy of over 94.9% on the derived DVS128 data set and requires much fewer neural network parameters than that without RBIS. Moreover, we demonstrate an optimized device with a vertically stacked structure and a stable nonvolatile bipolarity, which enables more efficient DVPP hardware. Our work demonstrates the potential of fabricating DVPP devices with a simple structure, high efficiency, and outputs compatible with advanced algorithms.

Bio-Inspired Sensory Receptors for Artificial-Intelligence Perception

In the era of artificial intelligence (AI), there is a growing interest in replicating human sensory perception. Selective and sensitive bio-inspired sensory receptors with synaptic plasticity have recently gained significant attention in developing energy-efficient AI perception. Various bio-inspired sensory receptors and their applications in AI perception are reviewed here. The critical challenges for the future development of bio-inspired sensory receptors are outlined, emphasizing the need for innovative solutions to overcome hurdles in sensor design, integration, and scalability. AI perception can revolutionize various fields, including human-machine interaction, autonomous systems, medical diagnostics, environmental monitoring, industrial optimization, and assistive technologies. As advancements in bio-inspired sensing continue to accelerate, the promise of creating more intelligent and adaptive AI systems becomes increasingly attainable, marking a significant step forward in the evolution of human-like sensory perception.

Trainable bioinspired magnetic sensitivity adaptation using ferromagnetic colloidal assemblies

Nature has already suggested bioinspired functions. Beyond them, adaptive and trainable functions could be the inspiration for novel responsive soft matter beyond the state-of-the-art classic static bioinspired, stimulus-responsive, and shape-memory materials. Here, we describe magnetic assembly/disassembly of electrically conducting soft ferromagnetic nickel colloidal particles into surface topographical pillars for bistable electrical trainable memories. They allow magnetic sensing with adaptable and rescalable sensitivity ranges, enabled by bistable memories and kinetic concepts inspired by biological sensory adaptations. Based on the soft ferromagnetism of the nanogranular composition and the resulting rough particle surfaces prepared via a solvothermal synthesis, triggerable structural memory is achieved by the magnetic field-driven particle assembly and disassembly, promoted by interparticle jamming. Electrical conversion from current to frequency for electrical spikes facilitates rescalable and trainable frequency-based sensitivity on magnetic fields. This work suggests an avenue for designing trainable and adaptable life-inspired materials, for example, for soft robotics and interactive autonomous devices.

Utilizing a High-Performance Piezoelectric Nanocomposite as a Self-Activating Component in Piezotronic Artificial Mechanoreceptors

Challenges such as poor dispersion and insufficient polarization of BaTiO3 (BTO) nanoparticles (NPs) within poly(vinylidene fluoride-co-trifluoroethylene) (P(VDF-TrFE)) composites have hindered their piezoelectricity, limiting their uses in pressure sensors, nanogenerators, and artificial sensory synapses. Here, we introduce a high-performance piezoelectric nanocomposite material consisting of P(VDF-TrFE)/modified-BTO (mBTO) NPs for use as a self-activating component in a piezotronic artificial mechanoreceptor. To generate high-performance piezoelectric nanocomposite materials, the surface of BTO is hydroxylated, followed by the covalent attachment of (3-aminopropyl)triethoxysilane to improve the dispersibility of mBTO NPs within the P(VDF-TrFE) matrix. We also aim to enhance the crystallization degree of P(VDF-TrFE), the efficiency characteristics of mBTO, and the poling efficiency, even when incorporating small amounts of mBTO NPs. The piezoelectric potential mechanically induced from the P(VDF-TrFE)/mBTO NPs nanocomposite was three times greater than that from P(VDF-TrFE) and twice as high as that from the P(VDF-TrFE)/BTO NPs nanocomposite. The piezoelectric potential generated by mechanical stimuli on the piezoelectric nanocomposite was utilized to activate the synaptic ionogel-gated field-effect transistor for the development of self-powered piezotronics artificial mechanoreceptors on a polyimide substrate. The device successfully emulated fast-adapting (FA) functions found in biological FA mechanoreceptors. This approach has great potential for applications to future intelligent tactile perception technology.

Memristive Bionic Memory Circuit Implementation and Its Application in Multisensory Mutual Associative Learning Networks

Memory is vital and indispensable for organisms and brain-inspired intelligence to gain complete sensation and cognition of the environment. In this work, a memristive bionic memory circuit inspired by human memory model is proposed, which includes 1) receptor and sensory neuron (SN), 2) short-term memory (STM) module, and 3) long-term memory (LTM) module. By leveraging the in-memory computing characteristic of memristors, various functions such as sensation, learning, forgetting, recall, consolidation, reconsolidation, retrieval, and reset are realized. Besides, a multisensory mutual associative learning network is constructed with several bionic memory units to memorize and associate sensory information of different modalities bidirectionally. Except for association establishment, enhancement, and extinction, we also mimicked multisensory integration to manifest the synthetic process of information from different sensory channels. According to the simulation results in PSPICE, the proposed circuit performs high robustness, low area overhead, and low power consumption. Combining associative memory with human memory model, this work provides a possible idea for further research in associative learning networks.

A neuromorphic device mimicking synaptic plasticity under different body fluid K+ homeostasis for artificial reflex path construction and pattern recognition

The ionic environment of body fluids influences nervous functions for maintaining homeostasis in organisms and ensures normal perceptual abilities and reflex activities. Neural reflex activities, such as limb movements, are closely associated with potassium ions (K+). In this study, we developed artificial synaptic devices based on ion concentration-adjustable gels for emulating various synaptic plasticities under different K+ concentrations in body fluids. In addition to performing essential synaptic functions, potential applications in information processing and associative learning using short- and long-term plasticity realized using ion concentration-adjustable gels are presented. Artificial synaptic devices can be used for constructing an artificial neural pathway that controls artificial muscle reflex activities and can be used for image pattern recognition. All tests show a strong relationship with ion homeostasis. These devices could be applied to neuromorphic robots and human-machine interfaces.

Neuromorphic Nanoionics for Human-Machine Interaction: From Materials to Applications

Human-machine interaction (HMI) technology has undergone significant advancements in recent years, enabling seamless communication between humans and machines. Its expansion has extended into various emerging domains, including human healthcare, machine perception, and biointerfaces, thereby magnifying the demand for advanced intelligent technologies. Neuromorphic computing, a paradigm rooted in nanoionic devices that emulate the operations and architecture of the human brain, has emerged as a powerful tool for highly efficient information processing. This paper delivers a comprehensive review of recent developments in nanoionic device-based neuromorphic computing technologies and their pivotal role in shaping the next-generation of HMI. Through a detailed examination of fundamental mechanisms and behaviors, the paper explores the ability of nanoionic memristors and ion-gated transistors to emulate the intricate functions of neurons and synapses. Crucial performance metrics, such as reliability, energy efficiency, flexibility, and biocompatibility, are rigorously evaluated. Potential applications, challenges, and opportunities of using the neuromorphic computing technologies in emerging HMI technologies, are discussed and outlooked, shedding light on the fusion of humans with machines.

An in-sensor humidity computing system for contactless human-computer interaction

Being capable of processing large amounts of redundant data and decreasing power consumption, in-sensor computing approaches play significant roles in neuromorphic computing and are attracting increasing interest in perceptual information processing. Herein, we proposed a high performance humidity-sensitive memristor based on a Ti/graphene oxide (GO)/HfOx/Pt structure and verified its potential for application in remote health management and contactless human-machine interfaces. Since GO possesses abundant hydrophilic groups (carbonyl, epoxide, and hydroxyl), the memristor shows a high humidity sensitivity, fast response, and wide response range. By utilizing the proton-modulated redox reaction, humidity exposure to the memristor induces a dynamic change in the switching between high and low resistance states, ensuring essential synaptic learning functions, such as paired-pulse facilitation, spike number-dependent plasticity, and spike amplitude-dependent plasticity. More importantly, based on the humidity-induced salient features originating from the abundant hydrophilic functional groups in GO, we have implemented a noncontact human-machine interface utilizing the respiratory mode in humans, demonstrating the potential of promoting health monitoring applications and effectively blocking virus transmission. In addition, the high recognition accuracy of contactless handwriting in a 5 × 5 array artificial neural network was successfully achieved, which is attributed to the excellent emulated synaptic behaviors. This study provides a feasible method to develop an excellent humidity-sensitive memristor for constructing efficient in-sensor computing for application in health management and contactless human-computer interaction.

Toward a Brain-Neuromorphics Interface

Brain-computer interfaces (BCIs) that enable human-machine interaction have immense potential in restoring or augmenting human capabilities. Traditional BCIs are realized based on complementary metal-oxide-semiconductor (CMOS) technologies with complex, bulky, and low biocompatible circuits, and suffer with the low energy efficiency of the von Neumann architecture. The brain-neuromorphics interface (BNI) would offer a promising solution to advance the BCI technologies and shape the interactions with machineries. Neuromorphic devices and systems are able to provide substantial computation power with extremely high energy-efficiency by implementing in-materia computing such as in situ vector-matrix multiplication (VMM) and physical reservoir computing. Recent progresses on integrating neuromorphic components with sensing and/or actuating modules, give birth to the neuromorphic afferent nerve, efferent nerve, sensorimotor loop, and so on, which has advanced the technologies for future neurorobotics by achieving sophisticated sensorimotor capabilities as the biological system. With the development on the compact artificial spiking neuron and bioelectronic interfaces, the seamless communication between a BNI and a bioentity is reasonably expectable. In this review, the upcoming BNIs are profiled by introducing the brief history of neuromorphics, reviewing the recent progresses on related areas, and discussing the future advances and challenges that lie ahead.

Optoelectronic Synapses Based on MXene/Violet Phosphorus van der Waals Heterojunctions for Visual-Olfactory Crossmodal Perception

The crossmodal interaction of different senses, which is an important basis for learning and memory in the human brain, is highly desired to be mimicked at the device level for developing neuromorphic crossmodal perception, but related researches are scarce. Here, we demonstrate an optoelectronic synapse for vision-olfactory crossmodal perception based on MXene/violet phosphorus (VP) van der Waals heterojunctions. Benefiting from the efficient separation and transport of photogenerated carriers facilitated by conductive MXene, the photoelectric responsivity of VP is dramatically enhanced by 7 orders of magnitude, reaching up to 7.7 A W-1. Excited by ultraviolet light, multiple synaptic functions, including excitatory postsynaptic currents, paired-pulse facilitation, short/long-term plasticity and "learning-experience" behavior, were demonstrated with a low power consumption. Furthermore, the proposed optoelectronic synapse exhibits distinct synaptic behaviors in different gas environments, enabling it to simulate the interaction of visual and olfactory information for crossmodal perception. This work demonstrates the great potential of VP in optoelectronics and provides a promising platform for applications such as virtual reality and neurorobotics.

Hierarchies in Visual Pathway: Functions and Inspired Artificial Vision

The development of artificial intelligence has posed a challenge to machine vision based on conventional complementary metal-oxide semiconductor (CMOS) circuits owing to its high latency and inefficient power consumption originating from the data shuffling between memory and computation units. Gaining more insights into the function of every part of the visual pathway for visual perception can bring the capabilities of machine vision in terms of robustness and generality. Hardware acceleration of more energy-efficient and biorealistic artificial vision highly necessitates neuromorphic devices and circuits that are able to mimic the function of each part of the visual pathway. In this paper, we review the structure and function of the entire class of visual neurons from the retina to the primate visual cortex within reach (Chapter 2) are reviewed. Based on the extraction of biological principles, the recent hardware-implemented visual neurons located in different parts of the visual pathway are discussed in detail in Chapters 3 and 4. Furthermore, valuable applications of inspired artificial vision in different scenarios (Chapter 5) are provided. The functional description of the visual pathway and its inspired neuromorphic devices/circuits are expected to provide valuable insights for the design of next-generation artificial visual perception systems.

Toward grouped-reservoir computing: organic neuromorphic vertical transistor with distributed reservoir states for efficient recognition and prediction

Reservoir computing has attracted considerable attention due to its low training cost. However, existing neuromorphic hardware, focusing mainly on shallow-reservoir computing, faces challenges in providing adequate spatial and temporal scales characteristic for effective computing. Here, we report an ultra-short channel organic neuromorphic vertical transistor with distributed reservoir states. The carrier dynamics used to map signals are enriched by coupled multivariate physics mechanisms, while the vertical architecture employed greatly increases the feedback intensity of the device. Consequently, the device as a reservoir, effectively mapping sequential signals into distributed reservoir state space with 1152 reservoir states, and the range ratio of temporal and spatial characteristics can simultaneously reach 2640 and 650, respectively. The grouped-reservoir computing based on the device can simultaneously adapt to different spatiotemporal task, achieving recognition accuracy over 94% and prediction correlation over 95%. This work proposes a new strategy for developing high-performance reservoir computing networks.

Bionic Luminescent Skin as Ultrasensitive Temperature-Acoustic Sensor for Underwater Information Perception and Transmission

Bioinspired artificial luminescent skin (L-skin) integrated with multiple sensing functions significantly promotes the development of smart devices. It is considerably challenging to realize underwater sensing technologies. Here, a sharkskin-inspired Eu@HOF-TJ-1@TA L-skin (1) is prepared for both temperature and sound sensing. 1 is an ultrathin and flexible temperature sensor, in 298.15-358.15 K, exhibiting ultrahigh maximum relative sensitivity (97.669% K-1 ) and low minimum uncertainty (0.000 952 K). The temperature response mechanism is analyzed deeply. As a waterproofing acoustic sensor, 1 can monitor sound in both air and water with the greatest sound response frequencies of 400 and 300 Hz in air and water, respectively. The maximum sensitivities of 1 in air and water are 6 593 765.2 and 1 346 124.5 cps Pa-1 , respectively. The response times of 1 in air and water are as fast as 20 and 10 ms. The sound response processes of 1 in air and water are simulated by finite element simulation. Moreover, by using sharkskin-inspired 1, the actual water temperature can be monitored, and a series of water sound information can be recognized by using an artificial neural network. This work proposes a sharkskin-inspired L-skin for temperature and acoustic sensing and promotes the development of underwater sensing technology with high performances.

Optically driven intelligent computing with ZnO memristor

Artificial vision is crucial for most artificial intelligence applications. Conventional artificial visual systems have been facing challenges in terms of real-time information processing due to the physical separation of sensors, memories, and processors, which results in the production of a large amount of redundant data as well as the data conversion and transfer between these three components consuming most of the time and energy. Emergent optoelectronic memristors with the ability to realize integrated sensing-computing-memory (ISCM) are key candidates for solving such challenges and therefore attract increasing attention. At present, the memristive ISCM devices can only perform primary-level computing with external light signals due to the fact that only monotonic increase of memconductance upon light irradiation is achieved in most of these devices. Here, we propose an all-optically controlled memristive ISCM device based on a simple structure of Au/ZnO/Pt with the ZnO thin film sputtered at pure Ar atmosphere. This device can perform advanced computing tasks such as nonvolatile neuromorphic computing and complete Boolean logic functions only by light irradiation, owing to its ability to reversibly tune the memconductance with light. Moreover, the device shows excellent operation stability ascribed to a purely electronic memconductance tuning mechanism. Hence, this study is an important step towards the next generation of artificial visual systems.

The Framework and Memristive Circuit Design for Multisensory Mutual Associative Memory Networks

In this work, we propose a multisensory mutual associative memory networks framework and memristive circuit to mimic the ability of the biological brain to make associations of information received simultaneously. The circuit inspired by neural mechanisms of associative memory cells mainly consists of three modules: 1) the storage neurons module, which encodes external multimodal information into the firing rate of spikes; 2) the synapse module, which uses the nonvolatility memristor to achieve weight adjustment and associative learning; and 3) the retrieval neuron module, which feeds the retrieval signal output from each sensory pathway to other sensory pathways, so that achieve mutual association and retrieval between multiple modalities. Different from other one-to-one or many-to-one unidirectional associative memory work, this circuit achieves bidirectional association from multiple modalities to multiple modalities. In addition, we simulate the acquisition, extinction, recovery, transmission, and consolidation properties of associative memory. The circuit is applied to cross-modal association of image and audio recognition results, and episodic memory is simulated, where multiple images in a scene are intramodal associated. With power and area analysis, the circuit is validated as hardware-friendly. Further research to extend this work into large-scale associative memory networks, combined with visual-auditory-tactile-gustatory sensory sensors, is promising for application in intelligent robotic platforms to facilitate the development of neuromorphic systems and brain-like intelligence.

Intelligent Recognition Using Ultralight Multifunctional Nano-Layered Carbon Aerogel Sensors with Human-Like Tactile Perception

Humans can perceive our complex world through multi-sensory fusion. Under limited visual conditions, people can sense a variety of tactile signals to identify objects accurately and rapidly. However, replicating this unique capability in robots remains a significant challenge. Here, we present a new form of ultralight multifunctional tactile nano-layered carbon aerogel sensor that provides pressure, temperature, material recognition and 3D location capabilities, which is combined with multimodal supervised learning algorithms for object recognition. The sensor exhibits human-like pressure (0.04-100 kPa) and temperature (21.5-66.2 °C) detection, millisecond response times (11 ms), a pressure sensitivity of 92.22 kPa-1 and triboelectric durability of over 6000 cycles. The devised algorithm has universality and can accommodate a range of application scenarios. The tactile system can identify common foods in a kitchen scene with 94.63% accuracy and explore the topographic and geomorphic features of a Mars scene with 100% accuracy. This sensing approach empowers robots with versatile tactile perception to advance future society toward heightened sensing, recognition and intelligence.

Machine Learning-Enabled Intelligent Gesture Recognition and Communication System Using Printed Strain Sensors

Gesture contains abundant and complicated information in daily life; as a consequence, gesture recognition attracts a wide range of application prospects and academic values as an important way of achieving human-machine interactions (HMIs). Here, we report an intelligent system consisting of a smart glove made by printed CNT-graphene/PDMS strain sensors. The smart glove shows excellent fitness, comfort, and lightness for human hands. Inspired by machine learning strategies, several objects and gestures can be well classified and implemented by a customized artificial neural network. Several data sets of different sign language gestures and object-grabbing gestures were established, and the result shows that the intelligent system can achieve an average accuracy of 97% and up to 99.4% for a number of gesture groups. Moreover, a robot hand is connected to this system, which is able to react to the motion of human hands with certain gestures where simple sign communication is achieved. These features provide a feasible practical application scheme for gesture recognition in HMIs.

Mixed-Dimensional Nanoparticle-Nanowire Channels for Flexible Optoelectronic Artificial Synapse with Enhanced Photoelectric Response and Asymmetric Bidirectional Plasticity

A mixed-dimensional dual-channel synaptic transistor composed of inorganic nanoparticles and organic nanowires was fabricated to expand the photoelectric gain range. The device can actualize the sensitization features of the nociceptor and shows improved responsiveness to visible light. Under electrical pulses with different polarities, the apparatus exhibits reconfigurable asymmetric bidirectional plasticity. Moreover, the devices demonstrate good operational tolerance and mechanical stability, retaining more than 60% of their maximum responsiveness after 100 consecutive/bidirectional and 1000 flex/flat operations. The improved photoelectric response of the device endows a high image recognition accuracy of greater than 80%. Asymmetric bidirectional plasticity is used as punishment/reward in a psychological experiment to emulate the improvement of learning motivation and enables real-time forward and backward deflection (+7 and -25°) of artificial muscle. The mixed-dimensional optoelectronic artificial synapses with switchable behavior and electron/hole transport type have important prospects for neuromorphic processing and artificial somatosensory nerves.

Four levels of in-sensor computing in bionic olfaction: from discrete components to multi-modal integrations

Sensing and computing are two important ways in which humans attempt to perceive and understand the analog world through digital devices. Analog-to-digital converters (ADCs) discretize analog signals while the data bus transmits digital data between the components of a computer. With the increase in sensor nodes and the application of deep neural networks, the energy and time consumption limit the increment of data throughput. In-sensor computing is a computing paradigm that integrates sensing, storage, and processing in one device without ADCs and data transfer. According to the integration degree, herein, we summarize four levels of in-sensor computing in the field of artificial olfactory. In the first level, we show that different functions are conducted by using discrete components. Next, the data conversion and transfer are exempt within the in-memory computing architecture with necessary data encoding. Subsequently, in-sensor computing is integrated into a single device. Finally, multi-modal in-sensor computing is proposed to improve the quality and reliability of the classification results. At the end of this minireview, we provide an outlook on the use of metal nanoparticle devices to achieve such in-sensor computing for bionic olfaction.

An Artificial Olfactory System Based on a Chemi-Memristive Device

Technologies based on the fusion of gas sensors and neuromorphic computing to mimic the olfactory system have immense potential. However, the implementation of neuromorphic olfactory systems remains in a state of infancy because conventional gas sensors lack the necessary functions. Therefore, this study proposes a hysteretic "chemi-memristive gas sensor" based on oxygen vacancy chemi-memristive dynamics that differ from that of conventional gas sensors. After the memristive switching operation, the redox reaction with the external gas molecules is enhanced, resulting in the generation and elimination of oxygen vacancies that induce rapid current changes. In addition, the pre-generated oxygen vacancies enhance the post-sensing properties. Therefore, fast responses, short recovery times, and hysteretic gas response are achieved by the proposed sensor at room temperature. Based on the advantageous functionality of the sensor, device-level olfactory systems that can monitor the history of input gas stimuli are experimentally demonstrated as a potential application. Moreover, analog conductance modulation induced by oxidizing and reducing gases enables the conversion of external gas stimuli into synaptic weights and hence the realization of typical synaptic functionalities without an additional device or circuit. The proposed chemi-memristive device represents an advance in the bioinspired technology adopted in creating artificial intelligence systems.

A Bio-Inspired Chaos Sensor Model Based on the Perceptron Neural Network: Machine Learning Concept and Application for Computational Neuro-Science

The study presents a bio-inspired chaos sensor model based on the perceptron neural network for the estimation of entropy of spike train in neurodynamic systems. After training, the sensor on perceptron, having 50 neurons in the hidden layer and 1 neuron at the output, approximates the fuzzy entropy of a short time series with high accuracy, with a determination coefficient of R2~0.9. The Hindmarsh-Rose spike model was used to generate time series of spike intervals, and datasets for training and testing the perceptron. The selection of the hyperparameters of the perceptron model and the estimation of the sensor accuracy were performed using the K-block cross-validation method. Even for a hidden layer with one neuron, the model approximates the fuzzy entropy with good results and the metric R2~0.5 ÷ 0.8. In a simplified model with one neuron and equal weights in the first layer, the principle of approximation is based on the linear transformation of the average value of the time series into the entropy value. An example of using the chaos sensor on spike train of action potential recordings from the L5 dorsal rootlet of rat is provided. The bio-inspired chaos sensor model based on an ensemble of neurons is able to dynamically track the chaotic behavior of a spike signal and transmit this information to other parts of the neurodynamic model for further processing. The study will be useful for specialists in the field of computational neuroscience, and also to create humanoid and animal robots, and bio-robots with limited resources.

Memristor-based neural networks: a bridge from device to artificial intelligence

Since the beginning of the 21st century, there is no doubt that the importance of artificial intelligence has been highlighted in many fields, among which the memristor-based artificial neural network technology is expected to break through the limitation of von Neumann so as to realize the replication of the human brain by enabling strong parallel computing ability and efficient data processing and become an important way towards the next generation of artificial intelligence. A new type of nanodevice, namely memristor, which is based on the variability of its resistance value, not only has very important applications in nonvolatile information storage, but also presents obsessive progressiveness in highly integrated circuits, making it one of the most promising circuit components in the post-Moore era. In particular, memristors can effectively simulate neural synapses and build neural networks; thus, they can be applied for the preparation of various artificial intelligence systems. This study reviews the research progress of memristors in artificial neural networks in detail and highlights the structural advantages and frontier applications of neural networks based on memristors. Finally, some urgent problems and challenges in current research are summarized and corresponding solutions and future development trends are put forward.

A sensory memory processing system with multi-wavelength synaptic-polychromatic light emission for multi-modal information recognition

Realizing multi-modal information recognition tasks which can process external information efficiently and comprehensively is an urgent requirement in the field of artificial intelligence. However, it remains a challenge to achieve simple structure and high-performance multi-modal recognition demonstrations owing to the complex execution module and separation of memory processing based on the traditional complementary metal oxide semiconductor (CMOS) architecture. Here, we propose an efficient sensory memory processing system (SMPS), which can process sensory information and generate synapse-like and multi-wavelength light-emitting output, realizing diversified utilization of light in information processing and multi-modal information recognition. The SMPS exhibits strong robustness in information encoding/transmission and the capability of visible information display through the multi-level color responses, which can implement the multi-level pain warning process of organisms intuitively. Furthermore, different from the conventional multi-modal information processing system that requires independent and complex circuit modules, the proposed SMPS with unique optical multi-information parallel output can realize efficient multi-modal information recognition of dynamic step frequency and spatial positioning simultaneously with the accuracy of 99.5% and 98.2%, respectively. Therefore, the SMPS proposed in this work with simple component, flexible operation, strong robustness, and highly efficiency is promising for future sensory-neuromorphic photonic systems and interactive artificial intelligence.

Dynamic machine vision with retinomorphic photomemristor-reservoir computing

Dynamic machine vision requires recognizing the past and predicting the future of a moving object based on present vision. Current machine vision systems accomplish this by processing numerous image frames or using complex algorithms. Here, we report motion recognition and prediction in recurrent photomemristor networks. In our system, a retinomorphic photomemristor array, working as dynamic vision reservoir, embeds past motion frames as hidden states into the present frame through inherent dynamic memory. The informative present frame facilitates accurate recognition of past and prediction of future motions with machine learning algorithms. This in-sensor motion processing capability eliminates redundant data flows and promotes real-time perception of moving objects for dynamic machine vision.

Low-Voltage Stretchable Electroluminescent Loudspeakers with Synchronous Sound and Light Generation

Stretchable sound-in-displays, which can generate synchronous sound and light directly from the display without a separate speaker, allow immersive audio and visual perception even on curved surfaces. In stretchable sound-in-displays, alternating current electroluminescent (ACEL) devices have been used as light-emitting sources owing to their high brightness and stability. However, stretchable ACEL devices that use low dielectric constant (κ) materials require a high operating voltage for generating light and sound. Herein, we demonstrate a stretchable ACEL loudspeaker with a low operating voltage using stretchable high-κ dielectrics and strain-insensitive electrodes. Our device exhibits 87.7 cd/m² of luminance and 79.70 dB of sound pressure level at an operating voltage of 120 V and 10 kHz. As the next platform of wearable devices, the suggested ACEL loudspeaker exhibits high-quality synchronous light and sound generation performance even under various types of mechanical deformation, such as finger flexion and wrist bending.

Visual-Olfactory Synergistic Perception Based on Dual-Focus Imaging and a Bionic Learning Architecture

The synergistic interaction of vision and olfaction is critical for both natural and artificial intelligence systems to recognize and adapt to complex environments. However, current bioinspired systems with visual and olfactory sensations are mostly assembled with separate and heterogeneous sensors, inevitably leading to bulky systems and incompatible datasets. Here, we demonstrate on-chip integration of visual and olfactory sensations through a dual-focus imaging approach. By combining lens-based visual imaging and lensless colorimetric imaging, a target object and its odor fingerprint can be captured with a single complementary metal-oxide-semiconductor imager, and the obtained multimodal images are analyzed with a bionic learning architecture for information fusion and perception. To demonstrate the capability of this system, we adapted it to food detection and achieved 100% accuracy in identifying meat freshness and category with a 10 s sampling time. In addition to the highly integrated sensor design, our approach exhibits superior accuracy and efficiency in object recognition, providing a promising approach for robotic sensing and perception.

Neuromorphic Gustatory System with Salt-Taste Perception, Information Processing, and Excessive-Intake Warning Capabilities

Emulation of the process of a biological gustatory system could benefit the reconstruction of sense of taste. Here we demonstrate the first neuromorphic gustatory system that emulates the ability of taste perception, information processing, and excessive-intake warning functions. The system integrates a chitosan-derived ion-gel sensor, SnO₂ nanowire artificial synapses, and an effect-executive unit. The system accomplish perception and encoding behaviors for taste stimulation without using complex circuits and multivariate analysis, showing short response delay (<1 s), long taste memory duration (>2 h), and a wide perceptive concentration range (0.02-6 wt % salt solution). Especially, SnO₂ NW artificial synapses have extremely small response voltage (1 mV), exceeding the biological level by orders of magnitude, representing so-far the highest sensitivity record. This work provides a promising strategy to develop bioinspired and biointegrated electronics with the intention of mimicking and restoring the functions of biological sensory systems.

Emerging electrolyte-gated transistors for neuromorphic perception

With the rapid development of intelligent robotics, the Internet of Things, and smart sensor technologies, great enthusiasm has been devoted to developing next-generation intelligent systems for the emulation of advanced perception functions of humans. Neuromorphic devices, capable of emulating the learning, memory, analysis, and recognition functions of biological neural systems, offer solutions to intelligently process sensory information. As one of the most important neuromorphic devices, Electrolyte-gated transistors (EGTs) have shown great promise in implementing various vital neural functions and good compatibility with sensors. This review introduces the materials, operating principle, and performances of EGTs, followed by discussing the recent progress of EGTs for synapse and neuron emulation. Integrating EGTs with sensors that faithfully emulate diverse perception functions of humans such as tactile and visual perception is discussed. The challenges of EGTs for further development are given.

Progress and Perspectives of Single-Atom Catalysts for Gas Sensing

Single-atom catalysts (SACs) attract extensive attention in the field of heterogeneous catalysis in recent years due to the maximum atom utilization and unique physical and chemical properties. The gas sensing is actually a heterogeneous catalysis process but the SACs are new to this area. Although SACs show huge potential in gas sensing, the SACs gas sensing area currently is still at the infancy stage. This work critically reviews the recent advances and current status of single-atom gas sensing materials. General synthesis routes, characterization methods, and sensing performance indexes are introduced. At the end, the challenges and future prospects on SACs gas sensing are presented from the authors' perspectives. This work is anticipated to provide insights and guideline for the chemical sensing community.

Self-powered high-sensitivity all-in-one vertical tribo-transistor device for multi-sensing-memory-computing

Devices with sensing-memory-computing capability for the detection, recognition and memorization of real time sensory information could simplify data conversion, transmission, storage, and operations between different blocks in conventional chips, which are invaluable and sought-after to offer critical benefits of accomplishing diverse functions, simple design, and efficient computing simultaneously in the internet of things (IOT) era. Here, we develop a self-powered vertical tribo-transistor (VTT) based on MXenes for multi-sensing-memory-computing function and multi-task emotion recognition, which integrates triboelectric nanogenerator (TENG) and transistor in a single device with the simple configuration of vertical organic field effect transistor (VOFET). The tribo-potential is found to be able to tune ionic migration in insulating layer and Schottky barrier height at the MXene/semiconductor interface, and thus modulate the conductive channel between MXene and drain electrode. Meanwhile, the sensing sensitivity can be significantly improved by 711 times over the single TENG device, and the VTT exhibits excellent multi-sensing-memory-computing function. Importantly, based on this function, the multi-sensing integration and multi-model emotion recognition are constructed, which improves the emotion recognition accuracy up to 94.05% with reliability. This simple structure and self-powered VTT device exhibits high sensitivity, high efficiency and high accuracy, which provides application prospects in future human-mechanical interaction, IOT and high-level intelligence.

Bioinspired Artificial Motion Sensory System for Rotation Recognition and Rapid Self-Protection

As one of the most common synergies between the exteroceptors and proprioceptors, the synergy between visual and vestibule enables the human brain to judge the state of human motion, which is essential for motion recognition and human self-protection. Hence, in this work, an artificial motion sensory system (AMSS) based on artificial vestibule and visual is developed, which consists of a tribo-nanogenerator (TENG) as a vestibule that can sense rotation and synaptic transistor array as retina. The principle of temporal congruency has been successfully realized by multisensory input. In addition, pattern recognition results show that the accuracy of multisensory integration is more than 15% higher than that of single sensory. Moreover, due to the rotation recognition and visual recognition functions of AMSS, we realized multimodal information recognition including angles and numbers in the spiking correlated neural network (SCNN), and the accuracy rate reached 89.82%. Besides, the rapid self-protection of a human was successfully realized by AMSS in the case of simulated amusement rides, and the reaction time of multiple motion sensory integration is only one-third of that of a single vestibule. The development of AMSS based on the synergy of simulated vision and vestibule will show great potential in neural robot, artificial limbs, and soft electronics.

In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array

Detection and recognition of latent fingerprints play crucial roles in identification and security. However, the separation of sensor, memory, and processor in conventional ex-situ fingerprint recognition system seriously deteriorates the latency of decision-making and inevitably increases the overall computing power. In this work, a photoelectronic reservoir computing (RC) system, consisting of DUV photo-synapses and nonvolatile memristor array, is developed to detect and recognize the latent fingerprint with in-sensor and parallel in-memory computing. Through the Ga-rich design, we achieve amorphous GaO_x (a-GaO_x) photo-synapses with an enhanced persistent photoconductivity (PPC) effect. The PPC effect, which induces nonlinearly tunable conductivity, renders the a-GaO_x photo-synapses an ideal deep ultraviolet (DUV) photoelectronic reservoir, thus mapping the complex input vector into a dimensionality-reduced output vector. Connecting the reservoirs and a memristor array, we further construct an in-sensor RC system for latent fingerprint identification. The system maintains over 90% recognition accuracy for latent fingerprint within 15% stochastic noise level via the proposed dual-feature strategy. This work provides a subversive prototype system of DUV in-sensor RC for highly efficient recognition of latent fingerprints.

A calibratable sensory neuron based on epitaxial VO2 for spike-based neuromorphic multisensory system

Neuromorphic perception systems inspired by biology have tremendous potential in efficiently processing multi-sensory signals from the physical world, but a highly efficient hardware element capable of sensing and encoding multiple physical signals is still lacking. Here, we report a spike-based neuromorphic perception system consisting of calibratable artificial sensory neurons based on epitaxial VO₂, where the high crystalline quality of VO₂ leads to significantly improved cycle-to-cycle uniformity. A calibration resistor is introduced to optimize device-to-device consistency, and to adapt the VO₂ neuron to different sensors with varied resistance level, a scaling resistor is further incorporated, demonstrating cross-sensory neuromorphic perception component that can encode illuminance, temperature, pressure and curvature signals into spikes. These components are utilized to monitor the curvatures of fingers, thereby achieving hand gesture classification. This study addresses the fundamental cycle-to-cycle and device-to-device variation issues of sensory neurons, therefore promoting the construction of neuromorphic perception systems for e-skin and neurorobotics.

A Heterogeneously Integrated Spiking Neuron Array for Multimode-Fused Perception and Object Classification

Multimode-fused sensing in the somatosensory system helps people obtain comprehensive object properties and make accurate judgments. However, building such multisensory systems with conventional metal-oxide-semiconductor technology presents serious device integration and circuit complexity challenges. Here, a multimode-fused spiking neuron (MFSN) with a compact structure to achieve human-like multisensory perception is reported. The MFSN heterogeneously integrates a pressure sensor to process pressure and a NbO_x -based memristor to sense temperature. Using this MFSN, multisensory analog information can be fused into one spike train, showing excellent data compression and conversion capabilities. Moreover, both pressure and temperature information are distinguished from fused spikes by decoupling the output frequencies and amplitudes, supporting multimodal tactile perception. Then, a 3 × 3 MFSN array is fabricated, and the fused frequency patterns are fed into a spiking neural network for enhanced tactile pattern recognition. Finally, a larger MFSN array is simulated for classifying objects with different shapes, temperatures, and weights, validating the feasibility of the MFSNs for practical applications. The proof-of-concept MFSNs enable the building of multimodal sensory systems and contribute to the development of highly intelligent robotics.

An Artificial Reflex Arc That Perceives Afferent Visual and Tactile Information and Controls Efferent Muscular Actions

Neural perception and action-inspired electronics is becoming important for interactive human-machine interfaces and intelligent robots. A system that implements neuromorphic environmental information coding, synaptic signal processing, and motion control is desired. We report a neuroinspired artificial reflex arc that possesses visual and somatosensory dual afferent nerve paths and an efferent nerve path to control artificial muscles. A self-powered photoelectric synapse between the afferent and efferent nerves was used as the key information processor. The artificial reflex arc successfully responds to external visual and tactile information and controls the actions of artificial muscle in response to these external stimuli and thus emulates reflex activities through a full reflex arc. The visual and somatosensory information is encoded as impulse spikes, the frequency of which exhibited a sublinear dependence on the obstacle proximity or pressure stimuli. The artificial reflex arc suggests a promising strategy toward developing soft neurorobotic systems and prostheses.

Interface Defect Detection and Identification of Triboelectric Nanogenerators via Voltage Waveforms and Artificial Neural Network

To provide a robust working environment for TENGs, most TENGs are designed as sealed structures that isolate TENGs from the external environment, and thus their operating conditions cannot be directly monitored. Here, for the first time, we propose an artificial neural network for interface defect detection and identification of triboelectric nanogenerators via training voltage waveforms. First, interface defects of TENGs are classified and their causes are discussed in detail. Then we build a lightweight artificial neural network model which shows high sensitivity to voltage waveforms and low time complexity. The model takes 2.1 s for training one epoch, and the recognition rate of defect detection is 98.9% after 100 epochs. Meanwhile, the model successfully demonstrates the learning ability for low-resolution samples (100 × 75 pixels), which can identify six types of TENG defects, such as edge fracture, adhesion, and abnormal vibration, with a high recognition rate of 93.6%. This work provides a new strategy for the fault diagnosis and intelligent application of TENGs.

A Skin-Inspired Artificial Mechanoreceptor for Tactile Enhancement and Integration

Mechanoreceptors endow humans with the sense of touch by translating the external stimuli into coded spikes, inspiring the rise of artificial mechanoreceptor systems. However, to incorporate slow adaptive receptors-like pressure sensors with artificial neurons remains a challenge. Here we demonstrate an artificial mechanoreceptor by rationally integrating a polypyrrole-based resistive pressure sensor with a volatile NbO_x memristor, to mimic the tactile sensation and perception in natural skin, respectively. The artificial mechanoreceptor enables the tactile sensory coding by converting the external mechanical stimuli into strength-modulated electrical spikes. Also, tactile sensation enhancement is achieved by processing the spike frequency characteristics with the pulse coupled neural network. Furthermore, the artificial mechanoreceptor can integrate signals from parallel sensor channels and encode them into unified electrical spikes, resembling the coding of intensity in tactile neural processing. These results provide simple and efficient strategies for constructing future bio-inspired electronic systems.