1
|
Ding G, Li H, Zhao J, Zhou K, Zhai Y, Lv Z, Zhang M, Yan Y, Han ST, Zhou Y. Nanomaterials for Flexible Neuromorphics. Chem Rev 2024. [PMID: 39499851 DOI: 10.1021/acs.chemrev.4c00369] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2024]
Abstract
The quest to imbue machines with intelligence akin to that of humans, through the development of adaptable neuromorphic devices and the creation of artificial neural systems, has long stood as a pivotal goal in both scientific inquiry and industrial advancement. Recent advancements in flexible neuromorphic electronics primarily rely on nanomaterials and polymers owing to their inherent uniformity, superior mechanical and electrical capabilities, and versatile functionalities. However, this field is still in its nascent stage, necessitating continuous efforts in materials innovation and device/system design. Therefore, it is imperative to conduct an extensive and comprehensive analysis to summarize current progress. This review highlights the advancements and applications of flexible neuromorphics, involving inorganic nanomaterials (zero-/one-/two-dimensional, and heterostructure), carbon-based nanomaterials such as carbon nanotubes (CNTs) and graphene, and polymers. Additionally, a comprehensive comparison and summary of the structural compositions, design strategies, key performance, and significant applications of these devices are provided. Furthermore, the challenges and future directions pertaining to materials/devices/systems associated with flexible neuromorphics are also addressed. The aim of this review is to shed light on the rapidly growing field of flexible neuromorphics, attract experts from diverse disciplines (e.g., electronics, materials science, neurobiology), and foster further innovation for its accelerated development.
Collapse
Affiliation(s)
- Guanglong Ding
- State Key Laboratory of Radio Frequency Heterogeneous Integration, Shenzhen University, Shenzhen 518060, PR China
- College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, PR China
| | - Hang Li
- Institute for Advanced Study, Shenzhen University, Shenzhen 518060, PR China
| | - JiYu Zhao
- Institute for Advanced Study, Shenzhen University, Shenzhen 518060, PR China
- State Key Laboratory of Fine Chemicals, Frontiers Science Center for Smart Materials, Dalian University of Technology, Dalian 116024, China
| | - Kui Zhou
- Institute for Advanced Study, Shenzhen University, Shenzhen 518060, PR China
- The Construction Quality Supervision and Inspection Station of Zhuhai, Zhuhai 519000, PR China
| | - Yongbiao Zhai
- College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, PR China
| | - Ziyu Lv
- College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, PR China
| | - Meng Zhang
- State Key Laboratory of Radio Frequency Heterogeneous Integration, Shenzhen University, Shenzhen 518060, PR China
- College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, PR China
| | - Yan Yan
- State Key Laboratory of Radio Frequency Heterogeneous Integration, Shenzhen University, Shenzhen 518060, PR China
- College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, PR China
| | - Su-Ting Han
- Department of Applied Biology and Chemical Technology, The Hong Kong Polytechnic University, Hung Hom 999077, Hong Kong SAR PR China
| | - Ye Zhou
- State Key Laboratory of Radio Frequency Heterogeneous Integration, Shenzhen University, Shenzhen 518060, PR China
- Institute for Advanced Study, Shenzhen University, Shenzhen 518060, PR China
| |
Collapse
|
2
|
Fu C, Pei M, Cui H, Ke S, Zhu Y, Wan C, Wan Q. IGZO/PVP Composite Nanofiber Neuromorphic Transistors with Optoelectronic Synapse Emulation and Reservoir Computing. J Phys Chem Lett 2024; 15:9585-9592. [PMID: 39269773 DOI: 10.1021/acs.jpclett.4c02234] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/15/2024]
Abstract
Nanofiber neuromorphic transistors are regarded as promising candidates for mimicking brain-like learning and advancing high-performance computing. Composite nanofibers (CNFs) typically exhibit enhanced optoelectronic and mechanical properties. In this study, indium-gallium-zinc oxide (IGZO)/polyvinylpyrrolidone (PVP) CNFs were synthesized, and the neuromorphic transistor was integrated on both rigid and flexible substrates. The learning behavior, characterized by the transition from short-term plasticity (STP) to long-term plasticity, was achieved through photoelectric stimulation of the rigid neuromorphic transistor. The nonlinear STP was simulated by the flexible neuromorphic transistor through electrical pulses, matching effectively with a reservoir computing (RC) system. Hand gesture recognition with little energy consumption (49 pJ per reservoir state) and a maximum accuracy of 92.86% has been achieved by the RC system, proving the substantial potential of the IGZO/PVP CNF neuromorphic transistor for wearable intelligent processing tasks.
Collapse
Affiliation(s)
- Chuanyu Fu
- School of Electronic Science and Engineering, Nanjing University, Nanjing, Jiangsu 210093, China
- Yong jiang Laboratory (Y-LAB), Ningbo, Zhejiang 315202, China
| | - Mengjiao Pei
- School of Electronic Science and Engineering, Nanjing University, Nanjing, Jiangsu 210093, China
| | - Hangyuan Cui
- School of Electronic Science and Engineering, Nanjing University, Nanjing, Jiangsu 210093, China
| | - Shuo Ke
- School of Electronic Science and Engineering, Nanjing University, Nanjing, Jiangsu 210093, China
| | - Yixin Zhu
- School of Electronic Science and Engineering, Nanjing University, Nanjing, Jiangsu 210093, China
- Yong jiang Laboratory (Y-LAB), Ningbo, Zhejiang 315202, China
| | - Changjin Wan
- National Laboratory of Solid-State Microstructures, School of Electronic Science and Engineering, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing 210093, China
| | - Qing Wan
- Yong jiang Laboratory (Y-LAB), Ningbo, Zhejiang 315202, China
- National Laboratory of Solid-State Microstructures, School of Electronic Science and Engineering, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing 210093, China
| |
Collapse
|
3
|
Wan C, Pei M, Shi K, Cui H, Long H, Qiao L, Xing Q, Wan Q. Toward a Brain-Neuromorphics Interface. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2024; 36:e2311288. [PMID: 38339866 DOI: 10.1002/adma.202311288] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Revised: 01/17/2024] [Indexed: 02/12/2024]
Abstract
Brain-computer interfaces (BCIs) that enable human-machine interaction have immense potential in restoring or augmenting human capabilities. Traditional BCIs are realized based on complementary metal-oxide-semiconductor (CMOS) technologies with complex, bulky, and low biocompatible circuits, and suffer with the low energy efficiency of the von Neumann architecture. The brain-neuromorphics interface (BNI) would offer a promising solution to advance the BCI technologies and shape the interactions with machineries. Neuromorphic devices and systems are able to provide substantial computation power with extremely high energy-efficiency by implementing in-materia computing such as in situ vector-matrix multiplication (VMM) and physical reservoir computing. Recent progresses on integrating neuromorphic components with sensing and/or actuating modules, give birth to the neuromorphic afferent nerve, efferent nerve, sensorimotor loop, and so on, which has advanced the technologies for future neurorobotics by achieving sophisticated sensorimotor capabilities as the biological system. With the development on the compact artificial spiking neuron and bioelectronic interfaces, the seamless communication between a BNI and a bioentity is reasonably expectable. In this review, the upcoming BNIs are profiled by introducing the brief history of neuromorphics, reviewing the recent progresses on related areas, and discussing the future advances and challenges that lie ahead.
Collapse
Affiliation(s)
- Changjin Wan
- Yongjiang Laboratory (Y-LAB), Ningbo, Zhejiang, 315202, China
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
- Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo, 315201, China
| | - Mengjiao Pei
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Kailu Shi
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Hangyuan Cui
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Haotian Long
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Lesheng Qiao
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Qianye Xing
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Qing Wan
- Yongjiang Laboratory (Y-LAB), Ningbo, Zhejiang, 315202, China
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
- Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo, 315201, China
| |
Collapse
|
4
|
Hu J, Li H, Zhang Y, Zhou J, Zhao Y, Xu Y, Yu B. Reconfigurable Neuromorphic Computing with 2D Material Heterostructures for Versatile Neural Information Processing. NANO LETTERS 2024. [PMID: 39038296 DOI: 10.1021/acs.nanolett.4c02658] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/24/2024]
Abstract
Reconfigurable neuromorphic computing holds promise for advancing energy-efficient neural network implementation and functional versatility. Previous work has focused on emulating specific neural functions rather than an integrated approach. We propose an all two-dimensional (2D) material-based heterostructure capable of performing multiple neuromorphic operations by reconfiguring output terminals in response to stimuli. Specifically, our device can synergistically emulate the key neural elements of the synapse, neuron, and dendrite, which play important and interrelated roles in information processing. Dendrites, the branches that receive and transmit presynaptic action potentials, possess the ability to nonlinearly integrate and filter incoming signals. The proposed heterostructure allows reconfiguration between different operation modes, demonstrating its potential for diverse computing tasks. As a proof of concept, we show that the device can perform basic Boolean logic functions. This highlights its applicability to complex neural-network-based information processing problems. Our integrated neuromorphic approach may advance the development of versatile, low-power neuromorphic hardware.
Collapse
Affiliation(s)
- Jiayang Hu
- College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China 311200
- ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, Zhejiang, China 311200
| | - Hanxi Li
- College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China 311200
- ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, Zhejiang, China 311200
| | - Yishu Zhang
- College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China 311200
- ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, Zhejiang, China 311200
| | - Jiachao Zhou
- College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China 311200
- ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, Zhejiang, China 311200
| | - Yuda Zhao
- College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China 311200
- ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, Zhejiang, China 311200
| | - Yang Xu
- College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China 311200
- ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, Zhejiang, China 311200
| | - Bin Yu
- College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China 311200
- ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, Zhejiang, China 311200
| |
Collapse
|
5
|
Fernández JG, Keemink S, van Gerven M. Gradient-free training of recurrent neural networks using random perturbations. Front Neurosci 2024; 18:1439155. [PMID: 39050673 PMCID: PMC11267880 DOI: 10.3389/fnins.2024.1439155] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2024] [Accepted: 06/25/2024] [Indexed: 07/27/2024] Open
Abstract
Recurrent neural networks (RNNs) hold immense potential for computations due to their Turing completeness and sequential processing capabilities, yet existing methods for their training encounter efficiency challenges. Backpropagation through time (BPTT), the prevailing method, extends the backpropagation (BP) algorithm by unrolling the RNN over time. However, this approach suffers from significant drawbacks, including the need to interleave forward and backward phases and store exact gradient information. Furthermore, BPTT has been shown to struggle to propagate gradient information for long sequences, leading to vanishing gradients. An alternative strategy to using gradient-based methods like BPTT involves stochastically approximating gradients through perturbation-based methods. This learning approach is exceptionally simple, necessitating only forward passes in the network and a global reinforcement signal as feedback. Despite its simplicity, the random nature of its updates typically leads to inefficient optimization, limiting its effectiveness in training neural networks. In this study, we present a new approach to perturbation-based learning in RNNs whose performance is competitive with BPTT, while maintaining the inherent advantages over gradient-based learning. To this end, we extend the recently introduced activity-based node perturbation (ANP) method to operate in the time domain, leading to more efficient learning and generalization. We subsequently conduct a range of experiments to validate our approach. Our results show similar performance, convergence time and scalability when compared to BPTT, strongly outperforming standard node perturbation and weight perturbation methods. These findings suggest that perturbation-based learning methods offer a versatile alternative to gradient-based methods for training RNNs which can be ideally suited for neuromorphic computing applications.
Collapse
Affiliation(s)
- Jesús García Fernández
- Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | | | | |
Collapse
|
6
|
Krauhausen I, Griggs S, McCulloch I, den Toonder JMJ, Gkoupidenis P, van de Burgt Y. Bio-inspired multimodal learning with organic neuromorphic electronics for behavioral conditioning in robotics. Nat Commun 2024; 15:4765. [PMID: 38834541 DOI: 10.1038/s41467-024-48881-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 05/13/2024] [Indexed: 06/06/2024] Open
Abstract
Biological systems interact directly with the environment and learn by receiving multimodal feedback via sensory stimuli that shape the formation of internal neuronal representations. Drawing inspiration from biological concepts such as exploration and sensory processing that eventually lead to behavioral conditioning, we present a robotic system handling objects through multimodal learning. A small-scale organic neuromorphic circuit locally integrates and adaptively processes multimodal sensory stimuli, enabling the robot to interact intelligently with its surroundings. The real-time handling of sensory stimuli via low-voltage organic neuromorphic devices with synaptic functionality forms multimodal associative connections that lead to behavioral conditioning, and thus the robot learns to avoid potentially dangerous objects. This work demonstrates that adaptive neuro-inspired circuitry with multifunctional organic materials, can accommodate locally efficient bio-inspired learning for advancing intelligent robotics.
Collapse
Affiliation(s)
- Imke Krauhausen
- Institute for Complex Molecular Systems, Eindhoven University of Technology, Eindhoven, The Netherlands
- Microsystems, Department of Mechanical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
- Max Planck Institute for Polymer Research, Mainz, Germany
| | - Sophie Griggs
- Department of Chemistry, University of Oxford, Oxford, UK
| | - Iain McCulloch
- Department of Chemistry, University of Oxford, Oxford, UK
| | - Jaap M J den Toonder
- Institute for Complex Molecular Systems, Eindhoven University of Technology, Eindhoven, The Netherlands
- Microsystems, Department of Mechanical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
| | | | - Yoeri van de Burgt
- Institute for Complex Molecular Systems, Eindhoven University of Technology, Eindhoven, The Netherlands.
- Microsystems, Department of Mechanical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands.
| |
Collapse
|
7
|
Yang Y, Zhu F, Zhang X, Chen P, Wang Y, Zhu J, Ding Y, Cheng L, Li C, Jiang H, Wang Z, Lin P, Shi T, Wang M, Liu Q, Xu N, Liu M. Firing feature-driven neural circuits with scalable memristive neurons for robotic obstacle avoidance. Nat Commun 2024; 15:4318. [PMID: 38773067 PMCID: PMC11109161 DOI: 10.1038/s41467-024-48399-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Accepted: 04/30/2024] [Indexed: 05/23/2024] Open
Abstract
Neural circuits with specific structures and diverse neuronal firing features are the foundation for supporting intelligent tasks in biology and are regarded as the driver for catalyzing next-generation artificial intelligence. Emulating neural circuits in hardware underpins engineering highly efficient neuromorphic chips, however, implementing a firing features-driven functional neural circuit is still an open question. In this work, inspired by avoidance neural circuits of crickets, we construct a spiking feature-driven sensorimotor control neural circuit consisting of three memristive Hodgkin-Huxley neurons. The ascending neurons exhibit mixed tonic spiking and bursting features, which are used for encoding sensing input. Additionally, we innovatively introduce a selective communication scheme in biology to decode mixed firing features using two descending neurons. We proceed to integrate such a neural circuit with a robot for avoidance control and achieve lower latency than conventional platforms. These results provide a foundation for implementing real brain-like systems driven by firing features with memristive neurons and put constructing high-order intelligent machines on the agenda.
Collapse
Affiliation(s)
- Yue Yang
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Fangduo Zhu
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Xumeng Zhang
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China.
| | - Pei Chen
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Yongzhou Wang
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Jiaxue Zhu
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Yanting Ding
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Lingli Cheng
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Chao Li
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Hao Jiang
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Zhongrui Wang
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, 999077, China
| | - Peng Lin
- College of Computer Science and Technology, Zhejiang University, Zhejiang, 310027, China
| | - Tuo Shi
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Ming Wang
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Qi Liu
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China.
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China.
| | - Ningsheng Xu
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Ming Liu
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| |
Collapse
|
8
|
Paredes-Vallés F, Hagenaars JJ, Dupeyroux J, Stroobants S, Xu Y, de Croon GCHE. Fully neuromorphic vision and control for autonomous drone flight. Sci Robot 2024; 9:eadi0591. [PMID: 38748781 DOI: 10.1126/scirobotics.adi0591] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2023] [Accepted: 04/17/2024] [Indexed: 06/28/2024]
Abstract
Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions because of the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present a fully neuromorphic vision-to-control pipeline for controlling a flying drone. Specifically, we trained a spiking neural network that accepts raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28,800 neurons, maps incoming raw events to ego-motion estimates and was trained with self-supervised learning on real event data. The control part consists of a single decoding layer and was learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to-real transfer of the fully learned neuromorphic pipeline. The drone could accurately control its ego-motion, allowing for hovering, landing, and maneuvering sideways-even while yawing at the same time. The neuromorphic pipeline runs on board on Intel's Loihi neuromorphic processor with an execution frequency of 200 hertz, consuming 0.94 watt of idle power and a mere additional 7 to 12 milliwatts when running the network. These results illustrate the potential of neuromorphic sensing and processing for enabling insect-sized intelligent robots.
Collapse
Affiliation(s)
- F Paredes-Vallés
- Micro Air Vehicle Laboratory, Faculty of Aerospace Engineering, Delft University of Technology, Delft, Netherlands
| | - J J Hagenaars
- Micro Air Vehicle Laboratory, Faculty of Aerospace Engineering, Delft University of Technology, Delft, Netherlands
| | - J Dupeyroux
- Micro Air Vehicle Laboratory, Faculty of Aerospace Engineering, Delft University of Technology, Delft, Netherlands
| | - S Stroobants
- Micro Air Vehicle Laboratory, Faculty of Aerospace Engineering, Delft University of Technology, Delft, Netherlands
| | - Y Xu
- Micro Air Vehicle Laboratory, Faculty of Aerospace Engineering, Delft University of Technology, Delft, Netherlands
| | - G C H E de Croon
- Micro Air Vehicle Laboratory, Faculty of Aerospace Engineering, Delft University of Technology, Delft, Netherlands
| |
Collapse
|
9
|
Li F, Li D, Wang C, Liu G, Wang R, Ren H, Tang Y, Wang Y, Chen Y, Liang K, Huang Q, Sawan M, Qiu M, Wang H, Zhu B. An artificial visual neuron with multiplexed rate and time-to-first-spike coding. Nat Commun 2024; 15:3689. [PMID: 38693165 PMCID: PMC11063071 DOI: 10.1038/s41467-024-48103-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2023] [Accepted: 04/17/2024] [Indexed: 05/03/2024] Open
Abstract
Human visual neurons rely on event-driven, energy-efficient spikes for communication, while silicon image sensors do not. The energy-budget mismatch between biological systems and machine vision technology has inspired the development of artificial visual neurons for use in spiking neural network (SNN). However, the lack of multiplexed data coding schemes reduces the ability of artificial visual neurons in SNN to emulate the visual perception ability of biological systems. Here, we present an artificial visual spiking neuron that enables rate and temporal fusion (RTF) coding of external visual information. The artificial neuron can code visual information at different spiking frequencies (rate coding) and enables precise and energy-efficient time-to-first-spike (TTFS) coding. This multiplexed sensory coding scheme could improve the computing capability and efficacy of artificial visual neurons. A hardware-based SNN with the RTF coding scheme exhibits good consistency with real-world ground truth data and achieves highly accurate steering and speed predictions for self-driving vehicles in complex conditions. The multiplexed RTF coding scheme demonstrates the feasibility of developing highly efficient spike-based neuromorphic hardware.
Collapse
Affiliation(s)
- Fanfan Li
- School of Materials Science and Engineering, Zhejiang University, Hangzhou, China
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
| | - Dingwei Li
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
| | - Chuanqing Wang
- CenBRAIN Neurotech, School of Engineering, Westlake University, Hangzhou, China
| | - Guolei Liu
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
| | - Rui Wang
- Key Laboratory of Wide Band Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China
| | - Huihui Ren
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
| | - Yingjie Tang
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
| | - Yan Wang
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
| | - Yitong Chen
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
| | - Kun Liang
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
| | - Qi Huang
- Westlake Institute for Optoelectronics, Westlake University, Hangzhou, China
| | - Mohamad Sawan
- CenBRAIN Neurotech, School of Engineering, Westlake University, Hangzhou, China
- Westlake Institute for Optoelectronics, Westlake University, Hangzhou, China
- Institute of Advanced Technology, Westlake Institute for Advanced Study, Hangzhou, China
| | - Min Qiu
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China
- Westlake Institute for Optoelectronics, Westlake University, Hangzhou, China
- Institute of Advanced Technology, Westlake Institute for Advanced Study, Hangzhou, China
| | - Hong Wang
- Key Laboratory of Wide Band Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China.
| | - Bowen Zhu
- Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, China.
- Westlake Institute for Optoelectronics, Westlake University, Hangzhou, China.
- Institute of Advanced Technology, Westlake Institute for Advanced Study, Hangzhou, China.
| |
Collapse
|
10
|
Song J, Liu H, Zhao Z, Lin P, Yan F. Flexible Organic Transistors for Biosensing: Devices and Applications. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2024; 36:e2300034. [PMID: 36853083 DOI: 10.1002/adma.202300034] [Citation(s) in RCA: 19] [Impact Index Per Article: 19.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/02/2023] [Revised: 02/20/2023] [Indexed: 06/18/2023]
Abstract
Flexible and stretchable biosensors can offer seamless and conformable biological-electronic interfaces for continuously acquiring high-fidelity signals, permitting numerous emerging applications. Organic thin film transistors (OTFTs) are ideal transducers for flexible and stretchable biosensing due to their soft nature, inherent amplification function, biocompatibility, ease of functionalization, low cost, and device diversity. In consideration of the rapid advances in flexible-OTFT-based biosensors and their broad applications, herein, a timely and comprehensive review is provided. It starts with a detailed introduction to the features of various OTFTs including organic field-effect transistors and organic electrochemical transistors, and the functionalization strategies for biosensing, with a highlight on the seminal work and up-to-date achievements. Then, the applications of flexible-OTFT-based biosensors in wearable, implantable, and portable electronics, as well as neuromorphic biointerfaces are detailed. Subsequently, special attention is paid to emerging stretchable organic transistors including planar and fibrous devices. The routes to impart stretchability, including structural engineering and material engineering, are discussed, and the implementations of stretchable organic transistors in e-skin and smart textiles are included. Finally, the remaining challenges and the future opportunities in this field are summarized.
Collapse
Affiliation(s)
- Jiajun Song
- Department of Applied Physics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, 999077, P. R. China
| | - Hong Liu
- Department of Applied Physics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, 999077, P. R. China
| | - Zeyu Zhao
- Department of Applied Physics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, 999077, P. R. China
| | - Peng Lin
- Shenzhen Key Laboratory of Special Functional Materials and Guangdong Research Center for Interfacial Engineering of Functional Materials, College of Materials Science and Engineering, Shenzhen University, Shenzhen, 518060, P. R. China
| | - Feng Yan
- Department of Applied Physics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, 999077, P. R. China
- Research Institute of Intelligent Wearable Systems, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, 999077, P. R. China
| |
Collapse
|
11
|
Matrone GM, van Doremaele ERW, Surendran A, Laswick Z, Griggs S, Ye G, McCulloch I, Santoro F, Rivnay J, van de Burgt Y. A modular organic neuromorphic spiking circuit for retina-inspired sensory coding and neurotransmitter-mediated neural pathways. Nat Commun 2024; 15:2868. [PMID: 38570478 PMCID: PMC10991258 DOI: 10.1038/s41467-024-47226-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2023] [Accepted: 03/25/2024] [Indexed: 04/05/2024] Open
Abstract
Signal communication mechanisms within the human body rely on the transmission and modulation of action potentials. Replicating the interdependent functions of receptors, neurons and synapses with organic artificial neurons and biohybrid synapses is an essential first step towards merging neuromorphic circuits and biological systems, crucial for computing at the biological interface. However, most organic neuromorphic systems are based on simple circuits which exhibit limited adaptability to both external and internal biological cues, and are restricted to emulate only specific the functions of an individual neuron/synapse. Here, we present a modular neuromorphic system which combines organic spiking neurons and biohybrid synapses to replicate a neural pathway. The spiking neuron mimics the sensory coding function of afferent neurons from light stimuli, while the neuromodulatory activity of interneurons is emulated by neurotransmitters-mediated biohybrid synapses. Combining these functions, we create a modular connection between multiple neurons to establish a pre-processing retinal pathway primitive.
Collapse
Affiliation(s)
- Giovanni Maria Matrone
- Microsystems, Institute for Complex Molecular Systems, Eindhoven University of Technology, 5612AJ, Eindhoven, The Netherlands.
- Department of Biomedical Engineering, Northwestern University, Evanston, IL, 60208, USA.
| | - Eveline R W van Doremaele
- Microsystems, Institute for Complex Molecular Systems, Eindhoven University of Technology, 5612AJ, Eindhoven, The Netherlands
| | - Abhijith Surendran
- Department of Biomedical Engineering, Northwestern University, Evanston, IL, 60208, USA
| | - Zachary Laswick
- Department of Biomedical Engineering, Northwestern University, Evanston, IL, 60208, USA
| | - Sophie Griggs
- Department of Chemistry, Chemistry Research Laboratory, University of Oxford, Oxford, OX1 3TA, UK
| | - Gang Ye
- Center for Biomedical Optics and Photonics (CBOP) & College of Physics and Optoelectronic Engineering, Key Laboratory of Optoelectronic Devices and Systems, Shenzhen University, Shenzhen, 518060, PR China
| | - Iain McCulloch
- Department of Chemistry, Chemistry Research Laboratory, University of Oxford, Oxford, OX1 3TA, UK
- King Abdullah University of Science and Technology (KAUST), KAUST Solar Center (KSC), Thuwal, 23955-6900, Saudi Arabia
| | - Francesca Santoro
- Tissue Electronics, Istituto Italiano di Tecnologia, Naples, 80125, Italy
- Institute of Biological Information Processing IBI-3 Bioelectronics, Forschungszentrum Juelich, 52428, Juelich, Germany
- Neuroelectronic Interfaces, Faculty of Electrical Engineering and IT, RWTH Aachen, 52074, Aachen, Germany
| | - Jonathan Rivnay
- Department of Biomedical Engineering, Northwestern University, Evanston, IL, 60208, USA
| | - Yoeri van de Burgt
- Microsystems, Institute for Complex Molecular Systems, Eindhoven University of Technology, 5612AJ, Eindhoven, The Netherlands.
| |
Collapse
|
12
|
Xia H, Zhang Y, Rajabi N, Taleb F, Yang Q, Kragic D, Li Z. Shaping high-performance wearable robots for human motor and sensory reconstruction and enhancement. Nat Commun 2024; 15:1760. [PMID: 38409128 PMCID: PMC10897332 DOI: 10.1038/s41467-024-46249-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Accepted: 02/19/2024] [Indexed: 02/28/2024] Open
Abstract
Most wearable robots such as exoskeletons and prostheses can operate with dexterity, while wearers do not perceive them as part of their bodies. In this perspective, we contend that integrating environmental, physiological, and physical information through multi-modal fusion, incorporating human-in-the-loop control, utilizing neuromuscular interface, employing flexible electronics, and acquiring and processing human-robot information with biomechatronic chips, should all be leveraged towards building the next generation of wearable robots. These technologies could improve the embodiment of wearable robots. With optimizations in mechanical structure and clinical training, the next generation of wearable robots should better facilitate human motor and sensory reconstruction and enhancement.
Collapse
Affiliation(s)
- Haisheng Xia
- School of Mechanical Engineering, Tongji University, Shanghai, 201804, China
- Translational Research Center, Shanghai YangZhi Rehabilitation Hospital (Shanghai Sunshine Rehabilitation Center), Tongji University, Shanghai, 201619, China
- Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230026, China
| | - Yuchong Zhang
- Robotics, Perception and Learning Lab, EECS at KTH Royal Institute of Technology Stockholm, 114 17, Stockholm, Sweden
| | - Nona Rajabi
- Robotics, Perception and Learning Lab, EECS at KTH Royal Institute of Technology Stockholm, 114 17, Stockholm, Sweden
| | - Farzaneh Taleb
- Robotics, Perception and Learning Lab, EECS at KTH Royal Institute of Technology Stockholm, 114 17, Stockholm, Sweden
| | - Qunting Yang
- Department of Automation, University of Science and Technology of China, Hefei, 230026, China
| | - Danica Kragic
- Robotics, Perception and Learning Lab, EECS at KTH Royal Institute of Technology Stockholm, 114 17, Stockholm, Sweden
| | - Zhijun Li
- School of Mechanical Engineering, Tongji University, Shanghai, 201804, China.
- Translational Research Center, Shanghai YangZhi Rehabilitation Hospital (Shanghai Sunshine Rehabilitation Center), Tongji University, Shanghai, 201619, China.
- Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230026, China.
| |
Collapse
|
13
|
Qiu E, Zhang YH, Ventra MD, Schuller IK. Reconfigurable Cascaded Thermal Neuristors for Neuromorphic Computing. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2024; 36:e2306818. [PMID: 37770043 DOI: 10.1002/adma.202306818] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Revised: 09/25/2023] [Indexed: 10/03/2023]
Abstract
While the complementary metal-oxide semiconductor (CMOS) technology is the mainstream for the hardware implementation of neural networks, an alternative route is explored based on a new class of spiking oscillators called "thermal neuristors", which operate and interact solely via thermal processes. Utilizing the insulator-to-metal transition (IMT) in vanadium dioxide, a wide variety of reconfigurable electrical dynamics mirroring biological neurons is demonstrated. Notably, inhibitory functionality is achieved just in a single oxide device, and cascaded information flow is realized exclusively through thermal interactions. To elucidate the underlying mechanisms of the neuristors, a detailed theoretical model is developed, which accurately reflects the experimental results. This study establishes the foundation for scalable and energy-efficient thermal neural networks, fostering progress in brain-inspired computing.
Collapse
Affiliation(s)
- Erbin Qiu
- Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, 92093, USA
- Department of Physics, University of California San Diego, La Jolla, CA, 92093, USA
| | - Yuan-Hang Zhang
- Department of Physics, University of California San Diego, La Jolla, CA, 92093, USA
| | | | - Ivan K Schuller
- Department of Physics, University of California San Diego, La Jolla, CA, 92093, USA
| |
Collapse
|
14
|
Maity K, Dayen JF, Doudin B, Gumeniuk R, Kundys B. Single Wavelength Operating Neuromorphic Device Based on a Graphene-Ferroelectric Transistor. ACS APPLIED MATERIALS & INTERFACES 2023; 15:55948-55956. [PMID: 37983566 DOI: 10.1021/acsami.3c10010] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2023]
Abstract
As global data generation continues to rise, there is an increasing demand for revolutionary in-memory computing methodologies and efficient machine learning solutions. Despite recent progress in electrical and electro-optical simulations of machine learning devices, the all-optical nonthermal function remains challenging, with single wavelength operation still elusive. Here we report on an optical and monochromatic way of neuromorphic signal processing for brain-inspired functions, eliminating the need for electrical pulses. Multilevel synaptic potentiation-depression cycles are successfully achieved optically by leveraging photovoltaic charge generation and polarization within the photoferroelectric substrate interfaced with the graphene sensor. Furthermore, the demonstrated low-power prototype device is able to reproduce exact signal profile of brain tissues yet with more than 2 orders of magnitude faster response. The reported properties should trigger all-optical and low power artificial neuromorphic development based on photoferroelectric structures.
Collapse
Affiliation(s)
- Krishna Maity
- Université de Strasbourg, CNRS, Institut de Physique et Chimie des Matériaux de Strasbourg, UMR 7504, 23 rue du Loess, Strasbourg F-67000, France
| | - Jean-François Dayen
- Université de Strasbourg, CNRS, Institut de Physique et Chimie des Matériaux de Strasbourg, UMR 7504, 23 rue du Loess, Strasbourg F-67000, France
| | - Bernard Doudin
- Université de Strasbourg, CNRS, Institut de Physique et Chimie des Matériaux de Strasbourg, UMR 7504, 23 rue du Loess, Strasbourg F-67000, France
| | - Roman Gumeniuk
- Institut für Experimentelle Physik, TU Bergakademie Freiberg, Leipziger Str. 23, Freiberg 09596, Germany
| | - Bohdan Kundys
- Université de Strasbourg, CNRS, Institut de Physique et Chimie des Matériaux de Strasbourg, UMR 7504, 23 rue du Loess, Strasbourg F-67000, France
| |
Collapse
|
15
|
Xu M, Chen X, Guo Y, Wang Y, Qiu D, Du X, Cui Y, Wang X, Xiong J. Reconfigurable Neuromorphic Computing: Materials, Devices, and Integration. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2023; 35:e2301063. [PMID: 37285592 DOI: 10.1002/adma.202301063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Revised: 05/15/2023] [Indexed: 06/09/2023]
Abstract
Neuromorphic computing has been attracting ever-increasing attention due to superior energy efficiency, with great promise to promote the next wave of artificial general intelligence in the post-Moore era. Current approaches are, however, broadly designed for stationary and unitary assignments, thus encountering reluctant interconnections, power consumption, and data-intensive computing in that domain. Reconfigurable neuromorphic computing, an on-demand paradigm inspired by the inherent programmability of brain, can maximally reallocate finite resources to perform the proliferation of reproducibly brain-inspired functions, highlighting a disruptive framework for bridging the gap between different primitives. Although relevant research has flourished in diverse materials and devices with novel mechanisms and architectures, a precise overview remains blank and urgently desirable. Herein, the recent strides along this pursuit are systematically reviewed from material, device, and integration perspectives. At the material and device level, one comprehensively conclude the dominant mechanisms for reconfigurability, categorized into ion migration, carrier migration, phase transition, spintronics, and photonics. Integration-level developments for reconfigurable neuromorphic computing are also exhibited. Finally, a perspective on the future challenges for reconfigurable neuromorphic computing is discussed, definitely expanding its horizon for scientific communities.
Collapse
Affiliation(s)
- Minyi Xu
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Xinrui Chen
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Yehao Guo
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Yang Wang
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Dong Qiu
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Xinchuan Du
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Yi Cui
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Xianfu Wang
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Jie Xiong
- State Key Laboratory of Electronic Thin Film and Integrated Devices, School of Physics, University of Electronic Science and Technology of China, Chengdu, 610054, China
| |
Collapse
|
16
|
Baltieri M, Iizuka H, Witkowski O, Sinapayen L, Suzuki K. Hybrid Life: Integrating biological, artificial, and cognitive systems. WILEY INTERDISCIPLINARY REVIEWS. COGNITIVE SCIENCE 2023; 14:e1662. [PMID: 37403661 DOI: 10.1002/wcs.1662] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/09/2022] [Revised: 05/22/2023] [Accepted: 05/30/2023] [Indexed: 07/06/2023]
Abstract
Artificial life is a research field studying what processes and properties define life, based on a multidisciplinary approach spanning the physical, natural, and computational sciences. Artificial life aims to foster a comprehensive study of life beyond "life as we know it" and toward "life as it could be," with theoretical, synthetic, and empirical models of the fundamental properties of living systems. While still a relatively young field, artificial life has flourished as an environment for researchers with different backgrounds, welcoming ideas, and contributions from a wide range of subjects. Hybrid Life brings our attention to some of the most recent developments within the artificial life community, rooted in more traditional artificial life studies but looking at new challenges emerging from interactions with other fields. Hybrid Life aims to cover studies that can lead to an understanding, from first principles, of what systems are and how biological and artificial systems can interact and integrate to form new kinds of hybrid (living) systems, individuals, and societies. To do so, it focuses on three complementary perspectives: theories of systems and agents, hybrid augmentation, and hybrid interaction. Theories of systems and agents are used to define systems, how they differ (e.g., biological or artificial, autonomous, or nonautonomous), and how multiple systems relate in order to form new hybrid systems. Hybrid augmentation focuses on implementations of systems so tightly connected that they act as a single, integrated one. Hybrid interaction is centered around interactions within a heterogeneous group of distinct living and nonliving systems. After discussing some of the major sources of inspiration for these themes, we will focus on an overview of the works that appeared in Hybrid Life special sessions, hosted by the annual Artificial Life Conference between 2018 and 2022. This article is categorized under: Neuroscience > Cognition Philosophy > Artificial Intelligence Computer Science and Robotics > Robotics.
Collapse
Affiliation(s)
- Manuel Baltieri
- Araya Inc., Tokyo, Japan
- Department of Informatics, University of Sussex, Brighton, UK
| | - Hiroyuki Iizuka
- Faculty of Information Science and Technology, Hokkaido University, Sapporo, Japan
- Center for Human Nature, Artificial Intelligence and Neuroscience (CHAIN), Hokkaido University, Sapporo, Japan
| | - Olaf Witkowski
- Center for Human Nature, Artificial Intelligence and Neuroscience (CHAIN), Hokkaido University, Sapporo, Japan
- Cross Labs, Cross Compass, Kyoto, Japan
- College of Arts and Sciences, University of Tokyo, Tokyo, Japan
| | - Lana Sinapayen
- Sony Computer Science Laboratories, Kyoto, Japan
- National Institute for Basic Biology, Okazaki, Japan
| | - Keisuke Suzuki
- Center for Human Nature, Artificial Intelligence and Neuroscience (CHAIN), Hokkaido University, Sapporo, Japan
| |
Collapse
|
17
|
Pham MD, D’Angiulli A, Dehnavi MM, Chhabra R. From Brain Models to Robotic Embodied Cognition: How Does Biological Plausibility Inform Neuromorphic Systems? Brain Sci 2023; 13:1316. [PMID: 37759917 PMCID: PMC10526461 DOI: 10.3390/brainsci13091316] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Revised: 09/05/2023] [Accepted: 09/07/2023] [Indexed: 09/29/2023] Open
Abstract
We examine the challenging "marriage" between computational efficiency and biological plausibility-A crucial node in the domain of spiking neural networks at the intersection of neuroscience, artificial intelligence, and robotics. Through a transdisciplinary review, we retrace the historical and most recent constraining influences that these parallel fields have exerted on descriptive analysis of the brain, construction of predictive brain models, and ultimately, the embodiment of neural networks in an enacted robotic agent. We study models of Spiking Neural Networks (SNN) as the central means enabling autonomous and intelligent behaviors in biological systems. We then provide a critical comparison of the available hardware and software to emulate SNNs for investigating biological entities and their application on artificial systems. Neuromorphics is identified as a promising tool to embody SNNs in real physical systems and different neuromorphic chips are compared. The concepts required for describing SNNs are dissected and contextualized in the new no man's land between cognitive neuroscience and artificial intelligence. Although there are recent reviews on the application of neuromorphic computing in various modules of the guidance, navigation, and control of robotic systems, the focus of this paper is more on closing the cognition loop in SNN-embodied robotics. We argue that biologically viable spiking neuronal models used for electroencephalogram signals are excellent candidates for furthering our knowledge of the explainability of SNNs. We complete our survey by reviewing different robotic modules that can benefit from neuromorphic hardware, e.g., perception (with a focus on vision), localization, and cognition. We conclude that the tradeoff between symbolic computational power and biological plausibility of hardware can be best addressed by neuromorphics, whose presence in neurorobotics provides an accountable empirical testbench for investigating synthetic and natural embodied cognition. We argue this is where both theoretical and empirical future work should converge in multidisciplinary efforts involving neuroscience, artificial intelligence, and robotics.
Collapse
Affiliation(s)
- Martin Do Pham
- Department of Computer Science, University of Toronto, Toronto, ON M5S 1A1, Canada; (M.D.P.); (M.M.D.)
| | - Amedeo D’Angiulli
- Department of Neuroscience, Carleton University, Ottawa, ON K1S 5B6, Canada;
| | - Maryam Mehri Dehnavi
- Department of Computer Science, University of Toronto, Toronto, ON M5S 1A1, Canada; (M.D.P.); (M.M.D.)
| | - Robin Chhabra
- Department of Mechanical and Aerospace Engineering, Carleton University, Ottawa, ON K1S 5B6, Canada
| |
Collapse
|
18
|
Zhu L, Mangan M, Webb B. Neuromorphic sequence learning with an event camera on routes through vegetation. Sci Robot 2023; 8:eadg3679. [PMID: 37756384 DOI: 10.1126/scirobotics.adg3679] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2022] [Accepted: 08/29/2023] [Indexed: 09/29/2023]
Abstract
For many robotics applications, it is desirable to have relatively low-power and efficient onboard solutions. We took inspiration from insects, such as ants, that are capable of learning and following routes in complex natural environments using relatively constrained sensory and neural systems. Such capabilities are particularly relevant to applications such as agricultural robotics, where visual navigation through dense vegetation remains a challenging task. In this scenario, a route is likely to have high self-similarity and be subject to changing lighting conditions and motion over uneven terrain, and the effects of wind on leaves increase the variability of the input. We used a bioinspired event camera on a terrestrial robot to collect visual sequences along routes in natural outdoor environments and applied a neural algorithm for spatiotemporal memory that is closely based on a known neural circuit in the insect brain. We show that this method is plausible to support route recognition for visual navigation and more robust than SeqSLAM when evaluated on repeated runs on the same route or routes with small lateral offsets. By encoding memory in a spiking neural network running on a neuromorphic computer, our model can evaluate visual familiarity in real time from event camera footage.
Collapse
Affiliation(s)
- Le Zhu
- School of Informatics, University of Edinburgh, EH8 9AB Edinburgh, UK
| | - Michael Mangan
- Sheffield Robotics, Department of Computer Science, University of Sheffield, S1 4DP Sheffield, UK
| | - Barbara Webb
- School of Informatics, University of Edinburgh, EH8 9AB Edinburgh, UK
| |
Collapse
|
19
|
Zaharia C, Popescu V, Sandu F. Hardware-Software Partitioning for Real-Time Object Detection Using Dynamic Parameter Optimization. SENSORS (BASEL, SWITZERLAND) 2023; 23:4894. [PMID: 37430806 DOI: 10.3390/s23104894] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/24/2023] [Revised: 05/12/2023] [Accepted: 05/16/2023] [Indexed: 07/12/2023]
Abstract
Computer vision algorithms implementations, especially for real-time applications, are present in a variety of devices that we are currently using (from smartphones or automotive applications to monitoring/security applications) and pose specific challenges, memory bandwidth or energy consumption (e.g., for mobility) being the most notable ones. This paper aims at providing a solution to improve the overall quality of real-time object detection computer vision algorithms using a hybrid hardware-software implementation. To this end, we explore the methods for a proper allocation of algorithm components towards hardware (as IP Cores) and the interfacing between hardware and software. Addressing specific design constraints, the relationship between the above components allows embedded artificial intelligence to select the operating hardware blocks (IP cores)-in the configuration phase-and to dynamically change the parameters of the aggregated hardware resources-in the instantiation phase, similar to the concretization of a class into a software object. The conclusions show the benefits of using hybrid hardware-software implementations, as well as major gains from using IP Cores, managed by artificial intelligence, for an object detection use-case, implemented on a FPGA demonstrator built around a Xilinx Zynq-7000 SoC Mini-ITX sub-system.
Collapse
Affiliation(s)
- Corneliu Zaharia
- Department of Electronics and Computers, Transilvania University, Bdul Eroilor 29, 500068 Brașov, Romania
| | - Vlad Popescu
- Department of Electronics and Computers, Transilvania University, Bdul Eroilor 29, 500068 Brașov, Romania
| | - Florin Sandu
- Department of Electronics and Computers, Transilvania University, Bdul Eroilor 29, 500068 Brașov, Romania
| |
Collapse
|
20
|
Yu F, Wu Y, Ma S, Xu M, Li H, Qu H, Song C, Wang T, Zhao R, Shi L. Brain-inspired multimodal hybrid neural network for robot place recognition. Sci Robot 2023; 8:eabm6996. [PMID: 37163608 DOI: 10.1126/scirobotics.abm6996] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Abstract
Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi-neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.
Collapse
Affiliation(s)
- Fangwen Yu
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Yujie Wu
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
- Institute of Theoretical Computer Science, Graz University of Technology, Graz, Austria
| | - Songchen Ma
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Mingkun Xu
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Hongyi Li
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Huanyu Qu
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Chenhang Song
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Taoyi Wang
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Rong Zhao
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China
| | - Luping Shi
- Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China
- IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China
- THU-CET HIK Joint Research Center for Brain-Inspired Computing, Tsinghua University, Beijing 100084, China
| |
Collapse
|
21
|
Pietrzak P, Szczęsny S, Huderek D, Przyborowski Ł. Overview of Spiking Neural Network Learning Approaches and Their Computational Complexities. SENSORS (BASEL, SWITZERLAND) 2023; 23:3037. [PMID: 36991750 PMCID: PMC10053242 DOI: 10.3390/s23063037] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Revised: 03/08/2023] [Accepted: 03/09/2023] [Indexed: 06/19/2023]
Abstract
Spiking neural networks (SNNs) are subjects of a topic that is gaining more and more interest nowadays. They more closely resemble actual neural networks in the brain than their second-generation counterparts, artificial neural networks (ANNs). SNNs have the potential to be more energy efficient than ANNs on event-driven neuromorphic hardware. This can yield drastic maintenance cost reduction for neural network models, as the energy consumption would be much lower in comparison to regular deep learning models hosted in the cloud today. However, such hardware is still not yet widely available. On standard computer architectures consisting mainly of central processing units (CPUs) and graphics processing units (GPUs) ANNs, due to simpler models of neurons and simpler models of connections between neurons, have the upper hand in terms of execution speed. In general, they also win in terms of learning algorithms, as SNNs do not reach the same levels of performance as their second-generation counterparts in typical machine learning benchmark tasks, such as classification. In this paper, we review existing learning algorithms for spiking neural networks, divide them into categories by type, and assess their computational complexity.
Collapse
|
22
|
Pang J, Peng S, Hou C, Zhao H, Fan Y, Ye C, Zhang N, Wang T, Cao Y, Zhou W, Sun D, Wang K, Rümmeli MH, Liu H, Cuniberti G. Applications of Graphene in Five Senses, Nervous System, and Artificial Muscles. ACS Sens 2023; 8:482-514. [PMID: 36656873 DOI: 10.1021/acssensors.2c02790] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/20/2023]
Abstract
Graphene remains of great interest in biomedical applications because of biocompatibility. Diseases relating to human senses interfere with life satisfaction and happiness. Therefore, the restoration by artificial organs or sensory devices may bring a bright future by the recovery of senses in patients. In this review, we update the most recent progress in graphene based sensors for mimicking human senses such as artificial retina for image sensors, artificial eardrums, gas sensors, chemical sensors, and tactile sensors. The brain-like processors are discussed based on conventional transistors as well as memristor related neuromorphic computing. The brain-machine interface is introduced for providing a single pathway. Besides, the artificial muscles based on graphene are summarized in the means of actuators in order to react to the physical world. Future opportunities remain for elevating the performances of human-like sensors and their clinical applications.
Collapse
Affiliation(s)
- Jinbo Pang
- Collaborative Innovation Center of Technology and Equipment for Biological Diagnosis and Therapy in Universities of Shandong, Institute for Advanced Interdisciplinary Research (iAIR), University of Jinan, Jinan 250022, China
| | - Songang Peng
- High-Frequency High-Voltage Device and Integrated Circuits R&D Center and Key Laboratory of Microelectronic Devices & Integrated Technology, Institute of Microelectronics, Chinese Academy of Sciences, Beijing, 100029, China
| | - Chongyang Hou
- Collaborative Innovation Center of Technology and Equipment for Biological Diagnosis and Therapy in Universities of Shandong, Institute for Advanced Interdisciplinary Research (iAIR), University of Jinan, Jinan 250022, China
| | - Hongbin Zhao
- State Key Laboratory of Advanced Materials for Smart Sensing, GRINM Group Co. Ltd., Xinwai Street 2, Beijing 100088, People's Republic of China
| | - Yingju Fan
- School of Chemistry and Chemical Engineering, University of Jinan, Shandong, Jinan 250022, China
| | - Chen Ye
- School of Chemistry and Chemical Engineering, University of Jinan, Shandong, Jinan 250022, China
| | - Nuo Zhang
- School of Chemistry and Chemical Engineering, University of Jinan, Shandong, Jinan 250022, China
| | - Ting Wang
- State Key Laboratory of Biobased Material and Green Papermaking and People's Republic of China School of Bioengineering, Qilu University of Technology, Shandong Academy of Sciences, No. 3501 Daxue Road, Jinan 250353, People's Republic of China
| | - Yu Cao
- Key Laboratory of Modern Power System Simulation and Control & Renewable Energy Technology (Ministry of Education) and School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, China
| | - Weijia Zhou
- Collaborative Innovation Center of Technology and Equipment for Biological Diagnosis and Therapy in Universities of Shandong, Institute for Advanced Interdisciplinary Research (iAIR), University of Jinan, Jinan 250022, China
| | - Ding Sun
- School of Electrical and Computer Engineering, Jilin Jianzhu University, Changchun 130118, P. R. China
| | - Kai Wang
- School of Electrical Engineering, Weihai Innovation Research Institute, Qingdao University, Qingdao 266000, China
| | - Mark H Rümmeli
- Leibniz Institute for Solid State and Materials Research Dresden, Dresden, D-01171, Germany.,College of Energy, Soochow Institute for Energy and Materials Innovations, and Key Laboratory of Advanced Carbon Materials and Wearable Energy Technologies of Jiangsu Province, Soochow University, Suzhou 215006, China.,Centre of Polymer and Carbon Materials, Polish Academy of Sciences, M. Curie Sklodowskiej 34, Zabrze 41-819, Poland.,Institute for Complex Materials, IFW Dresden, 20 Helmholtz Strasse, Dresden 01069, Germany.,Center for Energy and Environmental Technologies, VŠB-Technical University of Ostrava, 17. Listopadu 15, Ostrava 708 33, Czech Republic
| | - Hong Liu
- Collaborative Innovation Center of Technology and Equipment for Biological Diagnosis and Therapy in Universities of Shandong, Institute for Advanced Interdisciplinary Research (iAIR), University of Jinan, Jinan 250022, China.,State Key Laboratory of Crystal Materials, Center of Bio & Micro/Nano Functional Materials, Shandong University, 27 Shandanan Road, Jinan 250100, China
| | - Gianaurelio Cuniberti
- Institute for Materials Science and Max Bergmann Center of Biomaterials and Center for Advancing Electronics Dresden, Technische Universität Dresden, Dresden 01069, Germany
| |
Collapse
|
23
|
Ren Z, Shao Y. Future bio-inspired robots require delicate structures. Front Robot AI 2022; 9:1073329. [PMID: 36618011 PMCID: PMC9811312 DOI: 10.3389/frobt.2022.1073329] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2022] [Accepted: 12/12/2022] [Indexed: 12/24/2022] Open
Affiliation(s)
- Ziyu Ren
- Physical Intelligence Department, Max Planck Institute for Intelligent Systems, Stuttgart, Germany,*Correspondence: Ziyu Ren, ; Yuxiu Shao,
| | - Yuxiu Shao
- Laboratoire de Neurosciences Cognitives et Computationnelles, INSERM U960, Ecole Normale Superieure—PSL Research University, Paris, France,*Correspondence: Ziyu Ren, ; Yuxiu Shao,
| |
Collapse
|
24
|
Juárez-Lora A, García-Sebastián LM, Ponce-Ponce VH, Rubio-Espino E, Molina-Lozano H, Sossa H. Implementation of Kalman Filtering with Spiking Neural Networks. SENSORS (BASEL, SWITZERLAND) 2022; 22:8845. [PMID: 36433442 PMCID: PMC9695172 DOI: 10.3390/s22228845] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/28/2022] [Revised: 11/11/2022] [Accepted: 11/13/2022] [Indexed: 06/16/2023]
Abstract
A Kalman filter can be used to fill space-state reconstruction dynamics based on knowledge of a system and partial measurements. However, its performance relies on accurate modeling of the system dynamics and a proper characterization of the uncertainties, which can be hard to obtain in real-life scenarios. In this work, we explore how the values of a Kalman gain matrix can be estimated by using spiking neural networks through a combination of biologically plausible neuron models with spike-time-dependent plasticity learning algorithms. The performance of proposed neural architecture is verified with simulations of some representative nonlinear systems, which show promising results. This approach traces a path for its implementation in neuromorphic analog hardware that can learn and reconstruct partial and changing dynamics of a system without the massive power consumption that is typically needed in a Von Neumann-based computer architecture.
Collapse
|