1
|
Zhu Z, Yuan S, Yang Q, Jiang H, Zheng F, Lu J, Sun Q. Autonomous Scanning Tunneling Microscopy Imaging via Deep Learning. J Am Chem Soc 2024. [PMID: 39382312 DOI: 10.1021/jacs.4c11674] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/10/2024]
Abstract
Scanning tunneling microscopy (STM) is a powerful technique that provides the ability to manipulate and characterize individual atoms and molecules with atomic-level precision. However, the processes of scanning samples, operating the probe, and analyzing data are typically labor-intensive and subjective. Deep learning (DL) techniques have shown immense potential in automating complex tasks and solving high-dimensional problems. In this study, we developed an autonomous STM framework powered by DL to enable autonomous operations of the STM without human interventions. Our framework employs a convolutional neural network (CNN) for real-time evaluation of STM image quality, a U-net model for identifying bare surfaces, and a deep Q-learning network (DQN) agent for autonomous probe conditioning. Additionally, we integrated an object recognition model for the automated recognition of different adsorbates. This autonomous framework enables the acquisition of space-averaging information using STM techniques without compromising the high-resolution molecular imaging. We achieved measuring an area of approximately 1.9 μm2 within 48 h of continuous measurement and automatedly generated the statistics on the molecular species present within the mesoscopic area. We demonstrate the high robustness of the framework by conducting measurements at the liquid nitrogen temperature (∼78 K). We envision that the integration of DL techniques and high-resolution microscopy will not only extend the functionality and capability of scanning probe microscopes but also accelerate the understanding and discovery of new materials.
Collapse
Affiliation(s)
- Zhiwen Zhu
- Materials Genome Institute, Shanghai Engineering Research Center for Integrated Circuits and Advanced Display Materials, Shanghai University, Shanghai 200444, China
| | - Shaoxuan Yuan
- Materials Genome Institute, Shanghai University, Shanghai 200444, China
| | - Quan Yang
- Materials Genome Institute, Shanghai University, Shanghai 200444, China
| | - Hao Jiang
- Materials Genome Institute, Shanghai University, Shanghai 200444, China
| | - Fengru Zheng
- Materials Genome Institute, Shanghai University, Shanghai 200444, China
| | - Jiayi Lu
- Materials Genome Institute, Shanghai University, Shanghai 200444, China
| | - Qiang Sun
- Materials Genome Institute, Shanghai Engineering Research Center for Integrated Circuits and Advanced Display Materials, Shanghai University, Shanghai 200444, China
| |
Collapse
|
2
|
Jiao L, Wang Y, Liu X, Li L, Liu F, Ma W, Guo Y, Chen P, Yang S, Hou B. Causal Inference Meets Deep Learning: A Comprehensive Survey. RESEARCH (WASHINGTON, D.C.) 2024; 7:0467. [PMID: 39257419 PMCID: PMC11384545 DOI: 10.34133/research.0467] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/07/2024] [Accepted: 08/11/2024] [Indexed: 09/12/2024]
Abstract
Deep learning relies on learning from extensive data to generate prediction results. This approach may inadvertently capture spurious correlations within the data, leading to models that lack interpretability and robustness. Researchers have developed more profound and stable causal inference methods based on cognitive neuroscience. By replacing the correlation model with a stable and interpretable causal model, it is possible to mitigate the misleading nature of spurious correlations and overcome the limitations of model calculations. In this survey, we provide a comprehensive and structured review of causal inference methods in deep learning. Brain-like inference ideas are discussed from a brain-inspired perspective, and the basic concepts of causal learning are introduced. The article describes the integration of causal inference with traditional deep learning algorithms and illustrates its application to large model tasks as well as specific modalities in deep learning. The current limitations of causal inference and future research directions are discussed. Moreover, the commonly used benchmark datasets and the corresponding download links are summarized.
Collapse
Affiliation(s)
- Licheng Jiao
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Yuhan Wang
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Xu Liu
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Lingling Li
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Fang Liu
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Wenping Ma
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Yuwei Guo
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Puhua Chen
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Shuyuan Yang
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| | - Biao Hou
- The School of Artificial Intelligence, Xidian University, Xi'an, China
| |
Collapse
|
3
|
Zhang T, Liu Z, Yi J, Wu S, Pu Z, Zhao Y. Multiexperience-Assisted Efficient Multiagent Reinforcement Learning. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:12678-12692. [PMID: 37037246 DOI: 10.1109/tnnls.2023.3264275] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Recently, multiagent reinforcement learning (MARL) has shown great potential for learning cooperative policies in multiagent systems (MASs). However, a noticeable drawback of current MARL is the low sample efficiency, which causes a huge amount of interactions with environment. Such amount of interactions greatly hinders the real-world application of MARL. Fortunately, effectively incorporating experience knowledge can assist MARL to quickly find effective solutions, which can significantly alleviate the drawback. In this article, a novel multiexperience-assisted reinforcement learning (MEARL) method is proposed to improve the learning efficiency of MASs. Specifically, monotonicity-constrained reward shaping is innovatively designed using expert experience to provide additional individual rewards to guide multiagent learning efficiently, with the invariance guarantee of the team optimization objective. Furthermore, a reward distribution estimator is specially developed to model an implicated reward distribution of environment by using transition experience from environment, containing collected samples (state-action pair, reward, and next state). This estimator can predict the expectation reward of each agent for the taken action to accurately estimate the state value function and accelerate its convergence. Besides, the performance of MEARL is evaluated on two multiagent environment platforms: our designed unmanned aerial vehicle combat (UAV-C) and StarCraft II Micromanagement (SCII-M). Simulation results demonstrate that the proposed MEARL can greatly improve the learning efficiency and performance of MASs and is superior to the state-of-the-art methods in multiagent tasks.
Collapse
|
4
|
Figueiredo Prudencio R, Maximo MROA, Colombini EL. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:10237-10257. [PMID: 37030754 DOI: 10.1109/tnnls.2023.3250269] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.
Collapse
|
5
|
Xu T, Meng Z, Lu W, Tong Z. End-to-End Autonomous Driving Decision Method Based on Improved TD3 Algorithm in Complex Scenarios. SENSORS (BASEL, SWITZERLAND) 2024; 24:4962. [PMID: 39124010 PMCID: PMC11315049 DOI: 10.3390/s24154962] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/12/2024] [Revised: 07/24/2024] [Accepted: 07/28/2024] [Indexed: 08/12/2024]
Abstract
The ability to make informed decisions in complex scenarios is crucial for intelligent automotive systems. Traditional expert rules and other methods often fall short in complex contexts. Recently, reinforcement learning has garnered significant attention due to its superior decision-making capabilities. However, there exists the phenomenon of inaccurate target network estimation, which limits its decision-making ability in complex scenarios. This paper mainly focuses on the study of the underestimation phenomenon, and proposes an end-to-end autonomous driving decision-making method based on an improved TD3 algorithm. This method employs a forward camera to capture data. By introducing a new critic network to form a triple-critic structure and combining it with the target maximization operation, the underestimation problem in the TD3 algorithm is solved. Subsequently, the multi-timestep averaging method is used to address the policy instability caused by the new single critic. In addition, this paper uses Carla platform to construct multi-vehicle unprotected left turn and congested lane-center driving scenarios and verifies the algorithm. The results demonstrate that our method surpasses baseline DDPG and TD3 algorithms in aspects such as convergence speed, estimation accuracy, and policy stability.
Collapse
Affiliation(s)
- Tao Xu
- National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130015, China; (T.X.); (Z.M.); (Z.T.)
| | - Zhiwei Meng
- National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130015, China; (T.X.); (Z.M.); (Z.T.)
| | - Weike Lu
- School of Rail Transportation, Soochow University, Suzhou 215031, China
| | - Zhongwen Tong
- National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130015, China; (T.X.); (Z.M.); (Z.T.)
| |
Collapse
|
6
|
Xue J, Han F, Klaassen van Oorschot B, Clifton G, Fan D. Exploring storm petrel pattering and sea-anchoring using deep reinforcement learning. BIOINSPIRATION & BIOMIMETICS 2023; 18:066016. [PMID: 37797650 DOI: 10.1088/1748-3190/ad00a2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 10/05/2023] [Indexed: 10/07/2023]
Abstract
Developing hybrid aerial-aquatic vehicles that can interact with water surfaces while remaining aloft is valuable for various tasks, including ecological monitoring, water quality sampling, and search and rescue operations. Storm petrels are a group of pelagic seabirds that exhibit a unique locomotion pattern known as 'pattering' or 'sea-anchoring,' which is hypothesized to support forward locomotion and/or stationary posture at the water surface. In this study, we use morphological measurements of three storm petrel species and aero/hydrodynamic models to develop a computational storm petrel model and interact it with a hybrid fluid environment. Using deep reinforcement learning algorithms, we find that the storm petrel model exhibits high maneuverability and stability under a wide range of constant wind velocities after training. We also verify in the simulation that the storm petrel can use its 'pattering' or 'sea-anchoring' behavior to achieve different biomechanical sub-tasks (e.g. weight support, forward locomotion, stabilization) and adapt it under different wind speeds and optimization objectives. Specifically, we observe an adjustment in storm petrel's movement patterns as wind velocity increases and quantitively analyze its biomechanics underneath. Our results provide new insights into how storm petrels achieve efficient locomotion and dynamic stability at the air-water interface and adapt their behaviors to different wind velocities and tasks in open environments. Ultimately, our study will guide the design of next-generation biomimetic petrel-inspired robots for tasks requiring proximity to the water interface and efficiency.
Collapse
Affiliation(s)
- Jiaqi Xue
- Key Laboratory of Coastal Environment and Resources of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, Zhejiang 310030, People's Republic of China
- School of Engineering, Westlake University, Hangzhou, Zhejiang 310024, People's Republic of China
- Walker Department of Mechanical Engineering, University of Texas at Austin, Austin, TX, United States of America
| | - Fei Han
- Key Laboratory of Coastal Environment and Resources of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, Zhejiang 310030, People's Republic of China
- School of Engineering, Westlake University, Hangzhou, Zhejiang 310024, People's Republic of China
| | | | - Glenna Clifton
- Department of Biology, University of Portland, Portland, OR, United States of America
| | - Dixia Fan
- Key Laboratory of Coastal Environment and Resources of Zhejiang Province, School of Engineering, Westlake University, Hangzhou, Zhejiang 310030, People's Republic of China
- School of Engineering, Westlake University, Hangzhou, Zhejiang 310024, People's Republic of China
| |
Collapse
|
7
|
Liu Q, Li X, Tang Y, Gao X, Yang F, Li Z. Graph Reinforcement Learning-Based Decision-Making Technology for Connected and Autonomous Vehicles: Framework, Review, and Future Trends. SENSORS (BASEL, SWITZERLAND) 2023; 23:8229. [PMID: 37837063 PMCID: PMC10575438 DOI: 10.3390/s23198229] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Revised: 09/27/2023] [Accepted: 09/29/2023] [Indexed: 10/15/2023]
Abstract
The proper functioning of connected and autonomous vehicles (CAVs) is crucial for the safety and efficiency of future intelligent transport systems. Meanwhile, transitioning to fully autonomous driving requires a long period of mixed autonomy traffic, including both CAVs and human-driven vehicles. Thus, collaborative decision-making technology for CAVs is essential to generate appropriate driving behaviors to enhance the safety and efficiency of mixed autonomy traffic. In recent years, deep reinforcement learning (DRL) methods have become an efficient way in solving decision-making problems. However, with the development of computing technology, graph reinforcement learning (GRL) methods have gradually demonstrated the large potential to further improve the decision-making performance of CAVs, especially in the area of accurately representing the mutual effects of vehicles and modeling dynamic traffic environments. To facilitate the development of GRL-based methods for autonomous driving, this paper proposes a review of GRL-based methods for the decision-making technologies of CAVs. Firstly, a generic GRL framework is proposed in the beginning to gain an overall understanding of the decision-making technology. Then, the GRL-based decision-making technologies are reviewed from the perspective of the construction methods of mixed autonomy traffic, methods for graph representation of the driving environment, and related works about graph neural networks (GNN) and DRL in the field of decision-making for autonomous driving. Moreover, validation methods are summarized to provide an efficient way to verify the performance of decision-making methods. Finally, challenges and future research directions of GRL-based decision-making methods are summarized.
Collapse
Affiliation(s)
- Qi Liu
- School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100811, China; (Q.L.); (X.G.); (F.Y.); (Z.L.)
| | - Xueyuan Li
- School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100811, China; (Q.L.); (X.G.); (F.Y.); (Z.L.)
| | - Yujie Tang
- Faculty of Computer Science, Dalhousie University, Halifax, NS B3H 4R2, Canada;
| | - Xin Gao
- School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100811, China; (Q.L.); (X.G.); (F.Y.); (Z.L.)
| | - Fan Yang
- School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100811, China; (Q.L.); (X.G.); (F.Y.); (Z.L.)
| | - Zirui Li
- School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100811, China; (Q.L.); (X.G.); (F.Y.); (Z.L.)
| |
Collapse
|
8
|
Zhang H, Xu Z, Wang Y, Shen Y. An innovative parameter optimization of Spark Streaming based on D3QN with Gaussian process regression. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2023; 20:14464-14486. [PMID: 37679144 DOI: 10.3934/mbe.2023647] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/09/2023]
Abstract
Nowadays, Spark Streaming, a computing framework based on Spark, is widely used to process streaming data such as social media data, IoT sensor data or web logs. Due to the extensive utilization of streaming media data analysis, performance optimization for Spark Streaming has gradually developed into a popular research topic. Several methods for enhancing Spark Streaming's performance include task scheduling, resource allocation and data skew optimization, which primarily focus on how to manually tune the parameter configuration. However, it is indeed very challenging and inefficient to adjust more than 200 parameters by means of continuous debugging. In this paper, we propose an improved dueling double deep Q-network (DQN) technique for parameter tuning, which can significantly improve the performance of Spark Streaming. This approach fuses reinforcement learning and Gaussian process regression to cut down on the number of iterations and speed convergence dramatically. The experimental results demonstrate that the performance of the dueling double DQN method with Gaussian process regression can be enhanced by up to 30.24%.
Collapse
Affiliation(s)
- Hong Zhang
- School of Cyber Security and Computer, Hebei University, Baoding, China
| | - Zhenchao Xu
- School of Cyber Security and Computer, Hebei University, Baoding, China
| | - Yunxiang Wang
- School of Cyber Security and Computer, Hebei University, Baoding, China
| | - Yupeng Shen
- Bureau of Geophysical Prospecting, Baoding, China
| |
Collapse
|
9
|
Zheng LE, Barethiya S, Nordquist E, Chen J. Machine Learning Generation of Dynamic Protein Conformational Ensembles. Molecules 2023; 28:4047. [PMID: 37241789 PMCID: PMC10220786 DOI: 10.3390/molecules28104047] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 05/04/2023] [Accepted: 05/09/2023] [Indexed: 05/28/2023] Open
Abstract
Machine learning has achieved remarkable success across a broad range of scientific and engineering disciplines, particularly its use for predicting native protein structures from sequence information alone. However, biomolecules are inherently dynamic, and there is a pressing need for accurate predictions of dynamic structural ensembles across multiple functional levels. These problems range from the relatively well-defined task of predicting conformational dynamics around the native state of a protein, which traditional molecular dynamics (MD) simulations are particularly adept at handling, to generating large-scale conformational transitions connecting distinct functional states of structured proteins or numerous marginally stable states within the dynamic ensembles of intrinsically disordered proteins. Machine learning has been increasingly applied to learn low-dimensional representations of protein conformational spaces, which can then be used to drive additional MD sampling or directly generate novel conformations. These methods promise to greatly reduce the computational cost of generating dynamic protein ensembles, compared to traditional MD simulations. In this review, we examine recent progress in machine learning approaches towards generative modeling of dynamic protein ensembles and emphasize the crucial importance of integrating advances in machine learning, structural data, and physical principles to achieve these ambitious goals.
Collapse
Affiliation(s)
- Li-E Zheng
- Department of Gynecology, The First Affiliated Hospital of Fujian Medical University, Fuzhou 350005, China;
| | - Shrishti Barethiya
- Department of Chemistry, University of Massachusetts Amherst, Amherst, MA 01003, USA; (S.B.); (E.N.)
| | - Erik Nordquist
- Department of Chemistry, University of Massachusetts Amherst, Amherst, MA 01003, USA; (S.B.); (E.N.)
| | - Jianhan Chen
- Department of Chemistry, University of Massachusetts Amherst, Amherst, MA 01003, USA; (S.B.); (E.N.)
| |
Collapse
|