51
|
Averbeck B, O'Doherty JP. Reinforcement-learning in fronto-striatal circuits. Neuropsychopharmacology 2022; 47:147-162. [PMID: 34354249 PMCID: PMC8616931 DOI: 10.1038/s41386-021-01108-0] [Citation(s) in RCA: 34] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Revised: 07/06/2021] [Accepted: 07/09/2021] [Indexed: 01/03/2023]
Abstract
We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.
Collapse
Affiliation(s)
| | - John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA.
| |
Collapse
|
52
|
Hennig JA, Oby ER, Losey DM, Batista AP, Yu BM, Chase SM. How learning unfolds in the brain: toward an optimization view. Neuron 2021; 109:3720-3735. [PMID: 34648749 PMCID: PMC8639641 DOI: 10.1016/j.neuron.2021.09.005] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2021] [Revised: 08/25/2021] [Accepted: 09/02/2021] [Indexed: 12/17/2022]
Abstract
How do changes in the brain lead to learning? To answer this question, consider an artificial neural network (ANN), where learning proceeds by optimizing a given objective or cost function. This "optimization framework" may provide new insights into how the brain learns, as many idiosyncratic features of neural activity can be recapitulated by an ANN trained to perform the same task. Nevertheless, there are key features of how neural population activity changes throughout learning that cannot be readily explained in terms of optimization and are not typically features of ANNs. Here we detail three of these features: (1) the inflexibility of neural variability throughout learning, (2) the use of multiple learning processes even during simple tasks, and (3) the presence of large task-nonspecific activity changes. We propose that understanding the role of these features in the brain will be key to describing biological learning using an optimization framework.
Collapse
Affiliation(s)
- Jay A Hennig
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA; Center for the Neural Basis of Cognition, Pittsburgh, PA, USA; Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA.
| | - Emily R Oby
- Center for the Neural Basis of Cognition, Pittsburgh, PA, USA; Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA
| | - Darby M Losey
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA; Center for the Neural Basis of Cognition, Pittsburgh, PA, USA; Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Aaron P Batista
- Center for the Neural Basis of Cognition, Pittsburgh, PA, USA; Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA
| | - Byron M Yu
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA; Center for the Neural Basis of Cognition, Pittsburgh, PA, USA; Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA; Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Steven M Chase
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA; Center for the Neural Basis of Cognition, Pittsburgh, PA, USA; Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
| |
Collapse
|
53
|
Prediction of genetic alteration of phospholipase C isozymes in brain disorders: Studies with deep learning. Adv Biol Regul 2021; 82:100833. [PMID: 34773889 DOI: 10.1016/j.jbior.2021.100833] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2021] [Revised: 10/15/2021] [Accepted: 10/19/2021] [Indexed: 11/22/2022]
Abstract
Genetic mutations leading to the development of various diseases, such as cancer, diabetes, and neurodegenerative disorders, can be attributed to multiple mechanisms and exposure to diverse environments. These disorders further increase gene mutation rates and affect the activity of translated proteins, both phenomena associated with cellular responses. Therefore, maintaining the integrity of genetic and epigenetic information is critical for disease suppression and prevention. With the advent of genome sequencing technologies, large-scale genomic data-based machine learning tools, including deep learning, have been used to predict and identify somatic inactivation or negative dominant expression of target genes in various diseases. Although deep learning studies have recently been highlighted for their ability to distinguish between the genetic information of diseases, conventional wisdom is also necessary to explain the correlation between genotype and phenotype. Herein, we summarize the current understanding of phosphoinositide-specific phospholipase C isozymes (PLCs) and an overview of their associations with genetic variation, as well as their emerging roles in several diseases. We also predicted and discussed new findings of cryptic PLC splice variants by deep learning and the clinical implications of the PLC genetic variations predicted using these tools.
Collapse
|
54
|
Schilling M, Melnik A, Ohl FW, Ritter HJ, Hammer B. Decentralized control and local information for robust and adaptive decentralized Deep Reinforcement Learning. Neural Netw 2021; 144:699-725. [PMID: 34673323 DOI: 10.1016/j.neunet.2021.09.017] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2021] [Revised: 09/13/2021] [Accepted: 09/21/2021] [Indexed: 12/18/2022]
Abstract
Decentralization is a central characteristic of biological motor control that allows for fast responses relying on local sensory information. In contrast, the current trend of Deep Reinforcement Learning (DRL) based approaches to motor control follows a centralized paradigm using a single, holistic controller that has to untangle the whole input information space. This motivates to ask whether decentralization as seen in biological control architectures might also be beneficial for embodied sensori-motor control systems when using DRL. To answer this question, we provide an analysis and comparison of eight control architectures for adaptive locomotion that were derived for a four-legged agent, but with their degree of decentralization varying systematically between the extremes of fully centralized and fully decentralized. Our comparison shows that learning speed is significantly enhanced in distributed architectures-while still reaching the same high performance level of centralized architectures-due to smaller search spaces and local costs providing more focused information for learning. Second, we find an increased robustness of the learning process in the decentralized cases-it is less demanding to hyperparameter selection and less prone to becoming trapped in poor local minima. Finally, when examining generalization to uneven terrains-not used during training-we find best performance for an intermediate architecture that is decentralized, but integrates only local information from both neighboring legs. Together, these findings demonstrate beneficial effects of distributing control into decentralized units and relying on local information. This appears as a promising approach towards more robust DRL and better generalization towards adaptive behavior.
Collapse
Affiliation(s)
- Malte Schilling
- Machine Learning Group, Bielefeld University, 33501 Bielefeld, Germany.
| | - Andrew Melnik
- Neuroinformatics Group, Bielefeld University, 33501 Bielefeld, Germany
| | - Frank W Ohl
- Department of Systems Physiology of Learning, Leibniz Institute for Neurobiology, Magdeburg, Germany; Institute of Biology, Otto-von-Guericke University, Magdeburg, Germany
| | - Helge J Ritter
- Neuroinformatics Group, Bielefeld University, 33501 Bielefeld, Germany
| | - Barbara Hammer
- Machine Learning Group, Bielefeld University, 33501 Bielefeld, Germany
| |
Collapse
|
55
|
Abstract
We report a possible solution for the long-standing problem of the biological function of swirling motion, when a group of animals orbits a common center of the group. We exploit the hypothesis that learning processes in the nervous system of animals may be modelled by reinforcement learning (RL) and apply it to explain the phenomenon. In contrast to hardly justified models of physical interactions between animals, we propose a small set of rules to be learned by the agents, which results in swirling. The rules are extremely simple and thus applicable to animals with very limited level of information processing. We demonstrate that swirling may be understood in terms of the escort behavior, when an individual animal tries to reside within a certain distance from the swarm center. Moreover, we reveal the biological function of swirling motion: a trained for swirling swarm is by orders of magnitude more resistant to external perturbations, than an untrained one. Using our approach we analyze another class of a coordinated motion of animals-a group locomotion in viscous fluid. On a model example we demonstrate that RL provides an optimal disposition of coherently moving animals with a minimal dissipation of energy.
Collapse
|
56
|
Balch JA, Delitto D, Tighe PJ, Zarrinpar A, Efron PA, Rashidi P, Upchurch GR, Bihorac A, Loftus TJ. Machine Learning Applications in Solid Organ Transplantation and Related Complications. Front Immunol 2021; 12:739728. [PMID: 34603324 PMCID: PMC8481939 DOI: 10.3389/fimmu.2021.739728] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2021] [Accepted: 08/25/2021] [Indexed: 11/13/2022] Open
Abstract
The complexity of transplant medicine pushes the boundaries of innate, human reasoning. From networks of immune modulators to dynamic pharmacokinetics to variable postoperative graft survival to equitable allocation of scarce organs, machine learning promises to inform clinical decision making by deciphering prodigious amounts of available data. This paper reviews current research describing how algorithms have the potential to augment clinical practice in solid organ transplantation. We provide a general introduction to different machine learning techniques, describing their strengths, limitations, and barriers to clinical implementation. We summarize emerging evidence that recent advances that allow machine learning algorithms to predict acute post-surgical and long-term outcomes, classify biopsy and radiographic data, augment pharmacologic decision making, and accurately represent the complexity of host immune response. Yet, many of these applications exist in pre-clinical form only, supported primarily by evidence of single-center, retrospective studies. Prospective investigation of these technologies has the potential to unlock the potential of machine learning to augment solid organ transplantation clinical care and health care delivery systems.
Collapse
Affiliation(s)
- Jeremy A Balch
- Department of Surgery, University of Florida Health, Gainesville, FL, United States
| | - Daniel Delitto
- Department of Surgery, Johns Hopkins University, Baltimore, MD, United States
| | - Patrick J Tighe
- Department of Anesthesiology, University of Florida Health, Gainesville, FL, United States.,Department of Orthopedics, University of Florida Health, Gainesville, FL, United States.,Department of Information Systems/Operations Management, University of Florida Health, Gainesville, FL, United States
| | - Ali Zarrinpar
- Department of Surgery, University of Florida Health, Gainesville, FL, United States
| | - Philip A Efron
- Department of Surgery, University of Florida Health, Gainesville, FL, United States
| | - Parisa Rashidi
- Department of Biomedical Engineering, University of Florida, Gainesville, FL, United States.,Department of Computer and Information Science and Engineering University of Florida, Gainesville, FL, United States.,Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL, United States.,Precision and Intelligent Systems in Medicine (PrismaP), University of Florida, Gainesville, FL, United States
| | - Gilbert R Upchurch
- Department of Surgery, University of Florida Health, Gainesville, FL, United States
| | - Azra Bihorac
- Precision and Intelligent Systems in Medicine (PrismaP), University of Florida, Gainesville, FL, United States.,Department of Medicine, University of Florida Health, Gainesville, FL, United States
| | - Tyler J Loftus
- Department of Surgery, University of Florida Health, Gainesville, FL, United States.,Precision and Intelligent Systems in Medicine (PrismaP), University of Florida, Gainesville, FL, United States
| |
Collapse
|
57
|
|
58
|
Eckstein MK, Wilbrecht L, Collins AGE. What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience. Curr Opin Behav Sci 2021; 41:128-137. [PMID: 34984213 PMCID: PMC8722372 DOI: 10.1016/j.cobeha.2021.06.004] [Citation(s) in RCA: 37] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Reinforcement learning (RL) is a concept that has been invaluable to fields including machine learning, neuroscience, and cognitive science. However, what RL entails differs between fields, leading to difficulties when interpreting and translating findings. After laying out these differences, this paper focuses on cognitive (neuro)science to discuss how we as a field might over-interpret RL modeling results. We too often assume-implicitly-that modeling results generalize between tasks, models, and participant populations, despite negative empirical evidence for this assumption. We also often assume that parameters measure specific, unique (neuro)cognitive processes, a concept we call interpretability, when evidence suggests that they capture different functions across studies and tasks. We conclude that future computational research needs to pay increased attention to implicit assumptions when using RL models, and suggest that a more systematic understanding of contextual factors will help address issues and improve the ability of RL to explain brain and behavior.
Collapse
Affiliation(s)
- Maria K Eckstein
- Department of Psychology, UC Berkeley, 2121 Berkeley Way West, Berkeley, 94720, CA, USA
| | - Linda Wilbrecht
- Department of Psychology, UC Berkeley, 2121 Berkeley Way West, Berkeley, 94720, CA, USA
- Helen Wills Neuroscience Institute, UC Berkeley, 175 Li Ka Shing Center, Berkeley, 94720, CA, USA
| | - Anne G E Collins
- Department of Psychology, UC Berkeley, 2121 Berkeley Way West, Berkeley, 94720, CA, USA
- Helen Wills Neuroscience Institute, UC Berkeley, 175 Li Ka Shing Center, Berkeley, 94720, CA, USA
| |
Collapse
|
59
|
Stelzer F, Röhm A, Vicente R, Fischer I, Yanchuk S. Deep neural networks using a single neuron: folded-in-time architecture using feedback-modulated delay loops. Nat Commun 2021; 12:5164. [PMID: 34453053 PMCID: PMC8397757 DOI: 10.1038/s41467-021-25427-4] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2021] [Accepted: 08/10/2021] [Indexed: 11/21/2022] Open
Abstract
Deep neural networks are among the most widely applied machine learning tools showing outstanding performance in a broad range of tasks. We present a method for folding a deep neural network of arbitrary size into a single neuron with multiple time-delayed feedback loops. This single-neuron deep neural network comprises only a single nonlinearity and appropriately adjusted modulations of the feedback signals. The network states emerge in time as a temporal unfolding of the neuron's dynamics. By adjusting the feedback-modulation within the loops, we adapt the network's connection weights. These connection weights are determined via a back-propagation algorithm, where both the delay-induced and local network connections must be taken into account. Our approach can fully represent standard Deep Neural Networks (DNN), encompasses sparse DNNs, and extends the DNN concept toward dynamical systems implementations. The new method, which we call Folded-in-time DNN (Fit-DNN), exhibits promising performance in a set of benchmark tasks.
Collapse
Affiliation(s)
- Florian Stelzer
- Institute of Mathematics, Technische Universität Berlin, Berlin, Germany
- Department of Mathematics, Humboldt-Universität zu Berlin, Berlin, Germany
- Institute of Computer Science, University of Tartu, Tartu, Estonia
| | - André Röhm
- Instituto de Física Interdisciplinar y Sistemas Complejos, IFISC (UIB-CSIC), Campus Universitat de les Illes Baleares, Palma de Mallorca, Spain
| | - Raul Vicente
- Institute of Computer Science, University of Tartu, Tartu, Estonia
| | - Ingo Fischer
- Instituto de Física Interdisciplinar y Sistemas Complejos, IFISC (UIB-CSIC), Campus Universitat de les Illes Baleares, Palma de Mallorca, Spain
| | - Serhiy Yanchuk
- Institute of Mathematics, Technische Universität Berlin, Berlin, Germany.
| |
Collapse
|
60
|
Abstract
The concept of memory is traditionally associated with organisms possessing a nervous system. However, even very simple organisms store information about past experiences to thrive in a complex environment-successfully exploiting nutrient sources, avoiding danger, and warding off predators. How can simple organisms encode information about their environment? We here follow how the giant unicellular slime mold Physarum polycephalum responds to a nutrient source. We find that the network-like body plan of the organism itself serves to encode the location of a nutrient source. The organism entirely consists of interlaced tubes of varying diameters. Now, we observe that these tubes grow and shrink in diameter in response to a nutrient source, thereby imprinting the nutrient's location in the tube diameter hierarchy. Combining theoretical model and experimental data, we reveal how memory is encoded: a nutrient source locally releases a softening agent that gets transported by the cytoplasmic flows within the tubular network. Tubes receiving a lot of softening agent grow in diameter at the expense of other tubes shrinking. Thereby, the tubes' capacities for flow-based transport get permanently upgraded toward the nutrient location, redirecting future decisions and migration. This demonstrates that nutrient location is stored in and retrieved from the networks' tube diameter hierarchy. Our findings explain how network-forming organisms like slime molds and fungi thrive in complex environments. We here identify a flow networks' version of associative memory-very likely of relevance for the plethora of living flow networks as well as for bioinspired design.
Collapse
|
61
|
Harnessing artificial intelligence for the next generation of 3D printed medicines. Adv Drug Deliv Rev 2021; 175:113805. [PMID: 34019957 DOI: 10.1016/j.addr.2021.05.015] [Citation(s) in RCA: 57] [Impact Index Per Article: 19.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2021] [Revised: 05/02/2021] [Accepted: 05/13/2021] [Indexed: 02/06/2023]
Abstract
Artificial intelligence (AI) is redefining how we exist in the world. In almost every sector of society, AI is performing tasks with super-human speed and intellect; from the prediction of stock market trends to driverless vehicles, diagnosis of disease, and robotic surgery. Despite this growing success, the pharmaceutical field is yet to truly harness AI. Development and manufacture of medicines remains largely in a 'one size fits all' paradigm, in which mass-produced, identical formulations are expected to meet individual patient needs. Recently, 3D printing (3DP) has illuminated a path for on-demand production of fully customisable medicines. Due to its flexibility, pharmaceutical 3DP presents innumerable options during formulation development that generally require expert navigation. Leveraging AI within pharmaceutical 3DP removes the need for human expertise, as optimal process parameters can be accurately predicted by machine learning. AI can also be incorporated into a pharmaceutical 3DP 'Internet of Things', moving the personalised production of medicines into an intelligent, streamlined, and autonomous pipeline. Supportive infrastructure, such as The Cloud and blockchain, will also play a vital role. Crucially, these technologies will expedite the use of pharmaceutical 3DP in clinical settings and drive the global movement towards personalised medicine and Industry 4.0.
Collapse
|
62
|
Hoy CW, Steiner SC, Knight RT. Single-trial modeling separates multiple overlapping prediction errors during reward processing in human EEG. Commun Biol 2021; 4:910. [PMID: 34302057 PMCID: PMC8302587 DOI: 10.1038/s42003-021-02426-1] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2020] [Accepted: 07/05/2021] [Indexed: 02/07/2023] Open
Abstract
Learning signals during reinforcement learning and cognitive control rely on valenced reward prediction errors (RPEs) and non-valenced salience prediction errors (PEs) driven by surprise magnitude. A core debate in reward learning focuses on whether valenced and non-valenced PEs can be isolated in the human electroencephalogram (EEG). We combine behavioral modeling and single-trial EEG regression to disentangle sequential PEs in an interval timing task dissociating outcome valence, magnitude, and probability. Multiple regression across temporal, spatial, and frequency dimensions characterized a spatio-tempo-spectral cascade from early valenced RPE value to non-valenced RPE magnitude, followed by outcome probability indexed by a late frontal positivity. Separating negative and positive outcomes revealed the valenced RPE value effect is an artifact of overlap between two non-valenced RPE magnitude responses: frontal theta feedback-related negativity on losses and posterior delta reward positivity on wins. These results reconcile longstanding debates on the sequence of components representing reward and salience PEs in the human EEG.
Collapse
Affiliation(s)
- Colin W Hoy
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA.
| | - Sheila C Steiner
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA
| | - Robert T Knight
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA
- Department of Psychology, University of California Berkeley, Berkeley, CA, USA
| |
Collapse
|
63
|
van der Maas HL, Snoek L, Stevenson CE. How much intelligence is there in artificial intelligence? A 2020 update. INTELLIGENCE 2021. [DOI: 10.1016/j.intell.2021.101548] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
|
64
|
Ohta H, Satori K, Takarada Y, Arake M, Ishizuka T, Morimoto Y, Takahashi T. The asymmetric learning rates of murine exploratory behavior in sparse reward environments. Neural Netw 2021; 143:218-229. [PMID: 34157646 DOI: 10.1016/j.neunet.2021.05.030] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2021] [Revised: 04/16/2021] [Accepted: 05/26/2021] [Indexed: 11/29/2022]
Abstract
Goal-oriented behaviors of animals can be modeled by reinforcement learning algorithms. Such algorithms predict future outcomes of selected actions utilizing action values and updating those values in response to the positive and negative outcomes. In many models of animal behavior, the action values are updated symmetrically based on a common learning rate, that is, in the same way for both positive and negative outcomes. However, animals in environments with scarce rewards may have uneven learning rates. To investigate the asymmetry in learning rates in reward and non-reward, we analyzed the exploration behavior of mice in five-armed bandit tasks using a Q-learning model with differential learning rates for positive and negative outcomes. The positive learning rate was significantly higher in a scarce reward environment than in a rich reward environment, and conversely, the negative learning rate was significantly lower in the scarce environment. The positive to negative learning rate ratio was about 10 in the scarce environment and about 2 in the rich environment. This result suggests that when the reward probability was low, the mice tend to ignore failures and exploit the rare rewards. Computational modeling analysis revealed that the increased learning rates ratio could cause an overestimation of and perseveration on rare-rewarding events, increasing total reward acquisition in the scarce environment but disadvantaging impartial exploration.
Collapse
Affiliation(s)
- Hiroyuki Ohta
- Department of Pharmacology, National Defense Medical College, Saitama, 359-8513, Japan.
| | | | - Yu Takarada
- Tokyo Denki University, Saitama, 350-0394, Japan
| | - Masashi Arake
- Department of Physiology, National Defense Medical College, Saitama, 359-8513, Japan
| | - Toshiaki Ishizuka
- Department of Pharmacology, National Defense Medical College, Saitama, 359-8513, Japan
| | - Yuji Morimoto
- Department of Physiology, National Defense Medical College, Saitama, 359-8513, Japan
| | | |
Collapse
|
65
|
Piantadosi PT, Halladay LR, Radke AK, Holmes A. Advances in understanding meso-cortico-limbic-striatal systems mediating risky reward seeking. J Neurochem 2021; 157:1547-1571. [PMID: 33704784 PMCID: PMC8981567 DOI: 10.1111/jnc.15342] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2021] [Revised: 03/04/2021] [Accepted: 03/06/2021] [Indexed: 02/06/2023]
Abstract
The risk of an aversive consequence occurring as the result of a reward-seeking action can have a profound effect on subsequent behavior. Such aversive events can be described as punishers, as they decrease the probability that the same action will be produced again in the future and increase the exploration of less risky alternatives. Punishment can involve the omission of an expected rewarding event ("negative" punishment) or the addition of an unpleasant event ("positive" punishment). Although many individuals adaptively navigate situations associated with the risk of negative or positive punishment, those suffering from substance use disorders or behavioral addictions tend to be less able to curtail addictive behaviors despite the aversive consequences associated with them. Here, we discuss the psychological processes underpinning reward seeking despite the risk of negative and positive punishment and consider how behavioral assays in animals have been employed to provide insights into the neural mechanisms underlying addictive disorders. We then review the critical contributions of dopamine signaling to punishment learning and risky reward seeking, and address the roles of interconnected ventral striatal, cortical, and amygdala regions to these processes. We conclude by discussing the ample opportunities for future study to clarify critical gaps in the literature, particularly as related to delineating neural contributions to distinct phases of the risky decision-making process.
Collapse
Affiliation(s)
- Patrick T. Piantadosi
- Laboratory of Behavioral and Genomic Neuroscience, National Institute on Alcohol Abuse and Alcoholism, NIH, Bethesda, MD, USA
| | - Lindsay R. Halladay
- Department of Psychology, Santa Clara University, Santa Clara, California 95053, USA
| | - Anna K. Radke
- Department of Psychology and Center for Neuroscience and Behavior, Miami University, Oxford, OH, USA
| | - Andrew Holmes
- Laboratory of Behavioral and Genomic Neuroscience, National Institute on Alcohol Abuse and Alcoholism, NIH, Bethesda, MD, USA
| |
Collapse
|
66
|
Lange-Küttner C, Averbeck BB, Hentschel M, Baumbach J. Intelligence matters for stochastic feedback processing during sequence learning in adolescents and young adults. INTELLIGENCE 2021. [DOI: 10.1016/j.intell.2021.101542] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
|
67
|
Peng GCY, Alber M, Tepole AB, Cannon WR, De S, Dura-Bernal S, Garikipati K, Karniadakis G, Lytton WW, Perdikaris P, Petzold L, Kuhl E. Multiscale modeling meets machine learning: What can we learn? ARCHIVES OF COMPUTATIONAL METHODS IN ENGINEERING : STATE OF THE ART REVIEWS 2021; 28:1017-1037. [PMID: 34093005 PMCID: PMC8172124 DOI: 10.1007/s11831-020-09405-5] [Citation(s) in RCA: 59] [Impact Index Per Article: 19.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/30/2019] [Accepted: 02/09/2020] [Indexed: 05/10/2023]
Abstract
Machine learning is increasingly recognized as a promising technology in the biological, biomedical, and behavioral sciences. There can be no argument that this technique is incredibly successful in image recognition with immediate applications in diagnostics including electrophysiology, radiology, or pathology, where we have access to massive amounts of annotated data. However, machine learning often performs poorly in prognosis, especially when dealing with sparse data. This is a field where classical physics-based simulation seems to remain irreplaceable. In this review, we identify areas in the biomedical sciences where machine learning and multiscale modeling can mutually benefit from one another: Machine learning can integrate physics-based knowledge in the form of governing equations, boundary conditions, or constraints to manage ill-posted problems and robustly handle sparse and noisy data; multiscale modeling can integrate machine learning to create surrogate models, identify system dynamics and parameters, analyze sensitivities, and quantify uncertainty to bridge the scales and understand the emergence of function. With a view towards applications in the life sciences, we discuss the state of the art of combining machine learning and multiscale modeling, identify applications and opportunities, raise open questions, and address potential challenges and limitations. We anticipate that it will stimulate discussion within the community of computational mechanics and reach out to other disciplines including mathematics, statistics, computer science, artificial intelligence, biomedicine, systems biology, and precision medicine to join forces towards creating robust and efficient models for biological systems.
Collapse
Affiliation(s)
| | - Mark Alber
- University of California, Riverside, USA
| | | | - William R Cannon
- Pacific Northwest National Laboratory, Richland, Washington, USA
| | - Suvranu De
- Rensselaer Polytechnic Institute, Troy, New York, USA
| | | | | | | | | | | | - Linda Petzold
- University of California, Santa Barbara, California, USA
| | - Ellen Kuhl
- Stanford University, Stanford, California, USA
| |
Collapse
|
68
|
Singh B, Kumar R, Singh VP. Reinforcement learning in robotic applications: a comprehensive survey. Artif Intell Rev 2021. [DOI: 10.1007/s10462-021-09997-9] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
69
|
|
70
|
Nguyen HT, Nguyen KTQ, Le TC, Zhang G. Review on the Use of Artificial Intelligence to Predict Fire Performance of Construction Materials and Their Flame Retardancy. Molecules 2021; 26:1022. [PMID: 33672068 PMCID: PMC7919694 DOI: 10.3390/molecules26041022] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2021] [Revised: 02/04/2021] [Accepted: 02/06/2021] [Indexed: 11/16/2022] Open
Abstract
The evaluation and interpretation of the behavior of construction materials under fire conditions have been complicated. Over the last few years, artificial intelligence (AI) has emerged as a reliable method to tackle this engineering problem. This review summarizes existing studies that applied AI to predict the fire performance of different construction materials (e.g., concrete, steel, timber, and composites). The prediction of the flame retardancy of some structural components such as beams, columns, slabs, and connections by utilizing AI-based models is also discussed. The end of this review offers insights on the advantages, existing challenges, and recommendations for the development of AI techniques used to evaluate the fire performance of construction materials and their flame retardancy. This review offers a comprehensive overview to researchers in the fields of fire engineering and material science, and it encourages them to explore and consider the use of AI in future research projects.
Collapse
Affiliation(s)
- Hoang T. Nguyen
- Department of Infrastructure Engineering, School of Engineering, RMIT University, GPO Box 2476, Melbourne, VIC 3001, Australia; (H.T.N.); (G.Z.)
| | - Kate T. Q. Nguyen
- Department of Infrastructure Engineering, School of Engineering, RMIT University, GPO Box 2476, Melbourne, VIC 3001, Australia; (H.T.N.); (G.Z.)
| | - Tu C. Le
- Manufacturing, Materials and Mechatronics, School of Engineering, RMIT University, GPO Box 2476, Melbourne, VIC 3001, Australia;
| | - Guomin Zhang
- Department of Infrastructure Engineering, School of Engineering, RMIT University, GPO Box 2476, Melbourne, VIC 3001, Australia; (H.T.N.); (G.Z.)
| |
Collapse
|
71
|
Machine Learning-Based Cognitive Position and Force Controls for Power-Assisted Human–Robot Collaborative Manipulation. MACHINES 2021. [DOI: 10.3390/machines9020028] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Manipulation of heavy objects in industries is very necessary, but manual manipulation is tedious, adversely affects a worker’s health and safety, and reduces efficiency. On the contrary, autonomous robots are not flexible to manipulate heavy objects. Hence, we proposed human–robot systems, such as power assist systems, to manipulate heavy objects in industries. Again, the selection of appropriate control methods as well as inclusion of human factors in the controls is important to make the systems human friendly. However, existing power assist systems do not address these issues properly. Hence, we present a 1-DoF (degree of freedom) testbed power assist robotic system for lifting different objects. We also included a human factor, such as weight perception (a cognitive cue), in the robotic system dynamics and derived several position and force control strategies/methods for the system based on the human-centric dynamics. We developed a reinforcement learning method to predict the control parameters producing the best/optimal control performance. We also derived a novel adaptive control algorithm based on human characteristics. We experimentally evaluated those control methods and compared the system performance between the control methods. Results showed that both position and force controls produced satisfactory performance, but the position control produced significantly better performance than the force controls. We then proposed using the results to design control methods for power assist robotic systems for handling large and heavy materials and objects in various industries, which may improve human–robot interactions (HRIs) and system performance.
Collapse
|
72
|
Haeufle DFB, Wochner I, Holzmüller D, Driess D, Günther M, Schmitt S. Muscles Reduce Neuronal Information Load: Quantification of Control Effort in Biological vs. Robotic Pointing and Walking. Front Robot AI 2021; 7:77. [PMID: 33501244 PMCID: PMC7805995 DOI: 10.3389/frobt.2020.00077] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2019] [Accepted: 05/07/2020] [Indexed: 12/17/2022] Open
Abstract
It is hypothesized that the nonlinear muscle characteristic of biomechanical systems simplify control in the sense that the information the nervous system has to process is reduced through off-loading computation to the morphological structure. It has been proposed to quantify the required information with an information-entropy based approach, which evaluates the minimally required information to control a desired movement, i.e., control effort. The key idea is to compare the same movement but generated by different actuators, e.g., muscles and torque actuators, and determine which of the two morphologies requires less information to generate the same movement. In this work, for the first time, we apply this measure to numerical simulations of more complex human movements: point-to-point arm movements and walking. These models consider up to 24 control signals rendering the brute force approach of the previous implementation to search for the minimally required information futile. We therefore propose a novel algorithm based on the pattern search approach specifically designed to solve this constraint optimization problem. We apply this algorithm to numerical models, which include Hill-type muscle-tendon actuation as well as ideal torque sources acting directly on the joints. The controller for the point-to-point movements was obtained by deep reinforcement learning for muscle and torque actuators. Walking was controlled by proprioceptive neural feedback in the muscular system and a PD controller in the torque model. Results show that the neuromuscular models consistently require less information to successfully generate the movement than the torque-driven counterparts. These findings were consistent for all investigated controllers in our experiments, implying that this is a system property, not a controller property. The proposed algorithm to determine the control effort is more efficient than other standard optimization techniques and provided as open source.
Collapse
Affiliation(s)
- Daniel F B Haeufle
- Hertie Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany.,Center for Integrative Neuroscience, University of Tübingen, Tübingen, Germany
| | - Isabell Wochner
- Institute for Modelling and Simulation of Biomechanical Systems, University of Stuttgart, Stuttgart, Germany.,Stuttgart Center for Simulation Science, University of Stuttgart, Stuttgart, Germany
| | - David Holzmüller
- Machine Learning and Robotics Lab, University of Stuttgart, Stuttgart, Germany.,Institute for Stochastics and Applications, University of Stuttgart, Stuttgart, Germany
| | - Danny Driess
- Machine Learning and Robotics Lab, University of Stuttgart, Stuttgart, Germany.,Max-Planck Institute for Intelligent Systems, Stuttgart, Germany
| | - Michael Günther
- Institute for Modelling and Simulation of Biomechanical Systems, University of Stuttgart, Stuttgart, Germany
| | - Syn Schmitt
- Institute for Modelling and Simulation of Biomechanical Systems, University of Stuttgart, Stuttgart, Germany.,Stuttgart Center for Simulation Science, University of Stuttgart, Stuttgart, Germany
| |
Collapse
|
73
|
McCoubrey LE, Elbadawi M, Orlu M, Gaisford S, Basit AW. Harnessing machine learning for development of microbiome therapeutics. Gut Microbes 2021; 13:1-20. [PMID: 33522391 PMCID: PMC7872042 DOI: 10.1080/19490976.2021.1872323] [Citation(s) in RCA: 37] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/27/2020] [Accepted: 12/20/2020] [Indexed: 02/06/2023] Open
Abstract
The last twenty years of seminal microbiome research has uncovered microbiota's intrinsic relationship with human health. Studies elucidating the relationship between an unbalanced microbiome and disease are currently published daily. As such, microbiome big data have become a reality that provide a mine of information for the development of new therapeutics. Machine learning (ML), a branch of artificial intelligence, offers powerful techniques for big data analysis and prediction-making, that are out of reach of human intellect alone. This review will explore how ML can be applied for the development of microbiome-targeted therapeutics. A background on ML will be given, followed by a guide on where to find reliable microbiome big data. Existing applications and opportunities will be discussed, including the use of ML to discover, design, and characterize microbiome therapeutics. The use of ML to optimize advanced processes, such as 3D printing and in silico prediction of drug-microbiome interactions, will also be highlighted. Finally, barriers to adoption of ML in academic and industrial settings will be examined, concluded by a future outlook for the field.
Collapse
Affiliation(s)
| | - Moe Elbadawi
- UCL School of Pharmacy, University College London, London, UK
| | - Mine Orlu
- UCL School of Pharmacy, University College London, London, UK
| | - Simon Gaisford
- UCL School of Pharmacy, University College London, London, UK
- FabRx Ltd., Ashford, Kent, UK
| | - Abdul W. Basit
- UCL School of Pharmacy, University College London, London, UK
| |
Collapse
|
74
|
Schilling M, Paskarbeit J, Ritter H, Schneider A, Cruse H. From Adaptive Locomotion to Predictive Action Selection – Cognitive Control for a Six-Legged Walker. IEEE T ROBOT 2021. [DOI: 10.1109/tro.2021.3106832] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
75
|
Yang C, Yuan K, Zhu Q, Yu W, Li Z. Multi-expert learning of adaptive legged locomotion. Sci Robot 2020; 5:5/49/eabb2174. [PMID: 33298515 DOI: 10.1126/scirobotics.abb2174] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2020] [Accepted: 11/13/2020] [Indexed: 12/15/2022]
Abstract
Achieving versatile robot locomotion requires motor skills that can adapt to previously unseen situations. We propose a multi-expert learning architecture (MELA) that learns to generate adaptive skills from a group of representative expert skills. During training, MELA is first initialized by a distinct set of pretrained experts, each in a separate deep neural network (DNN). Then, by learning the combination of these DNNs using a gating neural network (GNN), MELA can acquire more specialized experts and transitional skills across various locomotion modes. During runtime, MELA constantly blends multiple DNNs and dynamically synthesizes a new DNN to produce adaptive behaviors in response to changing situations. This approach leverages the advantages of trained expert skills and the fast online synthesis of adaptive policies to generate responsive motor skills during the changing tasks. Using one unified MELA framework, we demonstrated successful multiskill locomotion on a real quadruped robot that performed coherent trotting, steering, and fall recovery autonomously and showed the merit of multi-expert learning generating behaviors that can adapt to unseen scenarios.
Collapse
Affiliation(s)
- Chuanyu Yang
- School of Informatics, University of Edinburgh, Edinburgh, UK
| | - Kai Yuan
- School of Informatics, University of Edinburgh, Edinburgh, UK
| | - Qiuguo Zhu
- Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou, China
| | - Wanming Yu
- School of Informatics, University of Edinburgh, Edinburgh, UK
| | - Zhibin Li
- School of Informatics, University of Edinburgh, Edinburgh, UK.
| |
Collapse
|
76
|
Song MK, Namgung SD, Choi D, Kim H, Seo H, Ju M, Lee YH, Sung T, Lee YS, Nam KT, Kwon JY. Proton-enabled activation of peptide materials for biological bimodal memory. Nat Commun 2020; 11:5896. [PMID: 33214548 PMCID: PMC7677316 DOI: 10.1038/s41467-020-19750-5] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2020] [Accepted: 10/29/2020] [Indexed: 02/06/2023] Open
Abstract
The process of memory and learning in biological systems is multimodal, as several kinds of input signals cooperatively determine the weight of information transfer and storage. This study describes a peptide-based platform of materials and devices that can control the coupled conduction of protons and electrons and thus create distinct regions of synapse-like performance depending on the proton activity. We utilized tyrosine-rich peptide-based films and generalized our principles by demonstrating both memristor and synaptic devices. Interestingly, even memristive behavior can be controlled by both voltage and humidity inputs, learning and forgetting process in the device can be initiated and terminated by protons alone in peptide films. We believe that this work can help to understand the mechanism of biological memory and lay a foundation to realize a brain-like device based on ions and electrons. The structural programmability and functionality of peptide materials can be leverage for various next-generation devices such as non-volatile memories. The authors report a proton-coupled mechanism in tyrosine-rich peptides for realizing multimodal memory devices.
Collapse
Affiliation(s)
- Min-Kyu Song
- School of Integrated Technology, Yonsei University, Incheon, 21983, Republic of Korea
| | - Seok Daniel Namgung
- Department of Materials Science and Engineering, Seoul National University, Seoul, 08826, Republic of Korea
| | - Daehwan Choi
- School of Integrated Technology, Yonsei University, Incheon, 21983, Republic of Korea
| | - Hyeohn Kim
- Department of Materials Science and Engineering, Seoul National University, Seoul, 08826, Republic of Korea
| | - Hongmin Seo
- Department of Materials Science and Engineering, Seoul National University, Seoul, 08826, Republic of Korea
| | - Misong Ju
- Department of Materials Science and Engineering, Seoul National University, Seoul, 08826, Republic of Korea
| | - Yoon Ho Lee
- Department of Materials Science and Engineering, Seoul National University, Seoul, 08826, Republic of Korea
| | - Taehoon Sung
- School of Integrated Technology, Yonsei University, Incheon, 21983, Republic of Korea
| | - Yoon-Sik Lee
- School of Chemical and Biological Engineering, Nano Systems Institute, Seoul National University, Seoul, 08826, Republic of Korea
| | - Ki Tae Nam
- Department of Materials Science and Engineering, Seoul National University, Seoul, 08826, Republic of Korea.
| | - Jang-Yeon Kwon
- School of Integrated Technology, Yonsei University, Incheon, 21983, Republic of Korea.
| |
Collapse
|
77
|
Abstract
AbstractRecommender systems provide personalized service support to users by learning their previous behaviors and predicting their current preferences for particular products. Artificial intelligence (AI), particularly computational intelligence and machine learning methods and algorithms, has been naturally applied in the development of recommender systems to improve prediction accuracy and solve data sparsity and cold start problems. This position paper systematically discusses the basic methodologies and prevailing techniques in recommender systems and how AI can effectively improve the technological development and application of recommender systems. The paper not only reviews cutting-edge theoretical and practical contributions, but also identifies current research issues and indicates new research directions. It carefully surveys various issues related to recommender systems that use AI, and also reviews the improvements made to these systems through the use of such AI approaches as fuzzy techniques, transfer learning, genetic algorithms, evolutionary algorithms, neural networks and deep learning, and active learning. The observations in this paper will directly support researchers and professionals to better understand current developments and new directions in the field of recommender systems using AI.
Collapse
|
78
|
Denervaud S, Hess A, Sander D, Pourtois G. Children's automatic evaluation of self-generated actions is different from adults. Dev Sci 2020; 24:e13045. [PMID: 33090680 DOI: 10.1111/desc.13045] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2019] [Revised: 09/04/2020] [Accepted: 09/16/2020] [Indexed: 11/28/2022]
Abstract
Performance monitoring (PM) is central to learning and decision making. It allows individuals to swiftly detect deviations between actions and intentions, such as response errors, and adapt behavior accordingly. Previous research showed that in adult participants, error monitoring is associated with two distinct and robust behavioral effects. First, a systematic slowing down of reaction time speed is typically observed following error commission, which is known as post-error slowing (PES). Second, response errors have been reported to be automatically evaluated as negative events in adults. However, it remains unclear whether (1) children process response errors as adults do (PES), (2) they also evaluate them as negative events, and (3) their responses vary according to the pedagogy experienced. To address these questions, we adapted a simple decision-making task previously validated in adults to measure PES as well as the affective processing of response errors. We recruited 8- to 12-year-old children enrolled in traditional (N = 56) or Montessori (N = 45) schools, and compared them to adults (N = 46) on the exact same task. Results showed that children processed correct actions as positive events, and that adults processed errors as negative events. By contrast, PES was similarly observed in all groups. Moreover, the former effect was observed in traditional schoolchildren, but not in Montessori schoolchildren. These findings suggest that unlike PES, which likely reflects an age-invariant attention orienting toward response errors, their affective processing depends on both age and pedagogy.
Collapse
Affiliation(s)
- Solange Denervaud
- Swiss Center for Affective Sciences, Campus Biotech, University of Geneva, Geneva, Switzerland.,Connectomics Lab, Department of Radiology, Lausanne University Hospital and University of Lausanne (CHUV-UNIL), Lausanne, Switzerland
| | - Adrien Hess
- Department of Psychology, Faculty of Psychology and Educational Sciences (FPSE), University of Geneva, Geneva, Switzerland
| | - David Sander
- Swiss Center for Affective Sciences, Campus Biotech, University of Geneva, Geneva, Switzerland.,Department of Psychology, Faculty of Psychology and Educational Sciences (FPSE), University of Geneva, Geneva, Switzerland
| | - Gilles Pourtois
- Cap Lab, Department of Experimental Clinical and Health Psychology, Ghent University, Gent, Belgium
| |
Collapse
|
79
|
Jin C, Chen W, Cao Y, Xu Z, Tan Z, Zhang X, Deng L, Zheng C, Zhou J, Shi H, Feng J. Development and evaluation of an artificial intelligence system for COVID-19 diagnosis. Nat Commun 2020; 11:5088. [PMID: 33037212 DOI: 10.1101/823377] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2020] [Accepted: 09/04/2020] [Indexed: 05/22/2023] Open
Abstract
Early detection of COVID-19 based on chest CT enables timely treatment of patients and helps control the spread of the disease. We proposed an artificial intelligence (AI) system for rapid COVID-19 detection and performed extensive statistical analysis of CTs of COVID-19 based on the AI system. We developed and evaluated our system on a large dataset with more than 10 thousand CT volumes from COVID-19, influenza-A/B, non-viral community acquired pneumonia (CAP) and non-pneumonia subjects. In such a difficult multi-class diagnosis task, our deep convolutional neural network-based system is able to achieve an area under the receiver operating characteristic curve (AUC) of 97.81% for multi-way classification on test cohort of 3,199 scans, AUC of 92.99% and 93.25% on two publicly available datasets, CC-CCII and MosMedData respectively. In a reader study involving five radiologists, the AI system outperforms all of radiologists in more challenging tasks at a speed of two orders of magnitude above them. Diagnosis performance of chest x-ray (CXR) is compared to that of CT. Detailed interpretation of deep network is also performed to relate system outputs with CT presentations. The code is available at https://github.com/ChenWWWeixiang/diagnosis_covid19 .
Collapse
Affiliation(s)
- Cheng Jin
- Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Weixiang Chen
- Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Yukun Cao
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Hubei Province Key Laboratory of Molecular Imaging, Wuhan, China
| | - Zhanwei Xu
- Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Zimeng Tan
- Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Xin Zhang
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Hubei Province Key Laboratory of Molecular Imaging, Wuhan, China
| | - Lei Deng
- Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Chuansheng Zheng
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Hubei Province Key Laboratory of Molecular Imaging, Wuhan, China
| | - Jie Zhou
- Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Heshui Shi
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.
- Hubei Province Key Laboratory of Molecular Imaging, Wuhan, China.
| | - Jianjiang Feng
- Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China.
| |
Collapse
|
80
|
Alexiadis A, Simmons MJH, Stamatopoulos K, Batchelor HK, Moulitsas I. The duality between particle methods and artificial neural networks. Sci Rep 2020; 10:16247. [PMID: 33004941 PMCID: PMC7530753 DOI: 10.1038/s41598-020-73329-0] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2020] [Accepted: 09/14/2020] [Indexed: 11/20/2022] Open
Abstract
The algorithm behind particle methods is extremely versatile and used in a variety of applications that range from molecular dynamics to astrophysics. For continuum mechanics applications, the concept of ‘particle’ can be generalized to include discrete portions of solid and liquid matter. This study shows that it is possible to further extend the concept of ‘particle’ to include artificial neurons used in Artificial Intelligence. This produces a new class of computational methods based on ‘particle-neuron duals’ that combines the ability of computational particles to model physical systems and the ability of artificial neurons to learn from data. The method is validated with a multiphysics model of the intestine that autonomously learns how to coordinate its contractions to propel the luminal content forward (peristalsis). Training is achieved with Deep Reinforcement Learning. The particle-neuron duality has the advantage of extending particle methods to systems where the underlying physics is only partially known, but we have observations that allow us to empirically describe the missing features in terms of reward function. During the simulation, the model evolves autonomously adapting its response to the available observations, while remaining consistent with the known physics of the system.
Collapse
Affiliation(s)
- A Alexiadis
- School of Chemical Engineering, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK.
| | - M J H Simmons
- School of Chemical Engineering, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
| | - K Stamatopoulos
- School of Chemical Engineering, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK.
| | - H K Batchelor
- College of Medical and Dental Sciences, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK.,Strathclyde Institute of Pharmacy and Biomedical Sciences, University of Strathclyde, 161 Cathedral Street, Glasgow, G4 0RE, UK
| | - I Moulitsas
- Centre for Computational Engineering Sciences, Cranfield University, Bedford, MK43 0AL, UK
| |
Collapse
|
81
|
Taswell CA, Costa VD, Basile BM, Pujara MS, Jones B, Manem N, Murray EA, Averbeck BB. Effects of Amygdala Lesions on Object-Based Versus Action-Based Learning in Macaques. Cereb Cortex 2020; 31:529-546. [PMID: 32954409 DOI: 10.1093/cercor/bhaa241] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2019] [Revised: 08/05/2020] [Accepted: 08/05/2020] [Indexed: 01/01/2023] Open
Abstract
The neural systems that underlie reinforcement learning (RL) allow animals to adapt to changes in their environment. In the present study, we examined the hypothesis that the amygdala would have a preferential role in learning the values of visual objects. We compared a group of monkeys (Macaca mulatta) with amygdala lesions to a group of unoperated controls on a two-armed bandit reversal learning task. The task had two conditions. In the What condition, the animals had to learn to select a visual object, independent of its location. And in the Where condition, the animals had to learn to saccade to a location, independent of the object at the location. In both conditions choice-outcome mappings reversed in the middle of the block. We found that monkeys with amygdala lesions had learning deficits in both conditions. Monkeys with amygdala lesions did not have deficits in learning to reverse choice-outcome mappings. Rather, amygdala lesions caused the monkeys to become overly sensitive to negative feedback which impaired their ability to consistently select the more highly valued action or object. These results imply that the amygdala is generally necessary for RL.
Collapse
Affiliation(s)
- Craig A Taswell
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA
| | - Vincent D Costa
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA
| | - Benjamin M Basile
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA
| | - Maia S Pujara
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA
| | - Breonda Jones
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA
| | - Nihita Manem
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA
| | - Elisabeth A Murray
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA
| | - Bruno B Averbeck
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA
| |
Collapse
|
82
|
Value-guided remapping of sensory cortex by lateral orbitofrontal cortex. Nature 2020; 585:245-250. [PMID: 32884146 DOI: 10.1038/s41586-020-2704-z] [Citation(s) in RCA: 78] [Impact Index Per Article: 19.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2019] [Accepted: 06/22/2020] [Indexed: 01/07/2023]
Abstract
Adaptive behaviour crucially depends on flexible decision-making, which in mammals relies on the frontal cortex, specifically the orbitofrontal cortex (OFC)1-9. How OFC encodes decision variables and instructs sensory areas to guide adaptive behaviour are key open questions. Here we developed a reversal learning task for head-fixed mice, monitored the activity of neurons of the lateral OFC using two-photon calcium imaging and investigated how OFC dynamically interacts with primary somatosensory cortex (S1). Mice learned to discriminate 'go' from 'no-go' tactile stimuli10,11 and adapt their behaviour upon reversal of stimulus-reward contingency ('rule switch'). Imaging individual neurons longitudinally across all behavioural phases revealed a distinct engagement of S1 and lateral OFC, with S1 neural activity reflecting initial task learning, whereas lateral OFC neurons responded saliently and transiently to the rule switch. We identified direct long-range projections from lateral OFC to S1 that can feed this activity back to S1 as value prediction error. This top-down signal updated sensory representations in S1 by functionally remapping responses in a subpopulation of neurons that was sensitive to reward history. Functional remapping crucially depended on top-down feedback as chemogenetic silencing of lateral OFC neurons disrupted reversal learning, as well as plasticity in S1. The dynamic interaction of lateral OFC with sensory cortex thus implements computations critical for value prediction that are history dependent and error based, providing plasticity essential for flexible decision-making.
Collapse
|
83
|
Averbeck BB, Murray EA. Hypothalamic Interactions with Large-Scale Neural Circuits Underlying Reinforcement Learning and Motivated Behavior. Trends Neurosci 2020; 43:681-694. [PMID: 32762959 PMCID: PMC7483858 DOI: 10.1016/j.tins.2020.06.006] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2020] [Revised: 06/02/2020] [Accepted: 06/19/2020] [Indexed: 02/02/2023]
Abstract
Biological agents adapt behavior to support the survival needs of the individual and the species. In this review we outline the anatomical, physiological, and computational processes that support reinforcement learning (RL). We describe two circuits in the primate brain that are linked to specific aspects of learning and goal-directed behavior. The ventral circuit, that includes the amygdala, ventral medial prefrontal cortex, and ventral striatum, has substantial connectivity with the hypothalamus. The dorsal circuit, that includes inferior parietal cortex, dorsal lateral prefrontal cortex, and the dorsal striatum, has minimal connectivity with the hypothalamus. The hypothalamic connectivity suggests distinct roles for these circuits. We propose that the ventral circuit defines behavioral goals, and the dorsal circuit orchestrates behavior to achieve those goals.
Collapse
Affiliation(s)
- Bruno B Averbeck
- Laboratory of Neuropsychology, National Institute of Mental Health (NIMH), National Institutes of Health, Bethesda, MD 20892-4415, USA.
| | - Elisabeth A Murray
- Laboratory of Neuropsychology, National Institute of Mental Health (NIMH), National Institutes of Health, Bethesda, MD 20892-4415, USA
| |
Collapse
|
84
|
Isozaki A, Harmon J, Zhou Y, Li S, Nakagawa Y, Hayashi M, Mikami H, Lei C, Goda K. AI on a chip. LAB ON A CHIP 2020; 20:3074-3090. [PMID: 32644061 DOI: 10.1039/d0lc00521e] [Citation(s) in RCA: 60] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/09/2023]
Abstract
Artificial intelligence (AI) has dramatically changed the landscape of science, industry, defence, and medicine in the last several years. Supported by considerably enhanced computational power and cloud storage, the field of AI has shifted from mostly theoretical studies in the discipline of computer science to diverse real-life applications such as drug design, material discovery, speech recognition, self-driving cars, advertising, finance, medical imaging, and astronomical observation, where AI-produced outcomes have been proven to be comparable or even superior to the performance of human experts. In these applications, what is essentially important for the development of AI is the data needed for machine learning. Despite its prominent importance, the very first process of the AI development, namely data collection and data preparation, is typically the most laborious task and is often a limiting factor of constructing functional AI algorithms. Lab-on-a-chip technology, in particular microfluidics, is a powerful platform for both the construction and implementation of AI in a large-scale, cost-effective, high-throughput, automated, and multiplexed manner, thereby overcoming the above bottleneck. On this platform, high-throughput imaging is a critical tool as it can generate high-content information (e.g., size, shape, structure, composition, interaction) of objects on a large scale. High-throughput imaging can also be paired with sorting and DNA/RNA sequencing to conduct a massive survey of phenotype-genotype relations whose data is too complex to analyze with traditional computational tools, but is analyzable with the power of AI. In addition to its function as a data provider, lab-on-a-chip technology can also be employed to implement the developed AI for accurate identification, characterization, classification, and prediction of objects in mixed, heterogeneous, or unknown samples. In this review article, motivated by the excellent synergy between AI and lab-on-a-chip technology, we outline fundamental elements, recent advances, future challenges, and emerging opportunities of AI with lab-on-a-chip technology or "AI on a chip" for short.
Collapse
Affiliation(s)
- Akihiro Isozaki
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan. and Kanagawa Institute of Industrial Science and Technology, Kanagawa 213-0012, Japan
| | - Jeffrey Harmon
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan.
| | - Yuqi Zhou
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan.
| | - Shuai Li
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan. and The Cambridge Centre for Data-Driven Discovery, Cambridge University, Cambridge CB3 0WA, UK
| | - Yuta Nakagawa
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan.
| | - Mika Hayashi
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan.
| | - Hideharu Mikami
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan.
| | - Cheng Lei
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan. and Institute of Technological Sciences, Wuhan University, Hubei 430072, China
| | - Keisuke Goda
- Department of Chemistry, University of Tokyo, Tokyo 113-0033, Japan. and Institute of Technological Sciences, Wuhan University, Hubei 430072, China and Department of Bioengineering, University of California, Los Angeles, California 90095, USA
| |
Collapse
|
85
|
Biologically Inspired Visual System Architecture for Object Recognition in Autonomous Systems. ALGORITHMS 2020. [DOI: 10.3390/a13070167] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Computer vision is currently one of the most exciting and rapidly evolving fields of science, which affects numerous industries. Research and development breakthroughs, mainly in the field of convolutional neural networks (CNNs), opened the way to unprecedented sensitivity and precision in object detection and recognition tasks. Nevertheless, the findings in recent years on the sensitivity of neural networks to additive noise, light conditions, and to the wholeness of the training dataset, indicate that this technology still lacks the robustness needed for the autonomous robotic industry. In an attempt to bring computer vision algorithms closer to the capabilities of a human operator, the mechanisms of the human visual system was analyzed in this work. Recent studies show that the mechanisms behind the recognition process in the human brain include continuous generation of predictions based on prior knowledge of the world. These predictions enable rapid generation of contextual hypotheses that bias the outcome of the recognition process. This mechanism is especially advantageous in situations of uncertainty, when visual input is ambiguous. In addition, the human visual system continuously updates its knowledge about the world based on the gaps between its prediction and the visual feedback. CNNs are feed forward in nature and lack such top-down contextual attenuation mechanisms. As a result, although they process massive amounts of visual information during their operation, the information is not transformed into knowledge that can be used to generate contextual predictions and improve their performance. In this work, an architecture was designed that aims to integrate the concepts behind the top-down prediction and learning processes of the human visual system with the state-of-the-art bottom-up object recognition models, e.g., deep CNNs. The work focuses on two mechanisms of the human visual system: anticipation-driven perception and reinforcement-driven learning. Imitating these top-down mechanisms, together with the state-of-the-art bottom-up feed-forward algorithms, resulted in an accurate, robust, and continuously improving target recognition model.
Collapse
|
86
|
Carlucho I, De Paula M, Acosta GG. An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots. ISA TRANSACTIONS 2020; 102:280-294. [PMID: 32085878 DOI: 10.1016/j.isatra.2020.02.017] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/05/2019] [Revised: 02/11/2020] [Accepted: 02/11/2020] [Indexed: 06/10/2023]
Abstract
Intelligent control systems are being developed for the control of plants with complex dynamics. However, the simplicity of the PID (proportional-integrative-derivative) controller makes it still widely used in industrial applications and robotics. This paper proposes an intelligent control system based on a deep reinforcement learning approach for self-adaptive multiple PID controllers for mobile robots. The proposed hybrid control strategy uses an actor-critic structure and it only receives low-level dynamic information as input and simultaneously estimates the multiple parameters or gains of the PID controllers. The proposed approach was tested in several simulated environments and in a real time robotic platform showing the feasibility of the approach for the low-level control of mobile robots. From the simulation and experimental results, our proposed approach demonstrated that it can be of aid by providing with behavior that can compensate or even adapt to changes in the uncertain environments providing a model free unsupervised solution. Also, a comparative study against other adaptive methods for multiple PIDs tuning is presented, showing a successful performance of the approach.
Collapse
Affiliation(s)
- Ignacio Carlucho
- INTELYMEC, Centro de Investigaciones en Física e Ingeniería del Centro CIFICEN - UNICEN - CICpBA - CONICET, 7400 Olavarría, Argentina.
| | - Mariano De Paula
- INTELYMEC, Centro de Investigaciones en Física e Ingeniería del Centro CIFICEN - UNICEN - CICpBA - CONICET, 7400 Olavarría, Argentina.
| | - Gerardo G Acosta
- INTELYMEC, Centro de Investigaciones en Física e Ingeniería del Centro CIFICEN - UNICEN - CICpBA - CONICET, 7400 Olavarría, Argentina.
| |
Collapse
|
87
|
Dynamic representations in networked neural systems. Nat Neurosci 2020; 23:908-917. [PMID: 32541963 DOI: 10.1038/s41593-020-0653-3] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2019] [Accepted: 05/12/2020] [Indexed: 11/08/2022]
Abstract
A group of neurons can generate patterns of activity that represent information about stimuli; subsequently, the group can transform and transmit activity patterns across synapses to spatially distributed areas. Recent studies in neuroscience have begun to independently address the two components of information processing: the representation of stimuli in neural activity and the transmission of information in networks that model neural interactions. Yet only recently are studies seeking to link these two types of approaches. Here we briefly review the two separate bodies of literature; we then review the recent strides made to address this gap. We continue with a discussion of how patterns of activity evolve from one representation to another, forming dynamic representations that unfold on the underlying network. Our goal is to offer a holistic framework for understanding and describing neural information representation and transmission while revealing exciting frontiers for future research.
Collapse
|
88
|
Baladron J, Hamker FH. Habit learning in hierarchical cortex-basal ganglia loops. Eur J Neurosci 2020; 52:4613-4638. [PMID: 32237250 DOI: 10.1111/ejn.14730] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2020] [Revised: 03/21/2020] [Accepted: 03/22/2020] [Indexed: 12/17/2022]
Abstract
How do the multiple cortico-basal ganglia-thalamo-cortical loops interact? Are they parallel and fully independent or controlled by an arbitrator, or are they hierarchically organized? We introduce here a set of four key concepts, integrated and evaluated by means of a neuro-computational model, that bring together current ideas regarding cortex-basal ganglia interactions in the context of habit learning. According to key concept 1, each loop learns to select an intermediate objective at a different abstraction level, moving from goals in the ventral striatum to motor in the putamen. Key concept 2 proposes that the cortex integrates the basal ganglia selection with environmental information regarding the achieved objective. Key concept 3 claims shortcuts between loops, and key concept 4 predicts that loops compute their own prediction error signal for learning. Computational benefits of the key concepts are demonstrated. Contrasting with former concepts of habit learning, the loops collaborate to select goal-directed actions while training slower shortcuts develops habitual responses.
Collapse
Affiliation(s)
- Javier Baladron
- Department of Computer Science, Chemnitz University of Technology, Chemnitz, Germany
| | - Fred H Hamker
- Department of Computer Science, Chemnitz University of Technology, Chemnitz, Germany
| |
Collapse
|
89
|
Bartolo R, Averbeck BB. Prefrontal Cortex Predicts State Switches during Reversal Learning. Neuron 2020; 106:1044-1054.e4. [PMID: 32315603 DOI: 10.1016/j.neuron.2020.03.024] [Citation(s) in RCA: 51] [Impact Index Per Article: 12.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2019] [Revised: 01/28/2020] [Accepted: 03/24/2020] [Indexed: 11/25/2022]
Abstract
Reinforcement learning allows organisms to predict future outcomes and to update their beliefs about value in the world. The dorsal-lateral prefrontal cortex (dlPFC) integrates information carried by reward circuits, which can be used to infer the current state of the world under uncertainty. Here, we explored the dlPFC computations related to updating current beliefs during stochastic reversal learning. We recorded the activity of populations up to 1,000 neurons, simultaneously, in two male macaques while they executed a two-armed bandit reversal learning task. Behavioral analyses using a Bayesian framework showed that animals inferred reversals and switched their choice preference rapidly, rather than slowly updating choice values, consistent with state inference. Furthermore, dlPFC neural populations accurately encoded choice preference switches. These results suggest that prefrontal neurons dynamically encode decisions associated with Bayesian subjective values, highlighting the role of the PFC in representing a belief about the current state of the world.
Collapse
Affiliation(s)
- Ramon Bartolo
- Laboratory of Neuropsychology, National Institute of Mental Health/National Institutes of Health, Bethesda, MD 20892-4415, USA.
| | - Bruno B Averbeck
- Laboratory of Neuropsychology, National Institute of Mental Health/National Institutes of Health, Bethesda, MD 20892-4415, USA
| |
Collapse
|
90
|
|
91
|
Alber M, Buganza Tepole A, Cannon WR, De S, Dura-Bernal S, Garikipati K, Karniadakis G, Lytton WW, Perdikaris P, Petzold L, Kuhl E. Integrating machine learning and multiscale modeling-perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences. NPJ Digit Med 2019; 2:115. [PMID: 31799423 PMCID: PMC6877584 DOI: 10.1038/s41746-019-0193-y] [Citation(s) in RCA: 160] [Impact Index Per Article: 32.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2019] [Accepted: 11/01/2019] [Indexed: 12/12/2022] Open
Abstract
Fueled by breakthrough technology developments, the biological, biomedical, and behavioral sciences are now collecting more data than ever before. There is a critical need for time- and cost-efficient strategies to analyze and interpret these data to advance human health. The recent rise of machine learning as a powerful technique to integrate multimodality, multifidelity data, and reveal correlations between intertwined phenomena presents a special opportunity in this regard. However, machine learning alone ignores the fundamental laws of physics and can result in ill-posed problems or non-physical solutions. Multiscale modeling is a successful strategy to integrate multiscale, multiphysics data and uncover mechanisms that explain the emergence of function. However, multiscale modeling alone often fails to efficiently combine large datasets from different sources and different levels of resolution. Here we demonstrate that machine learning and multiscale modeling can naturally complement each other to create robust predictive models that integrate the underlying physics to manage ill-posed problems and explore massive design spaces. We review the current literature, highlight applications and opportunities, address open questions, and discuss potential challenges and limitations in four overarching topical areas: ordinary differential equations, partial differential equations, data-driven approaches, and theory-driven approaches. Towards these goals, we leverage expertise in applied mathematics, computer science, computational biology, biophysics, biomechanics, engineering mechanics, experimentation, and medicine. Our multidisciplinary perspective suggests that integrating machine learning and multiscale modeling can provide new insights into disease mechanisms, help identify new targets and treatment strategies, and inform decision making for the benefit of human health.
Collapse
Affiliation(s)
- Mark Alber
- Department of Mathematics, University of California, Riverside, CA USA
| | | | - William R. Cannon
- Computational Biology Group, Pacific Northwest National Laboratory, Richland, WA USA
| | - Suvranu De
- Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, Troy, NY USA
| | | | - Krishna Garikipati
- Departments of Mechanical Engineering and Mathematics, University of Michigan, Ann Arbor, MI USA
| | | | - William W. Lytton
- SUNY Downstate Medical Center and Kings County Hospital, Brooklyn, NY USA
| | - Paris Perdikaris
- Department of Mechanical Engineering, University of Pennsylvania, Philadelphia, PA USA
| | - Linda Petzold
- Department of Computer Science and Mechanical Engineering, University of California, Santa Barbara, CA USA
| | - Ellen Kuhl
- Departments of Mechanical Engineering and Bioengineering, Stanford University, Stanford, CA USA
| |
Collapse
|
92
|
Costa VD. Of Pathways, Processes, and Orbitofrontal Cortex. Neuron 2019; 103:556-558. [PMID: 31437451 DOI: 10.1016/j.neuron.2019.07.034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Abstract
Orbitofrontal cortex (OFC) predicts the consequences of our actions and updates our expectations based on experienced outcomes. In this issue of Neuron, Groman et al. (2019) precisely ablate pathways between the OFC, amygdala, and nucleus accumbens to reveal their separable contributions to reinforcement learning.
Collapse
Affiliation(s)
- Vincent D Costa
- Department of Behavioral Neuroscience, Oregon Health & Science University, Portland, OR 97239, USA; Oregon National Primate Research Center, Portland, OR 97006, USA.
| |
Collapse
|
93
|
Pitti A, Quoy M, Lavandier C, Boucenna S. Gated spiking neural network using Iterative Free-Energy Optimization and rank-order coding for structure learning in memory sequences (INFERNO GATE). Neural Netw 2019; 121:242-258. [PMID: 31581065 DOI: 10.1016/j.neunet.2019.09.023] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2019] [Revised: 09/16/2019] [Accepted: 09/17/2019] [Indexed: 11/16/2022]
Abstract
We present a framework based on iterative free-energy optimization with spiking neural networks for modeling the fronto-striatal system (PFC-BG) for the generation and recall of audio memory sequences. In line with neuroimaging studies carried out in the PFC, we propose a genuine coding strategy using the gain-modulation mechanism to represent abstract sequences based solely on the rank and location of items within them. Based on this mechanism, we show that we can construct a repertoire of neurons sensitive to the temporal structure in sequences from which we can represent any novel sequences. Free-energy optimization is then used to explore and to retrieve the missing indices of the items in the correct order for executive control and compositionality. We show that the gain-modulation mechanism permits the network to be robust to variabilities and to have long-term dependencies as it implements a gated recurrent neural network. This model, called Inferno Gate, is an extension of the neural architecture Inferno standing for Iterative Free-Energy Optimization of Recurrent Neural Networks with Gating or Gain-modulation. In experiments performed with an audio database of ten thousand MFCC vectors, Inferno Gate is capable of encoding efficiently and retrieving chunks of fifty items length. We then discuss the potential of our network to model the features of working memory in the PFC-BG loop for structural learning, goal-direction and hierarchical reinforcement learning.
Collapse
Affiliation(s)
- Alexandre Pitti
- Laboratoire ETIS UMR 8051, Université Paris-Seine, Université de Cergy-Pontoise, ENSEA, CNRS, France.
| | - Mathias Quoy
- Laboratoire ETIS UMR 8051, Université Paris-Seine, Université de Cergy-Pontoise, ENSEA, CNRS, France.
| | - Catherine Lavandier
- Laboratoire ETIS UMR 8051, Université Paris-Seine, Université de Cergy-Pontoise, ENSEA, CNRS, France.
| | - Sofiane Boucenna
- Laboratoire ETIS UMR 8051, Université Paris-Seine, Université de Cergy-Pontoise, ENSEA, CNRS, France.
| |
Collapse
|
94
|
Xiong Z, Wang D, Liu X, Zhong F, Wan X, Li X, Li Z, Luo X, Chen K, Jiang H, Zheng M. Pushing the Boundaries of Molecular Representation for Drug Discovery with the Graph Attention Mechanism. J Med Chem 2019; 63:8749-8760. [DOI: 10.1021/acs.jmedchem.9b00959] [Citation(s) in RCA: 151] [Impact Index Per Article: 30.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Affiliation(s)
- Zhaoping Xiong
- Shanghai Institute for Advanced Immunochemical Studies, and School of Life Science and Technology, ShanghaiTech University, Shanghai 200031, China
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
- University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China
| | - Dingyan Wang
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
- University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China
| | - Xiaohong Liu
- Shanghai Institute for Advanced Immunochemical Studies, and School of Life Science and Technology, ShanghaiTech University, Shanghai 200031, China
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
| | - Feisheng Zhong
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
- University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China
| | - Xiaozhe Wan
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
- University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China
| | - Xutong Li
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
- University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China
| | - Zhaojun Li
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
| | - Xiaomin Luo
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
| | - Kaixian Chen
- Shanghai Institute for Advanced Immunochemical Studies, and School of Life Science and Technology, ShanghaiTech University, Shanghai 200031, China
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
| | - Hualiang Jiang
- Shanghai Institute for Advanced Immunochemical Studies, and School of Life Science and Technology, ShanghaiTech University, Shanghai 200031, China
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
| | - Mingyue Zheng
- Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, 555 Zuchongzhi Road, Shanghai 201203, China
| |
Collapse
|
95
|
Zhao KW, Kao WH, Wu KH, Kao YJ. Generation of ice states through deep reinforcement learning. Phys Rev E 2019; 99:062106. [PMID: 31330736 DOI: 10.1103/physreve.99.062106] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2019] [Indexed: 11/07/2022]
Abstract
We present a deep reinforcement learning framework where a machine agent is trained to search for a policy to generate a ground state for the square ice model by exploring the physical environment. After training, the agent is capable of proposing a sequence of local moves to achieve the goal. Analysis of the trained policy and the state value function indicates that the ice rule and loop-closing condition are learned without prior knowledge. We test the trained policy as a sampler in the Markov chain Monte Carlo and benchmark against the baseline loop algorithm. This framework can be generalized to other models with topological constraints where generation of constraint-preserving states is difficult.
Collapse
Affiliation(s)
- Kai-Wen Zhao
- Department of Physics and Center for Theoretical Physics, National Taiwan University, Taipei 10607, Taiwan
| | - Wen-Han Kao
- Department of Physics and Center for Theoretical Physics, National Taiwan University, Taipei 10607, Taiwan
| | - Kai-Hsin Wu
- Department of Physics and Center for Theoretical Physics, National Taiwan University, Taipei 10607, Taiwan
| | - Ying-Jer Kao
- Department of Physics and Center for Theoretical Physics, National Taiwan University, Taipei 10607, Taiwan.,National Center for Theoretical Sciences, National Tsing Hua University, Hsin-Chu 30013, Taiwan.,Department of Physics, Boston University, 590 Commonwealth Avenue, Boston, Massachusetts 02215, USA.,Kavli Institute for Theoretical Physics, University of California, Santa Barbara, California 93106-4030, USA
| |
Collapse
|