1
|
Alefantis P, Lakshminarasimhan K, Avila E, Noel JP, Pitkow X, Angelaki DE. Sensory Evidence Accumulation Using Optic Flow in a Naturalistic Navigation Task. J Neurosci 2022; 42:5451-5462. [PMID: 35641186 PMCID: PMC9270913 DOI: 10.1523/jneurosci.2203-21.2022] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2021] [Revised: 04/01/2022] [Accepted: 04/22/2022] [Indexed: 11/21/2022] Open
Abstract
Sensory evidence accumulation is considered a hallmark of decision-making in noisy environments. Integration of sensory inputs has been traditionally studied using passive stimuli, segregating perception from action. Lessons learned from this approach, however, may not generalize to ethological behaviors like navigation, where there is an active interplay between perception and action. We designed a sensory-based sequential decision task in virtual reality in which humans and monkeys navigated to a memorized location by integrating optic flow generated by their own joystick movements. A major challenge in such closed-loop tasks is that subjects' actions will determine future sensory input, causing ambiguity about whether they rely on sensory input rather than expectations based solely on a learned model of the dynamics. To test whether subjects integrated optic flow over time, we used three independent experimental manipulations, unpredictable optic flow perturbations, which pushed subjects off their trajectory; gain manipulation of the joystick controller, which changed the consequences of actions; and manipulation of the optic flow density, which changed the information borne by sensory evidence. Our results suggest that both macaques (male) and humans (female/male) relied heavily on optic flow, thereby demonstrating a critical role for sensory evidence accumulation during naturalistic action-perception closed-loop tasks.SIGNIFICANCE STATEMENT The temporal integration of evidence is a fundamental component of mammalian intelligence. Yet, it has traditionally been studied using experimental paradigms that fail to capture the closed-loop interaction between actions and sensations inherent in real-world continuous behaviors. These conventional paradigms use binary decision tasks and passive stimuli with statistics that remain stationary over time. Instead, we developed a naturalistic visuomotor visual navigation paradigm that mimics the causal structure of real-world sensorimotor interactions and probed the extent to which participants integrate sensory evidence by adding task manipulations that reveal complementary aspects of the computation.
Collapse
Affiliation(s)
- Panos Alefantis
- Center for Neural Science, New York University, New York, New York 10003
| | | | - Eric Avila
- Center for Neural Science, New York University, New York, New York 10003
| | - Jean-Paul Noel
- Center for Neural Science, New York University, New York, New York 10003
| | - Xaq Pitkow
- Department of Neuroscience, Baylor College of Medicine, Houston, Texas 77030
- Department of Electrical and Computer Engineering, Rice University, Houston, Texas 77005-1892
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, Texas 77030
| | - Dora E Angelaki
- Center for Neural Science, New York University, New York, New York 10003
- Tandon School of Engineering, New York University, New York, New York 11201
| |
Collapse
|
2
|
Mahony NO, Campbell S, Krpalkova L, Carvalho A, Walsh J, Riordan D. Representation Learning for Fine-Grained Change Detection. SENSORS (BASEL, SWITZERLAND) 2021; 21:4486. [PMID: 34209075 PMCID: PMC8271830 DOI: 10.3390/s21134486] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/29/2021] [Revised: 06/16/2021] [Accepted: 06/25/2021] [Indexed: 11/16/2022]
Abstract
Fine-grained change detection in sensor data is very challenging for artificial intelligence though it is critically important in practice. It is the process of identifying differences in the state of an object or phenomenon where the differences are class-specific and are difficult to generalise. As a result, many recent technologies that leverage big data and deep learning struggle with this task. This review focuses on the state-of-the-art methods, applications, and challenges of representation learning for fine-grained change detection. Our research focuses on methods of harnessing the latent metric space of representation learning techniques as an interim output for hybrid human-machine intelligence. We review methods for transforming and projecting embedding space such that significant changes can be communicated more effectively and a more comprehensive interpretation of underlying relationships in sensor data is facilitated. We conduct this research in our work towards developing a method for aligning the axes of latent embedding space with meaningful real-world metrics so that the reasoning behind the detection of change in relation to past observations may be revealed and adjusted. This is an important topic in many fields concerned with producing more meaningful and explainable outputs from deep learning and also for providing means for knowledge injection and model calibration in order to maintain user confidence.
Collapse
Affiliation(s)
- Niall O’ Mahony
- Lero—The Irish Software Research Centre, V92 CX88 Tralee, Ireland; (S.C.); (L.K.); (A.C.); (J.W.); (D.R.)
- Department of Agricultural and Manufacturing Engineering, School of Science Technology Engineering and Maths (STEM), Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
- IMaR Research Centre, Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
| | - Sean Campbell
- Lero—The Irish Software Research Centre, V92 CX88 Tralee, Ireland; (S.C.); (L.K.); (A.C.); (J.W.); (D.R.)
- Department of Agricultural and Manufacturing Engineering, School of Science Technology Engineering and Maths (STEM), Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
- IMaR Research Centre, Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
| | - Lenka Krpalkova
- Lero—The Irish Software Research Centre, V92 CX88 Tralee, Ireland; (S.C.); (L.K.); (A.C.); (J.W.); (D.R.)
- Department of Agricultural and Manufacturing Engineering, School of Science Technology Engineering and Maths (STEM), Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
- IMaR Research Centre, Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
| | - Anderson Carvalho
- Lero—The Irish Software Research Centre, V92 CX88 Tralee, Ireland; (S.C.); (L.K.); (A.C.); (J.W.); (D.R.)
- Department of Agricultural and Manufacturing Engineering, School of Science Technology Engineering and Maths (STEM), Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
- IMaR Research Centre, Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
| | - Joseph Walsh
- Lero—The Irish Software Research Centre, V92 CX88 Tralee, Ireland; (S.C.); (L.K.); (A.C.); (J.W.); (D.R.)
- Department of Agricultural and Manufacturing Engineering, School of Science Technology Engineering and Maths (STEM), Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
- IMaR Research Centre, Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
| | - Daniel Riordan
- Lero—The Irish Software Research Centre, V92 CX88 Tralee, Ireland; (S.C.); (L.K.); (A.C.); (J.W.); (D.R.)
- Department of Agricultural and Manufacturing Engineering, School of Science Technology Engineering and Maths (STEM), Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
- IMaR Research Centre, Kerry Campus, Munster Technological University, V92 CX88 Tralee, Ireland
| |
Collapse
|
3
|
Górska U, Rupp A, Celikel T, Englitz B. Assessing the state of consciousness for individual patients using complex, statistical stimuli. NEUROIMAGE-CLINICAL 2020; 29:102471. [PMID: 33388561 PMCID: PMC7788231 DOI: 10.1016/j.nicl.2020.102471] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/20/2019] [Revised: 09/29/2020] [Accepted: 10/14/2020] [Indexed: 12/01/2022]
Abstract
Patients with prolonged disorders of consciousness (PDOC) are often unable to communicate their state of consciousness. Determining the latter is essential for the patient's care and prospects of recovery. Auditory stimulation in combination with neural recordings is a promising technique towards an objective assessment of conscious awareness. Here, we investigated the potential of complex, acoustic stimuli to elicit EEG responses suitable for classifying multiple subject groups, from unconscious to responding. We presented naturalistic auditory textures with unexpectedly changing statistics to human listeners. Awake, active listeners were asked to indicate the change by button press, while all other groups (awake passive, asleep, minimally conscious state (MCS), and unresponsive wakefulness syndrome (UWS)) listened passively. We quantified the evoked potential at stimulus onset and change in stimulus statistics, as well as the complexity of neural response during the change of stimulus statistics. On the group level, onset and change potentials classified patients and healthy controls successfully but failed to differentiate between the UWS and MCS groups. Conversely, the Lempel-Ziv complexity of the scalp-level potential allowed reliable differentiation between UWS and MCS even for individual subjects, when compared with the clinical assessment aligned to the EEG measurements. The accuracy appears to improve further when taking the latest available clinical diagnosis into account. In summary, EEG signal complexity during onset and changes in complex acoustic stimuli provides an objective criterion for distinguishing states of consciousness in clinical patients. These results suggest EEG-recordings as a cost-effective tool to choose appropriate treatments for non-responsive PDOC patients.
Collapse
Affiliation(s)
- U Górska
- Computational Neuroscience Laboratory, Department of Neurophysiology, Donders Institute, Radboud University Nijmegen, The Netherlands; Psychophysiology Laboratory, Institute of Psychology, Jagiellonian University, Krakow, Poland; Smoluchowski Institute of Physics, Jagiellonian University, Krakow, Poland.
| | - A Rupp
- Section of Biomagnetism, Department of Neurology, University of Heidelberg, Heidelberg, Germany
| | - T Celikel
- Computational Neuroscience Laboratory, Department of Neurophysiology, Donders Institute, Radboud University Nijmegen, The Netherlands
| | - B Englitz
- Computational Neuroscience Laboratory, Department of Neurophysiology, Donders Institute, Radboud University Nijmegen, The Netherlands.
| |
Collapse
|
4
|
Hughes S, Celikel T. Prominent Inhibitory Projections Guide Sensorimotor Computation: An Invertebrate Perspective. Bioessays 2019; 41:e1900088. [DOI: 10.1002/bies.201900088] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2019] [Revised: 07/17/2019] [Indexed: 12/17/2022]
Affiliation(s)
- Samantha Hughes
- HAN BioCentreHAN University of Applied Sciences Nijmegen 6525EM The Netherlands
| | - Tansu Celikel
- Department of Neurophysiology, Donders Institute for Brain Cognition and BehaviourRadboud University Nijmegen 6525AJ The Netherlands
| |
Collapse
|