1
|
Srivastava S, Wang WY, Eckstein MP. Emergent human-like covert attention in feedforward convolutional neural networks. Curr Biol 2024; 34:579-593.e12. [PMID: 38244541 DOI: 10.1016/j.cub.2023.12.058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2023] [Revised: 10/09/2023] [Accepted: 12/19/2023] [Indexed: 01/22/2024]
Abstract
Covert attention allows the selection of locations or features of the visual scene without moving the eyes. Cues and contexts predictive of a target's location orient covert attention and improve perceptual performance. The performance benefits are widely attributed to theories of covert attention as a limited resource, zoom, spotlight, or weighting of visual information. However, such concepts are difficult to map to neuronal populations. We show that a feedforward convolutional neural network (CNN) trained on images to optimize target detection accuracy and with no explicit incorporation of an attention mechanism, a limited resource, or feedback connections learns to utilize cues and contexts in the three most prominent covert attention tasks (Posner cueing, set size effects in search, and contextual cueing) and predicts the cue/context influences on human accuracy. The CNN's cueing/context effects generalize across network training schemes, to peripheral and central pre-cues, discrimination tasks, and reaction time measures, and critically do not vary with reductions in network resources (size). The CNN shows comparable cueing/context effects to a model that optimally uses image information to make decisions (Bayesian ideal observer) but generalizes these effects to cue instances unseen during training. Together, the findings suggest that human-like behavioral signatures of covert attention in the three landmark paradigms might be an emergent property of task accuracy optimization in neuronal populations without positing limited attentional resources. The findings might explain recent behavioral results showing cueing and context effects across a variety of simple organisms with no neocortex, from archerfish to fruit flies.
Collapse
Affiliation(s)
- Sudhanshu Srivastava
- Graduate Program in Dynamical Neuroscience, University of California, Santa Barbara, Santa Barbara, CA 93106, USA; Institute for Collaborative Biotechnologies, University of California, Santa Barbara, Santa Barbara, CA 93106, USA.
| | - William Yang Wang
- Department of Computer Science, University of California, Santa Barbara, Santa Barbara, CA 93106, USA; Institute for Collaborative Biotechnologies, University of California, Santa Barbara, Santa Barbara, CA 93106, USA.
| | - Miguel P Eckstein
- Graduate Program in Dynamical Neuroscience, University of California, Santa Barbara, Santa Barbara, CA 93106, USA; Department of Psychological and Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA 93106, USA; Department of Computer Science, University of California, Santa Barbara, Santa Barbara, CA 93106, USA; Department of Electrical and Computer Engineering, University of California, Santa Barbara, Santa Barbara, CA 93106, USA; Institute for Collaborative Biotechnologies, University of California, Santa Barbara, Santa Barbara, CA 93106, USA.
| |
Collapse
|
2
|
Nicholson DA, Prinz AA. Could simplified stimuli change how the brain performs visual search tasks? A deep neural network study. J Vis 2022; 22:3. [PMID: 35675057 PMCID: PMC9187944 DOI: 10.1167/jov.22.7.3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2021] [Accepted: 05/04/2022] [Indexed: 11/24/2022] Open
Abstract
Visual search is a complex behavior influenced by many factors. To control for these factors, many studies use highly simplified stimuli. However, the statistics of these stimuli are very different from the statistics of the natural images that the human visual system is optimized by evolution and experience to perceive. Could this difference change search behavior? If so, simplified stimuli may contribute to effects typically attributed to cognitive processes, such as selective attention. Here we use deep neural networks to test how optimizing models for the statistics of one distribution of images constrains performance on a task using images from a different distribution. We train four deep neural network architectures on one of three source datasets-natural images, faces, and x-ray images-and then adapt them to a visual search task using simplified stimuli. This adaptation produces models that exhibit performance limitations similar to humans, whereas models trained on the search task alone exhibit no such limitations. However, we also find that deep neural networks trained to classify natural images exhibit similar limitations when adapted to a search task that uses a different set of natural images. Therefore, the distribution of data alone cannot explain this effect. We discuss how future work might integrate an optimization-based approach into existing models of visual search behavior.
Collapse
Affiliation(s)
- David A Nicholson
- Emory University, Department of Biology, O. Wayne Rollins Research Center, Atlanta, Georgia
| | - Astrid A Prinz
- Emory University, Department of Biology, O. Wayne Rollins Research Center, Atlanta, Georgia
| |
Collapse
|
3
|
Lev-Ari T, Beeri H, Gutfreund Y. The Ecological View of Selective Attention. Front Integr Neurosci 2022; 16:856207. [PMID: 35391754 PMCID: PMC8979825 DOI: 10.3389/fnint.2022.856207] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2022] [Accepted: 02/24/2022] [Indexed: 11/16/2022] Open
Abstract
Accumulating evidence is supporting the hypothesis that our selective attention is a manifestation of mechanisms that evolved early in evolution and are shared by many organisms from different taxa. This surge of new data calls for the re-examination of our notions about attention, which have been dominated mostly by human psychology. Here, we present an hypothesis that challenges, based on evolutionary grounds, a common view of attention as a means to manage limited brain resources. We begin by arguing that evolutionary considerations do not favor the basic proposition of the limited brain resources view of attention, namely, that the capacity of the sensory organs to provide information exceeds the capacity of the brain to process this information. Moreover, physiological studies in animals and humans show that mechanisms of selective attention are highly demanding of brain resources, making it paradoxical to see attention as a means to release brain resources. Next, we build on the above arguments to address the question why attention evolved in evolution. We hypothesize that, to a certain extent, limiting sensory processing is adaptive irrespective of brain capacity. We call this hypothesis the ecological view of attention (EVA) because it is centered on interactions of an animal with its environment rather than on internal brain resources. In its essence is the notion that inherently noisy and degraded sensory inputs serve the animal's adaptive, dynamic interactions with its environment. Attention primarily functions to resolve behavioral conflicts and false distractions. Hence, we evolved to focus on a particular target at the expense of others, not because of internal limitations, but to ensure that behavior is properly oriented and committed to its goals. Here, we expand on this notion and review evidence supporting it. We show how common results in human psychophysics and physiology can be reconciled with an EVA and discuss possible implications of the notion for interpreting current results and guiding future research.
Collapse
Affiliation(s)
| | | | - Yoram Gutfreund
- The Ruth and Bruce Rappaport Faculty of Medicine and Research Institute, The Technion, Haifa, Israel
| |
Collapse
|
4
|
Souto D, Kerzel D. Visual selective attention and the control of tracking eye movements: a critical review. J Neurophysiol 2021; 125:1552-1576. [DOI: 10.1152/jn.00145.2019] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
People’s eyes are directed at objects of interest with the aim of acquiring visual information. However, processing this information is constrained in capacity, requiring task-driven and salience-driven attentional mechanisms to select few among the many available objects. A wealth of behavioral and neurophysiological evidence has demonstrated that visual selection and the motor selection of saccade targets rely on shared mechanisms. This coupling supports the premotor theory of visual attention put forth more than 30 years ago, postulating visual selection as a necessary stage in motor selection. In this review, we examine to which extent the coupling of visual and motor selection observed with saccades is replicated during ocular tracking. Ocular tracking combines catch-up saccades and smooth pursuit to foveate a moving object. We find evidence that ocular tracking requires visual selection of the speed and direction of the moving target, but the position of the motion signal may not coincide with the position of the pursuit target. Further, visual and motor selection can be spatially decoupled when pursuit is initiated (open-loop pursuit). We propose that a main function of coupled visual and motor selection is to serve the coordination of catch-up saccades and pursuit eye movements. A simple race-to-threshold model is proposed to explain the variable coupling of visual selection during pursuit, catch-up and regular saccades, while generating testable predictions. We discuss pending issues, such as disentangling visual selection from preattentive visual processing and response selection, and the pinpointing of visual selection mechanisms, which have begun to be addressed in the neurophysiological literature.
Collapse
Affiliation(s)
- David Souto
- Department of Neuroscience, Psychology and Behaviour, University of Leicester, Leicester, United Kingdom
| | - Dirk Kerzel
- Faculté de Psychologie et des Sciences de l’Education, University of Geneva, Geneva, Switzerland
| |
Collapse
|
5
|
MacInnes WJ, Hunt AR, Clarke ADF, Dodd MD. A Generative Model of Cognitive State from Task and Eye Movements. Cognit Comput 2018; 10:703-717. [PMID: 30740186 DOI: 10.1007/s12559-018-9558-9] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
Abstract
The early eye tracking studies of Yarbus provided descriptive evidence that an observer's task influences patterns of eye movements, leading to the tantalizing prospect that an observer's intentions could be inferred from their saccade behavior. We investigate the predictive value of task and eye movement properties by creating a computational cognitive model of saccade selection based on instructed task and internal cognitive state using a Dynamic Bayesian Network (DBN). Understanding how humans generate saccades under different conditions and cognitive sets links recent work on salience models of low-level vision with higher level cognitive goals. This model provides a Bayesian, cognitive approach to top-down transitions in attentional set in pre-frontal areas along with vector-based saccade generation from the superior colliculus. Our approach is to begin with eye movement data that has previously been shown to differ across task. We first present an analysis of the extent to which individual saccadic features are diagnostic of an observer's task. Second, we use those features to infer an underlying cognitive state that potentially differs from the instructed task. Finally, we demonstrate how changes of cognitive state over time can be incorporated into a generative model of eye movement vectors without resorting to an external decision homunculus. Internal cognitive state frees the model from the assumption that instructed task is the only factor influencing observers' saccadic behavior. While the inclusion of hidden temporal state does not improve the classification accuracy of the model, it does allow accurate prediction of saccadic sequence results observed in search paradigms. Given the generative nature of this model, it is capable of saccadic simulation in real time. We demonstrated that the properties from its generated saccadic vectors closely match those of human observers given a particular task and cognitive state. Many current models of vision focus entirely on bottom-up salience to produce estimates of spatial "areas of interest" within a visual scene. While a few recent models do add top-down knowledge and task information, we believe our contribution is important in three key ways. First, we incorporate task as learned attentional sets that are capable of self-transition given only information available to the visual system. This matches influential theories of bias signals by (Miller and Cohen Annu Rev Neurosci 24:167-202, 2001) and implements selection of state without simply shifting the decision to an external homunculus. Second, our model is generative and capable of predicting sequence artifacts in saccade generation like those found in visual search. Third, our model generates relative saccadic vector information as opposed to absolute spatial coordinates. This matches more closely the internal saccadic representations as they are generated in the superior colliculus.
Collapse
Affiliation(s)
- W Joseph MacInnes
- School of Psychology, National Research University Higher School of Economics, Moscow, Russian Federation
| | - Amelia R Hunt
- School of Psychology, University of Aberdeen, Aberdeen, UK
| | | | | |
Collapse
|
8
|
Viswanathan V, Sheppard JP, Kim BW, Plantz CL, Ying H, Lee MJ, Raman K, Mulhern FJ, Block MP, Calder B, Lee S, Mortensen DT, Blood AJ, Breiter HC. A Quantitative Relationship between Signal Detection in Attention and Approach/Avoidance Behavior. Front Psychol 2017; 8:122. [PMID: 28270776 PMCID: PMC5318395 DOI: 10.3389/fpsyg.2017.00122] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2016] [Accepted: 01/17/2017] [Indexed: 11/13/2022] Open
Abstract
This study examines how the domains of reward and attention, which are often studied as independent processes, in fact interact at a systems level. We operationalize divided attention with a continuous performance task and variables from signal detection theory (SDT), and reward/aversion with a keypress task measuring approach/avoidance in the framework of relative preference theory (RPT). Independent experiments with the same subjects showed a significant association between one SDT and two RPT variables, visualized as a three-dimensional structure. Holding one of these three variables constant, further showed a significant relationship between a loss aversion-like metric from the approach/avoidance task, and the response bias observed during the divided attention task. These results indicate that a more liberal response bias under signal detection (i.e., a higher tolerance for noise, resulting in a greater proportion of false alarms) is associated with higher "loss aversion." Furthermore, our functional model suggests a mechanism for processing constraints with divided attention and reward/aversion. Together, our results argue for a systematic relationship between divided attention and reward/aversion processing in humans.
Collapse
Affiliation(s)
- Vijay Viswanathan
- Medill Integrated Marketing Communications, Northwestern UniversityEvanston, IL, USA; Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA
| | - John P Sheppard
- Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA; Warren Wright Adolescent Center, Department of Psychiatry and Behavioral Sciences, Northwestern University Feinberg School of MedicineChicago, IL, USA
| | - Byoung W Kim
- Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA; Warren Wright Adolescent Center, Department of Psychiatry and Behavioral Sciences, Northwestern University Feinberg School of MedicineChicago, IL, USA; Laboratory of Neuroimaging and Genetics, and Mood and Motor Control Laboratory, Department of Psychiatry, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; MGH Center for Translational Research in Prescription Drug Abuse, Department of Anesthesia, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; Northwestern University and Massachusetts General Hospital Phenotype Genotype Project in Addiction and Mood DisordersChicago, IL, USA
| | - Christopher L Plantz
- Department of Materials Science and Engineering, University of Illinois at Urbana-Champaign Urbana, IL, USA
| | - Hao Ying
- Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA; Department of Electrical Engineering, Wayne State UniversityDetroit, MI, USA
| | - Myung J Lee
- Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA; Warren Wright Adolescent Center, Department of Psychiatry and Behavioral Sciences, Northwestern University Feinberg School of MedicineChicago, IL, USA; Laboratory of Neuroimaging and Genetics, and Mood and Motor Control Laboratory, Department of Psychiatry, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; MGH Center for Translational Research in Prescription Drug Abuse, Department of Anesthesia, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; Northwestern University and Massachusetts General Hospital Phenotype Genotype Project in Addiction and Mood DisordersChicago, IL, USA
| | - Kalyan Raman
- Medill Integrated Marketing Communications, Northwestern UniversityEvanston, IL, USA; Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA
| | - Frank J Mulhern
- Medill Integrated Marketing Communications, Northwestern UniversityEvanston, IL, USA; Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA
| | - Martin P Block
- Medill Integrated Marketing Communications, Northwestern UniversityEvanston, IL, USA; Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA
| | - Bobby Calder
- Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA; Department of Marketing, Kellogg School of Management, Northwestern UniversityEvanston, IL, USA
| | - Sang Lee
- Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA; Laboratory of Neuroimaging and Genetics, and Mood and Motor Control Laboratory, Department of Psychiatry, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; MGH Center for Translational Research in Prescription Drug Abuse, Department of Anesthesia, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; Northwestern University and Massachusetts General Hospital Phenotype Genotype Project in Addiction and Mood DisordersChicago, IL, USA
| | - Dale T Mortensen
- Department of Economics, Northwestern University College of Arts and Sciences Evanston, IL, USA
| | - Anne J Blood
- Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA; Laboratory of Neuroimaging and Genetics, and Mood and Motor Control Laboratory, Department of Psychiatry, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; MGH Center for Translational Research in Prescription Drug Abuse, Department of Anesthesia, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; Northwestern University and Massachusetts General Hospital Phenotype Genotype Project in Addiction and Mood DisordersChicago, IL, USA
| | - Hans C Breiter
- Applied Neuromarketing Consortium, Medill, Kellogg, and Feinberg Schools, Northwestern UniversityEvanston, IL, USA; Warren Wright Adolescent Center, Department of Psychiatry and Behavioral Sciences, Northwestern University Feinberg School of MedicineChicago, IL, USA; Laboratory of Neuroimaging and Genetics, and Mood and Motor Control Laboratory, Department of Psychiatry, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; MGH Center for Translational Research in Prescription Drug Abuse, Department of Anesthesia, Massachusetts General Hospital and Harvard Medical SchoolBoston, MA, USA; Northwestern University and Massachusetts General Hospital Phenotype Genotype Project in Addiction and Mood DisordersChicago, IL, USA
| |
Collapse
|