1
|
Bays PM, Schneegans S, Ma WJ, Brady TF. Representation and computation in visual working memory. Nat Hum Behav 2024; 8:1016-1034. [PMID: 38849647 DOI: 10.1038/s41562-024-01871-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2022] [Accepted: 03/22/2024] [Indexed: 06/09/2024]
Abstract
The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.
Collapse
Affiliation(s)
- Paul M Bays
- Department of Psychology, University of Cambridge, Cambridge, UK
| | | | - Wei Ji Ma
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA
| | - Timothy F Brady
- Department of Psychology, University of California, San Diego, La Jolla, CA, USA.
| |
Collapse
|
2
|
Abstract
Probing memory of a complex visual image within a few hundred milliseconds after its disappearance reveals significantly greater fidelity of recall than if the probe is delayed by as little as a second. Classically interpreted, the former taps into a detailed but rapidly decaying visual sensory or 'iconic' memory (IM), while the latter relies on capacity-limited but comparatively stable visual working memory (VWM). While iconic decay and VWM capacity have been extensively studied independently, currently no single framework quantitatively accounts for the dynamics of memory fidelity over these time scales. Here, we extend a stationary neural population model of VWM with a temporal dimension, incorporating rapid sensory-driven accumulation of activity encoding each visual feature in memory, and a slower accumulation of internal error that causes memorized features to randomly drift over time. Instead of facilitating read-out from an independent sensory store, an early cue benefits recall by lifting the effective limit on VWM signal strength imposed when multiple items compete for representation, allowing memory for the cued item to be supplemented with information from the decaying sensory trace. Empirical measurements of human recall dynamics validate these predictions while excluding alternative model architectures. A key conclusion is that differences in capacity classically thought to distinguish IM and VWM are in fact contingent upon a single resource-limited WM store.
Collapse
Affiliation(s)
- Ivan Tomić
- Department of Psychology, University of CambridgeCambridgeUnited Kingdom
- Department of Psychology, Faculty of Humanities and Social Sciences, University of ZagrebZagrebCroatia
| | - Paul M Bays
- Department of Psychology, University of CambridgeCambridgeUnited Kingdom
| |
Collapse
|
3
|
Kuntzelman KM, Williams JM, Lim PC, Samal A, Rao PK, Johnson MR. Deep-Learning-Based Multivariate Pattern Analysis (dMVPA): A Tutorial and a Toolbox. Front Hum Neurosci 2021; 15:638052. [PMID: 33737872 PMCID: PMC7960649 DOI: 10.3389/fnhum.2021.638052] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2020] [Accepted: 02/08/2021] [Indexed: 11/17/2022] Open
Abstract
In recent years, multivariate pattern analysis (MVPA) has been hugely beneficial for cognitive neuroscience by making new experiment designs possible and by increasing the inferential power of functional magnetic resonance imaging (fMRI), electroencephalography (EEG), and other neuroimaging methodologies. In a similar time frame, “deep learning” (a term for the use of artificial neural networks with convolutional, recurrent, or similarly sophisticated architectures) has produced a parallel revolution in the field of machine learning and has been employed across a wide variety of applications. Traditional MVPA also uses a form of machine learning, but most commonly with much simpler techniques based on linear calculations; a number of studies have applied deep learning techniques to neuroimaging data, but we believe that those have barely scratched the surface of the potential deep learning holds for the field. In this paper, we provide a brief introduction to deep learning for those new to the technique, explore the logistical pros and cons of using deep learning to analyze neuroimaging data – which we term “deep MVPA,” or dMVPA – and introduce a new software toolbox (the “Deep Learning In Neuroimaging: Exploration, Analysis, Tools, and Education” package, DeLINEATE for short) intended to facilitate dMVPA for neuroscientists (and indeed, scientists more broadly) everywhere.
Collapse
Affiliation(s)
- Karl M Kuntzelman
- Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, United States.,Office of Technology Development and Coordination, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, United States
| | - Jacob M Williams
- Department of Computer Science and Engineering, University of Nebraska-Lincoln, Lincoln, NE, United States
| | - Phui Cheng Lim
- Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, United States.,Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, United States
| | - Ashok Samal
- Department of Computer Science and Engineering, University of Nebraska-Lincoln, Lincoln, NE, United States
| | - Prahalada K Rao
- Department of Mechanical and Materials Engineering, University of Nebraska-Lincoln, Lincoln, NE, United States
| | - Matthew R Johnson
- Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, United States.,Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, United States
| |
Collapse
|
4
|
Wolff MJ, Jochim J, Akyürek EG, Buschman TJ, Stokes MG. Drifting codes within a stable coding scheme for working memory. PLoS Biol 2020; 18:e3000625. [PMID: 32119658 PMCID: PMC7067474 DOI: 10.1371/journal.pbio.3000625] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2019] [Revised: 03/12/2020] [Accepted: 02/12/2020] [Indexed: 11/18/2022] Open
Abstract
Working memory (WM) is important to maintain information over short time periods to provide some stability in a constantly changing environment. However, brain activity is inherently dynamic, raising a challenge for maintaining stable mental states. To investigate the relationship between WM stability and neural dynamics, we used electroencephalography to measure the neural response to impulse stimuli during a WM delay. Multivariate pattern analysis revealed representations were both stable and dynamic: there was a clear difference in neural states between time-specific impulse responses, reflecting dynamic changes, yet the coding scheme for memorised orientations was stable. This suggests that a stable subcomponent in WM enables stable maintenance within a dynamic system. A stable coding scheme simplifies readout for WM-guided behaviour, whereas the low-dimensional dynamic component could provide additional temporal information. Despite having a stable subspace, WM is clearly not perfect-memory performance still degrades over time. Indeed, we find that even within the stable coding scheme, memories drift during maintenance. When averaged across trials, such drift contributes to the width of the error distribution.
Collapse
Affiliation(s)
- Michael J. Wolff
- Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
- Department of Experimental Psychology, University of Groningen, Groningen, the Netherlands
| | - Janina Jochim
- Oxford Centre for Human Brain Activity, University of Oxford, Oxford, United Kingdom
| | - Elkan G. Akyürek
- Department of Experimental Psychology, University of Groningen, Groningen, the Netherlands
| | - Timothy J. Buschman
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, New Jersey, United States of America
| | - Mark G. Stokes
- Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
- Oxford Centre for Human Brain Activity, University of Oxford, Oxford, United Kingdom
| |
Collapse
|