1
|
Ahsan Awais M, Ward T, Redmond P, Healy G. From lab to life: assessing the impact of real-world interactions on the operation of rapid serial visual presentation-based brain-computer interfaces. J Neural Eng 2024; 21:046011. [PMID: 38941986 DOI: 10.1088/1741-2552/ad5d17] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2024] [Accepted: 06/28/2024] [Indexed: 06/30/2024]
Abstract
Objective.Brain-computer interfaces (BCI) have been extensively researched in controlled lab settings where the P300 event-related potential (ERP), elicited in the rapid serial visual presentation (RSVP) paradigm, has shown promising potential. However, deploying BCIs outside of laboratory settings is challenging due to the presence of contaminating artifacts that often occur as a result of activities such as talking, head movements, and body movements. These artifacts can severely contaminate the measured EEG signals and consequently impede detection of the P300 ERP. Our goal is to assess the impact of these real-world noise factors on the performance of a RSVP-BCI, specifically focusing on single-trial P300 detection.Approach.In this study, we examine the impact of movement activity on the performance of a P300-based RSVP-BCI application designed to allow users to search images at high speed. Using machine learning, we assessed P300 detection performance using both EEG data captured in optimal recording conditions (e.g. where participants were instructed to refrain from moving) and a variety of conditions where the participant intentionally produced movements to contaminate the EEG recording.Main results.The results, presented as area under the receiver operating characteristic curve (ROC-AUC) scores, provide insight into the significant impact of noise on single-trial P300 detection. Notably, there is a reduction in classifier detection accuracy when intentionally contaminated RSVP trials are used for training and testing, when compared to using non-intentionally contaminated RSVP trials.Significance.Our findings underscore the necessity of addressing and mitigating noise in EEG recordings to facilitate the use of BCIs in real-world settings, thus extending the reach of EEG technology beyond the confines of the laboratory.
Collapse
Affiliation(s)
- Muhammad Ahsan Awais
- Insight SFI Research Centre for Data Analytics, School of Computing, Dublin City University, Dublin, Ireland
| | - Tomas Ward
- Insight SFI Research Centre for Data Analytics, School of Computing, Dublin City University, Dublin, Ireland
| | - Peter Redmond
- Insight SFI Research Centre for Data Analytics, School of Computing, Dublin City University, Dublin, Ireland
| | - Graham Healy
- Insight SFI Research Centre for Data Analytics, School of Computing, Dublin City University, Dublin, Ireland
| |
Collapse
|
2
|
Lin Y, Hsu YY, Cheng T, Hsiung PC, Wu CW, Hsieh PJ. Neural representations of perspectival shapes and attentional effects: Evidence from fMRI and MEG. Cortex 2024; 176:129-143. [PMID: 38781910 DOI: 10.1016/j.cortex.2024.04.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 02/14/2024] [Accepted: 04/05/2024] [Indexed: 05/25/2024]
Abstract
Does the human brain represent perspectival shapes, i.e., viewpoint-dependent object shapes, especially in relatively higher-level visual areas such as the lateral occipital cortex? What is the temporal profile of the appearance and disappearance of neural representations of perspectival shapes? And how does attention influence these neural representations? To answer these questions, we employed functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG), and multivariate decoding techniques to investigate spatiotemporal neural representations of perspectival shapes. Participants viewed rotated objects along with the corresponding objective shapes and perspectival shapes (i.e., rotated round, round, and oval) while we measured their brain activities. Our results revealed that shape classifiers trained on the basic shapes (i.e., round and oval) consistently identified neural representations in the lateral occipital cortex corresponding to the perspectival shapes of the viewed objects regardless of attentional manipulations. Additionally, this classification tendency toward the perspectival shapes emerged approximately 200 ms after stimulus presentation. Moreover, attention influenced the spatial dimension as the regions showing the perspectival shape classification tendency propagated from the occipital lobe to the temporal lobe. As for the temporal dimension, attention led to a more robust and enduring classification tendency towards perspectival shapes. In summary, our study outlines a spatiotemporal neural profile for perspectival shapes that suggests a greater degree of perspectival representation than is often acknowledged.
Collapse
Affiliation(s)
- Yi Lin
- Taiwan International Graduate Program in Interdisciplinary Neuroscience, National Cheng Kung University and Academia Sinica, Nankan, Taipei, Taiwan; Research Unit Brain and Cognition, KU Leuven, Leuven, Belgium.
| | - Yung-Yi Hsu
- Department of Psychology, National Taiwan University, Da'an, Taipei, Taiwan
| | - Tony Cheng
- Waseda Institute for Advanced Study, Waseda University, Tokyo, Japan
| | - Pin-Cheng Hsiung
- Department of Psychology, National Taiwan University, Da'an, Taipei, Taiwan
| | - Chen-Wei Wu
- Department of Philosophy, Georgia State University, Atlanta, GA, USA
| | - Po-Jang Hsieh
- Department of Psychology, National Taiwan University, Da'an, Taipei, Taiwan.
| |
Collapse
|
3
|
Karapetian A, Boyanova A, Pandaram M, Obermayer K, Kietzmann TC, Cichy RM. Empirically Identifying and Computationally Modeling the Brain-Behavior Relationship for Human Scene Categorization. J Cogn Neurosci 2023; 35:1879-1897. [PMID: 37590093 PMCID: PMC10586810 DOI: 10.1162/jocn_a_02043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/19/2023]
Abstract
Humans effortlessly make quick and accurate perceptual decisions about the nature of their immediate visual environment, such as the category of the scene they face. Previous research has revealed a rich set of cortical representations potentially underlying this feat. However, it remains unknown which of these representations are suitably formatted for decision-making. Here, we approached this question empirically and computationally, using neuroimaging and computational modeling. For the empirical part, we collected EEG data and RTs from human participants during a scene categorization task (natural vs. man-made). We then related EEG data to behavior to behavior using a multivariate extension of signal detection theory. We observed a correlation between neural data and behavior specifically between ∼100 msec and ∼200 msec after stimulus onset, suggesting that the neural scene representations in this time period are suitably formatted for decision-making. For the computational part, we evaluated a recurrent convolutional neural network (RCNN) as a model of brain and behavior. Unifying our previous observations in an image-computable model, the RCNN predicted well the neural representations, the behavioral scene categorization data, as well as the relationship between them. Our results identify and computationally characterize the neural and behavioral correlates of scene categorization in humans.
Collapse
Affiliation(s)
- Agnessa Karapetian
- Freie Universität Berlin, Germany
- Charité - Universitätsmedizin Berlin, Einstein Center for Neurosciences Berlin, Germany
- Bernstein Center for Computational Neuroscience Berlin, Germany
| | | | | | - Klaus Obermayer
- Charité - Universitätsmedizin Berlin, Einstein Center for Neurosciences Berlin, Germany
- Bernstein Center for Computational Neuroscience Berlin, Germany
- Technische Universität Berlin, Germany
- Humboldt-Universität zu Berlin, Germany
| | | | - Radoslaw M Cichy
- Freie Universität Berlin, Germany
- Charité - Universitätsmedizin Berlin, Einstein Center for Neurosciences Berlin, Germany
- Bernstein Center for Computational Neuroscience Berlin, Germany
- Humboldt-Universität zu Berlin, Germany
| |
Collapse
|
4
|
Dalski A, Kovács G, Ambrus GG. No semantic information is necessary to evoke general neural signatures of face familiarity: evidence from cross-experiment classification. Brain Struct Funct 2023; 228:449-462. [PMID: 36244002 PMCID: PMC9944719 DOI: 10.1007/s00429-022-02583-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2022] [Accepted: 10/09/2022] [Indexed: 11/28/2022]
Abstract
Recent theories on the neural correlates of face identification stressed the importance of the available identity-specific semantic and affective information. However, whether such information is essential for the emergence of neural signal of familiarity has not yet been studied in detail. Here, we explored the shared representation of face familiarity between perceptually and personally familiarized identities. We applied a cross-experiment multivariate pattern classification analysis (MVPA), to test if EEG patterns for passive viewing of personally familiar and unfamiliar faces are useful in decoding familiarity in a matching task where familiarity was attained thorough a short perceptual task. Importantly, no additional semantic, contextual, or affective information was provided for the familiarized identities during perceptual familiarization. Although the two datasets originate from different sets of participants who were engaged in two different tasks, familiarity was still decodable in the sorted, same-identity matching trials. This finding indicates that the visual processing of the faces of personally familiar and purely perceptually familiarized identities involve similar mechanisms, leading to cross-classifiable neural patterns.
Collapse
Affiliation(s)
- Alexia Dalski
- Department of Psychology, Philipps-Universität Marburg, 35039 Marburg, Germany ,Center for Mind, Brain and Behavior – CMBB, Philipps-Universität Marburg and Justus-Liebig-Universität Giessen, 35039 Marburg, Germany
| | - Gyula Kovács
- Institute of Psychology, Friedrich Schiller University Jena, 07743 Jena, Germany
| | - Géza Gergely Ambrus
- Institute of Psychology, Friedrich Schiller University Jena, 07743, Jena, Germany. .,Department of Psychology, Bournemouth University, Poole House, Talbot Campus, Fern Barrow, Poole, BH12 5BB, Dorset, UK.
| |
Collapse
|
5
|
Characterizing the shared signals of face familiarity: Long-term acquaintance, voluntary control, and concealed knowledge. Brain Res 2022; 1796:148094. [PMID: 36116487 DOI: 10.1016/j.brainres.2022.148094] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2022] [Revised: 09/10/2022] [Accepted: 09/12/2022] [Indexed: 11/20/2022]
Abstract
In a recent study using cross-experiment multivariate classification of EEG patterns, we found evidence for a shared familiarity signal for faces, patterns of neural activity that successfully separate trials for familiar and unfamiliar faces across participants and modes of familiarization. Here, our aim was to expand upon this research to further characterize the spatio-temporal properties of this signal. By utilizing the information content present for incidental exposure to personally familiar and unfamiliar faces, we tested how the information content in the neural signal unfolds over time under different task demands - giving truthful or deceptive responses to photographs of genuinely familiar and unfamiliar individuals. For this goal, we re-analyzed data from two previously published experiments using within-experiment leave-one-subject-out and cross-experiment classification of face familiarity. We observed that the general face familiarity signal, consistent with its previously described spatio-temporal properties, is present for long-term personally familiar faces under passive viewing, as well as for acknowledged and concealed familiarity responses. Also, central-posterior regions contain information related to deception. We propose that signals in the 200-400 ms window are modulated by top-down task-related anticipation, while the patterns in the 400-600 ms window are influenced by conscious effort to deceive. To our knowledge, this is the first report describing the representational dynamics of concealed knowledge for faces, using time-resolved multivariate classification.
Collapse
|