1
|
Canny E, Vansteensel MJ, van der Salm SMA, Müller-Putz GR, Berezutskaya J. Boosting brain-computer interfaces with functional electrical stimulation: potential applications in people with locked-in syndrome. J Neuroeng Rehabil 2023; 20:157. [PMID: 37980536 PMCID: PMC10656959 DOI: 10.1186/s12984-023-01272-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Accepted: 10/23/2023] [Indexed: 11/20/2023] Open
Abstract
Individuals with a locked-in state live with severe whole-body paralysis that limits their ability to communicate with family and loved ones. Recent advances in brain-computer interface (BCI) technology have presented a potential alternative for these people to communicate by detecting neural activity associated with attempted hand or speech movements and translating the decoded intended movements to a control signal for a computer. A technique that could potentially enrich the communication capacity of BCIs is functional electrical stimulation (FES) of paralyzed limbs and face to restore body and facial movements of paralyzed individuals, allowing to add body language and facial expression to communication BCI utterances. Here, we review the current state of the art of existing BCI and FES work in people with paralysis of body and face and propose that a combined BCI-FES approach, which has already proved successful in several applications in stroke and spinal cord injury, can provide a novel promising mode of communication for locked-in individuals.
Collapse
Affiliation(s)
- Evan Canny
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Mariska J Vansteensel
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Sandra M A van der Salm
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Laboratory of Brain-Computer Interfaces, Graz University of Technology, Graz, Austria
| | - Julia Berezutskaya
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands.
| |
Collapse
|
2
|
Rainey S. A gap between reasons for skilled use of BCI speech devices and reasons for utterances, with implications for speech ownership. Front Hum Neurosci 2023; 17:1248806. [PMID: 37915755 PMCID: PMC10616452 DOI: 10.3389/fnhum.2023.1248806] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2023] [Accepted: 09/11/2023] [Indexed: 11/03/2023] Open
Abstract
The skilled use of a speech BCI device will draw upon practical experience gained through the use of that very device. The reasons a user may have for using a device in a particular way, reflecting that skill gained via familiarity with the device, may differ significantly from the reasons that a speaker might have for their utterances. The potential divergence between reasons constituting skilled use and BCI-mediated speech output may serve to make clear an instrumental relationship between speaker and BCI speech device. This will affect the way in which the device and the speech it produces for the user can be thought of as being "reasons responsive", hence the way in which the user can be said to be in control of their device. Ultimately, this divergence will come down to how ownership of produced speech can be considered. The upshot will be that skillful use of a synthetic speech device might include practices that diverge from standard speech in significant ways. This might further indicate that synthetic speech devices ought to be considered as different from, not continuous with, standard speech.
Collapse
Affiliation(s)
- Stephen Rainey
- Ethics and Philosophy of Technology, Delft University of Technology, Delft, Netherlands
| |
Collapse
|
3
|
Berezutskaya J, Freudenburg ZV, Vansteensel MJ, Aarnoutse EJ, Ramsey NF, van Gerven MAJ. Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models. J Neural Eng 2023; 20:056010. [PMID: 37467739 PMCID: PMC10510111 DOI: 10.1088/1741-2552/ace8be] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2022] [Revised: 07/12/2023] [Accepted: 07/19/2023] [Indexed: 07/21/2023]
Abstract
Objective.Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field.Approach.In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task.Main results.We show that (1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; (2) individual word decoding in reconstructed speech achieves 92%-100% accuracy (chance level is 8%); (3) direct reconstruction from sensorimotor brain activity produces intelligible speech.Significance.These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.
Collapse
Affiliation(s)
- Julia Berezutskaya
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
- Donders Center for Brain, Cognition and Behaviour, Nijmegen 6525 GD, The Netherlands
| | - Zachary V Freudenburg
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
| | - Mariska J Vansteensel
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
| | - Erik J Aarnoutse
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
| | - Nick F Ramsey
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
| | - Marcel A J van Gerven
- Donders Center for Brain, Cognition and Behaviour, Nijmegen 6525 GD, The Netherlands
| |
Collapse
|
4
|
Meng K, Goodarzy F, Kim E, Park YJ, Kim JS, Cook MJ, Chung CK, Grayden DB. Continuous synthesis of artificial speech sounds from human cortical surface recordings during silent speech production. J Neural Eng 2023; 20:046019. [PMID: 37459853 DOI: 10.1088/1741-2552/ace7f6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2023] [Accepted: 07/17/2023] [Indexed: 07/28/2023]
Abstract
Objective. Brain-computer interfaces can restore various forms of communication in paralyzed patients who have lost their ability to articulate intelligible speech. This study aimed to demonstrate the feasibility of closed-loop synthesis of artificial speech sounds from human cortical surface recordings during silent speech production.Approach. Ten participants with intractable epilepsy were temporarily implanted with intracranial electrode arrays over cortical surfaces. A decoding model that predicted audible outputs directly from patient-specific neural feature inputs was trained during overt word reading and immediately tested with overt, mimed and imagined word reading. Predicted outputs were later assessed objectively against corresponding voice recordings and subjectively through human perceptual judgments.Main results. Artificial speech sounds were successfully synthesized during overt and mimed utterances by two participants with some coverage of the precentral gyrus. About a third of these sounds were correctly identified by naïve listeners in two-alternative forced-choice tasks. A similar outcome could not be achieved during imagined utterances by any of the participants. However, neural feature contribution analyses suggested the presence of exploitable activation patterns during imagined speech in the postcentral gyrus and the superior temporal gyrus. In future work, a more comprehensive coverage of cortical surfaces, including posterior parts of the middle frontal gyrus and the inferior frontal gyrus, could improve synthesis performance during imagined speech.Significance.As the field of speech neuroprostheses is rapidly moving toward clinical trials, this study addressed important considerations about task instructions and brain coverage when conducting research on silent speech with non-target participants.
Collapse
Affiliation(s)
- Kevin Meng
- Department of Biomedical Engineering, The University of Melbourne, Melbourne, Australia
- Graeme Clark Institute for Biomedical Engineering, The University of Melbourne, Melbourne, Australia
| | - Farhad Goodarzy
- Department of Medicine, St Vincent's Hospital, The University of Melbourne, Melbourne, Australia
| | - EuiYoung Kim
- Interdisciplinary Program in Neuroscience, Seoul National University, Seoul, Republic of Korea
| | - Ye Jin Park
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, Republic of Korea
| | - June Sic Kim
- Research Institute of Basic Sciences, Seoul National University, Seoul, Republic of Korea
| | - Mark J Cook
- Department of Biomedical Engineering, The University of Melbourne, Melbourne, Australia
- Graeme Clark Institute for Biomedical Engineering, The University of Melbourne, Melbourne, Australia
- Department of Medicine, St Vincent's Hospital, The University of Melbourne, Melbourne, Australia
| | - Chun Kee Chung
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, Republic of Korea
- Department of Neurosurgery, Seoul National University Hospital, Seoul, Republic of Korea
| | - David B Grayden
- Department of Biomedical Engineering, The University of Melbourne, Melbourne, Australia
- Graeme Clark Institute for Biomedical Engineering, The University of Melbourne, Melbourne, Australia
- Department of Medicine, St Vincent's Hospital, The University of Melbourne, Melbourne, Australia
| |
Collapse
|
5
|
Lima C, Lopes JA, Souza V, Barros S, Winkler I, Senna V. Analysis of brain activation and wave frequencies during a sentence completion task: a paradigm used with EEG in aphasic participants. PeerJ 2023; 11:e15518. [PMID: 37334126 PMCID: PMC10269574 DOI: 10.7717/peerj.15518] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2022] [Accepted: 05/15/2023] [Indexed: 06/20/2023] Open
Abstract
Aphasia is a language disorder that occurs after brain injury and directly affects an individual's communication. The incidence of stroke increases with age, and one-third of people who have had a stroke develop aphasia. The severity of aphasia changes over time and some aspects of language may improve, while others remain compromised. Battery task training strategies are used in the rehabilitation of aphasics. The idea of this research is to use electroencephalography (EEG) as a non-invasive method, of electrophysiological monitoring, with a group of aphasic patients in rehabilitation process in a prevention and rehabilitation unit of the person with disabilities of the Unified Health System (SUS), of reference in the state of Bahia-Brazil. In this study, the goal is to analyze brain activation and wave frequencies of aphasic individuals during a sentence completion task, to possibly assist health professionals with the analysis of the aphasic subject's rehabilitation and task redefinition. We adopted the functional magnetic resonance imaging (fMRI) paradigm, proposed by the American Society for Functional Neuroradiology as a reference paradigm. We applied the paradigm in the group of aphasics with preserved comprehension, right hemiparesis, and left hemisphere injured or affected by stroke. We analyzed four electrodes (F3/F4 and F7/F8) corresponding to the left/right frontal cortex. Preliminary results of this study indicate a more robust activation in the right hemisphere (average of aphasics), with a difference of approximately 14% higher in Theta and Alpha frequencies, with 8% higher in low Beta (BetaL) and with approximately 1% higher in high Beta frequency (BetaH), Gamma frequency was higher by approximately 3% in the left hemisphere of the brain. The difference in electrical activation may be revealing to us a migration of language to the non-language dominant hemisphere. We point to possible evidence suggesting that EEG may be a promising tool for monitoring the rehabilitation of the aphasic subject.
Collapse
Affiliation(s)
- Claudia Lima
- MCTI, Senai Cimatec University Center, Salvador, Bahia, Brazil
| | | | | | - Sarah Barros
- Neurological Rehabilitation Sector, CEPRED, Salvador, Bahia, Brazil
| | - Ingrid Winkler
- GETEC, Senai Cimatec University Center, Salvador, Bahia, Brazil
| | - Valter Senna
- MCTI, Senai Cimatec University Center, Salvador, Bahia, Brazil
| |
Collapse
|
6
|
Pan H, Li Z, Tian C, Wang L, Fu Y, Qin X, Liu F. The LightGBM-based classification algorithm for Chinese characters speech imagery BCI system. Cogn Neurodyn 2023; 17:373-384. [PMID: 37007202 PMCID: PMC10050290 DOI: 10.1007/s11571-022-09819-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2021] [Revised: 03/26/2022] [Accepted: 04/09/2022] [Indexed: 11/03/2022] Open
Abstract
Brain-computer interface (BCI) can obtain text information by decoding language induced electroencephalogram (EEG) signals, so as to restore communication ability for patients with language impairment. At present, the BCI system based on speech imagery of Chinese characters has the problem of low accuracy of features classification. In this paper, the light gradient boosting machine (LightGBM) is adopted to recognize Chinese characters and solve the above problems. Firstly, the Db4 wavelet basis function is selected to decompose the EEG signals in six-layer of full frequency band, and the correlation features of Chinese characters speech imagery with high time resolution and high frequency resolution are extracted. Secondly, the two core algorithms of LightGBM, gradient-based one-side sampling and exclusive feature bundling, are used to classify the extracted features. Finally, we verify that classification performance of LightGBM is more accurate and applicable than the traditional classifiers according to the statistical analysis methods. We evaluate the proposed method through contrast experiment. The experimental results show that the average classification accuracy of the subjects' silent reading of Chinese characters "(left)", "(one)" and simultaneous silent reading is improved by 5.24%, 4.90% and 12.44% respectively.
Collapse
Affiliation(s)
- Hongguang Pan
- College of Electrical and Control Engineering, Xi’an University of Science and Technology, Xi’an, 710054 Shaanxi China
| | - Zhuoyi Li
- College of Electrical and Control Engineering, Xi’an University of Science and Technology, Xi’an, 710054 Shaanxi China
| | - Chen Tian
- Shaanxi Broadcasting Corporation, Xi’an, 710061 Shaanxi China
| | - Li Wang
- School of Electronics and Communication Engineering, Guangzhou University, Guangzhou, 510006 Guangdong China
| | - Yunpeng Fu
- College of Electrical and Control Engineering, Xi’an University of Science and Technology, Xi’an, 710054 Shaanxi China
| | - Xuebin Qin
- College of Electrical and Control Engineering, Xi’an University of Science and Technology, Xi’an, 710054 Shaanxi China
| | - Fei Liu
- College of Electrical and Control Engineering, Xi’an University of Science and Technology, Xi’an, 710054 Shaanxi China
| |
Collapse
|
7
|
Verwoert M, Ottenhoff MC, Goulis S, Colon AJ, Wagner L, Tousseyn S, van Dijk JP, Kubben PL, Herff C. Dataset of Speech Production in intracranial.Electroencephalography. Sci Data 2022; 9:434. [PMID: 35869138 PMCID: PMC9307753 DOI: 10.1038/s41597-022-01542-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2022] [Accepted: 07/08/2022] [Indexed: 11/28/2022] Open
Abstract
Speech production is an intricate process involving a large number of muscles and cognitive processes. The neural processes underlying speech production are not completely understood. As speech is a uniquely human ability, it can not be investigated in animal models. High-fidelity human data can only be obtained in clinical settings and is therefore not easily available to all researchers. Here, we provide a dataset of 10 participants reading out individual words while we measured intracranial EEG from a total of 1103 electrodes. The data, with its high temporal resolution and coverage of a large variety of cortical and sub-cortical brain regions, can help in understanding the speech production process better. Simultaneously, the data can be used to test speech decoding and synthesis approaches from neural data to develop speech Brain-Computer Interfaces and speech neuroprostheses. Measurement(s) | Brain activity | Technology Type(s) | Stereotactic electroencephalography | Sample Characteristic - Organism | Homo sapiens | Sample Characteristic - Environment | Epilepsy monitoring center | Sample Characteristic - Location | The Netherlands |
Collapse
|
8
|
Rainey S. Speaker Responsibility for Synthetic Speech Derived from Neural Activity. THE JOURNAL OF MEDICINE AND PHILOSOPHY 2022; 47:503-515. [PMID: 36333930 DOI: 10.1093/jmp/jhac011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022] Open
Abstract
This article provides analysis of the mechanisms and outputs involved in language-use mediated by a neuroprosthetic device. It is motivated by the thought that users of speech neuroprostheses require sufficient control over what their devices externalize as synthetic speech if they are to be thought of as responsible for it, but that the nature of this control, and so the status of their responsibility, is not clear.
Collapse
|
9
|
Tong J, Wei X, Dong E, Sun Z, Du S, Duan F. Hybrid mental tasks based human computer interface via integration of pronunciation and motor imagery. J Neural Eng 2022; 19. [PMID: 36228578 DOI: 10.1088/1741-2552/ac9a01] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2022] [Accepted: 10/13/2022] [Indexed: 12/24/2022]
Abstract
Objective.Among the existing active brain-computer interfaces (BCI), the motor imagination (MI) is widely used. To operate the MI BCI effectively, subjects need to carry out trainings on corresponding imagining tasks. Here, we studied how to reduce the discomfort and fatigue of active BCI imaginary tasks and the inability to concentrate on them while improving the accuracy.Approach.This paper proposes a hybrid BCI composed of MI and pronunciation imagination (PI). The electroencephalogram signals of ten subjects are recognized by the adaptive Riemannian distance classification and the improved frequency selective filter-bank Common Spatial Pattern recognition.Main results.The results show that under the new paradigm with the combination of MI and PI, the recognition accuracy is higher than the MI alone. The highest recognition rate of the proposed hybrid system can reach more than 90%. Furthermore, through the subjects' scoring results of the operation difficulty, it is concluded that the designed hybrid paradigm is more operable than the traditional BCI paradigm.Significance.The separable tasks in the active BCI are limited and the accuracy needs to be improved. The new hybrid paradigm proposed by us improves the accuracy and operability of the active BCI system, providing a new possibility for the research direction of the active BCI.
Collapse
Affiliation(s)
- Jigang Tong
- Tianjin Key Laboratory of Control Theory and Applications in Complicated Systems, TianjinUniversity of Technology, Tianjin 300384, People's Republic of China
| | - Xiaoying Wei
- Tianjin Key Laboratory of Control Theory and Applications in Complicated Systems, TianjinUniversity of Technology, Tianjin 300384, People's Republic of China
| | - Enzeng Dong
- Tianjin Key Laboratory of Control Theory and Applications in Complicated Systems, TianjinUniversity of Technology, Tianjin 300384, People's Republic of China
| | - Zhe Sun
- Computational Engineering Applications Unit, Head Office for Information Systems and Cybersecurity, RIKEN, Saitama, Japan
| | - Shengzhi Du
- Department of Electrical Engineering, Tshwane University of Technology, Pretoria 0001, South Africa
| | - Feng Duan
- College of Artificial Intelligence, Nankai University, Tianjin, People's Republic of China
| |
Collapse
|
10
|
Shimizu H, Srinivasan R. Improving classification and reconstruction of imagined images from EEG signals. PLoS One 2022; 17:e0274847. [PMID: 36129927 PMCID: PMC9491577 DOI: 10.1371/journal.pone.0274847] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Accepted: 09/05/2022] [Indexed: 11/19/2022] Open
Abstract
Decoding brain activity related to specific tasks, such as imagining something, is important for brain computer interface (BCI) control. While decoding of brain signals, such as functional magnetic resonance imaging (fMRI) signals and electroencephalography (EEG) signals, during observing visual images and while imagining images has been previously reported, further development of methods for improving training, performance, and interpretation of brain data was the goal of this study. We applied a Sinc-EEGNet to decode brain activity during perception and imagination of visual stimuli, and added an attention module to extract the importance of each electrode or frequency band. We also reconstructed images from brain activity by using a generative adversarial network (GAN). By combining the EEG recorded during a visual task (perception) and an imagination task, we have successfully boosted the accuracy of classifying EEG data in the imagination task and improved the quality of reconstruction by GAN. Our result indicates that the brain activity evoked during the visual task is present in the imagination task and can be used for better classification of the imagined image. By using the attention module, we can derive the spatial weights in each frequency band and contrast spatial or frequency importance between tasks from our model. Imagination tasks are classified by low frequency EEG signals over temporal cortex, while perception tasks are classified by high frequency EEG signals over occipital and frontal cortex. Combining data sets in training results in a balanced model improving classification of the imagination task without significantly changing performance in the visual task. Our approach not only improves performance and interpretability but also potentially reduces the burden on training since we can improve the accuracy of classifying a relatively hard task with high variability (imagination) by combining with the data of the relatively easy task, observing visual images.
Collapse
Affiliation(s)
- Hirokatsu Shimizu
- Department of Cognitive Sciences, University of California, Irvine, CA, United States of America
- * E-mail:
| | - Ramesh Srinivasan
- Department of Cognitive Sciences, University of California, Irvine, CA, United States of America
- Department of Biomedical Engineering, University of California, Irvine, CA, United States of America
| |
Collapse
|
11
|
A hybrid autoencoder framework of dimensionality reduction for brain-computer interface decoding. Comput Biol Med 2022; 148:105871. [DOI: 10.1016/j.compbiomed.2022.105871] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Revised: 06/20/2022] [Accepted: 07/09/2022] [Indexed: 11/19/2022]
|
12
|
Chandler JA, Van der Loos KI, Boehnke S, Beaudry JS, Buchman DZ, Illes J. Brain Computer Interfaces and Communication Disabilities: Ethical, Legal, and Social Aspects of Decoding Speech From the Brain. Front Hum Neurosci 2022; 16:841035. [PMID: 35529778 PMCID: PMC9069963 DOI: 10.3389/fnhum.2022.841035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2021] [Accepted: 03/03/2022] [Indexed: 11/28/2022] Open
Abstract
A brain-computer interface technology that can decode the neural signals associated with attempted but unarticulated speech could offer a future efficient means of communication for people with severe motor impairments. Recent demonstrations have validated this approach. Here we assume that it will be possible in future to decode imagined (i.e., attempted but unarticulated) speech in people with severe motor impairments, and we consider the characteristics that could maximize the social utility of a BCI for communication. As a social interaction, communication involves the needs and goals of both speaker and listener, particularly in contexts that have significant potential consequences. We explore three high-consequence legal situations in which neurally-decoded speech could have implications: Testimony, where decoded speech is used as evidence; Consent and Capacity, where it may be used as a means of agency and participation such as consent to medical treatment; and Harm, where such communications may be networked or may cause harm to others. We then illustrate how design choices might impact the social and legal acceptability of these technologies.
Collapse
Affiliation(s)
- Jennifer A. Chandler
- Bertram Loeb Research Chair, Faculty of Law, University of Ottawa, Ottawa, ON, Canada
- *Correspondence: Jennifer A. Chandler,
| | | | - Susan Boehnke
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON, Canada
| | - Jonas S. Beaudry
- Institute for Health and Social Policy (IHSP) and Faculty of Law, McGill University, Montreal, QC, Canada
| | - Daniel Z. Buchman
- Centre for Addiction and Mental Health, Dalla Lana School of Public Health, Krembil Research Institute, University of Toronto Joint Centre for Bioethics, Toronto, ON, Canada
| | - Judy Illes
- Division of Neurology, Department of Medicine, University of British Columbia, Vancouver, BC, Canada
| |
Collapse
|
13
|
Nagata K, Kunii N, Shimada S, Fujitani S, Takasago M, Saito N. Spatiotemporal target selection for intracranial neural decoding of abstract and concrete semantics. Cereb Cortex 2022; 32:5544-5554. [PMID: 35169837 PMCID: PMC9753048 DOI: 10.1093/cercor/bhac034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Revised: 01/18/2022] [Accepted: 01/19/2021] [Indexed: 01/25/2023] Open
Abstract
Decoding the inner representation of a word meaning from human cortical activity is a substantial challenge in the development of speech brain-machine interfaces (BMIs). The semantic aspect of speech is a novel target of speech decoding that may enable versatile communication platforms for individuals with impaired speech ability; however, there is a paucity of electrocorticography studies in this field. We decoded the semantic representation of a word from single-trial cortical activity during an imageability-based property identification task that required participants to discriminate between the abstract and concrete words. Using high gamma activity in the language-dominant hemisphere, a support vector machine classifier could discriminate the 2-word categories with significantly high accuracy (73.1 ± 7.5%). Activities in specific time components from two brain regions were identified as significant predictors of abstract and concrete dichotomy. Classification using these feature components revealed that comparable prediction accuracy could be obtained based on a spatiotemporally targeted decoding approach. Our study demonstrated that mental representations of abstract and concrete word processing could be decoded from cortical high gamma activities, and the coverage of implanted electrodes and time window of analysis could be successfully minimized. Our findings lay the foundation for the future development of semantic-based speech BMIs.
Collapse
Affiliation(s)
- Keisuke Nagata
- Department of Neurosurgery, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan
| | - Naoto Kunii
- Corresponding author: Department of Neurosurgery, The University of Tokyo, 73-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan.
| | - Seijiro Shimada
- Department of Neurosurgery, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan
| | - Shigeta Fujitani
- Department of Neurosurgery, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan
| | - Megumi Takasago
- Department of Neurosurgery, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan
| | - Nobuhito Saito
- Department of Neurosurgery, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan
| |
Collapse
|
14
|
Proix T, Delgado Saa J, Christen A, Martin S, Pasley BN, Knight RT, Tian X, Poeppel D, Doyle WK, Devinsky O, Arnal LH, Mégevand P, Giraud AL. Imagined speech can be decoded from low- and cross-frequency intracranial EEG features. Nat Commun 2022; 13:48. [PMID: 35013268 PMCID: PMC8748882 DOI: 10.1038/s41467-021-27725-3] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2021] [Accepted: 12/03/2021] [Indexed: 01/19/2023] Open
Abstract
Reconstructing intended speech from neural activity using brain-computer interfaces holds great promises for people with severe speech production deficits. While decoding overt speech has progressed, decoding imagined speech has met limited success, mainly because the associated neural signals are weak and variable compared to overt speech, hence difficult to decode by learning algorithms. We obtained three electrocorticography datasets from 13 patients, with electrodes implanted for epilepsy evaluation, who performed overt and imagined speech production tasks. Based on recent theories of speech neural processing, we extracted consistent and specific neural features usable for future brain computer interfaces, and assessed their performance to discriminate speech items in articulatory, phonetic, and vocalic representation spaces. While high-frequency activity provided the best signal for overt speech, both low- and higher-frequency power and local cross-frequency contributed to imagined speech decoding, in particular in phonetic and vocalic, i.e. perceptual, spaces. These findings show that low-frequency power and cross-frequency dynamics contain key information for imagined speech decoding.
Collapse
Affiliation(s)
- Timothée Proix
- Department of Basic Neurosciences, Faculty of Medicine, University of Geneva, Geneva, Switzerland.
| | - Jaime Delgado Saa
- Department of Basic Neurosciences, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| | - Andy Christen
- Department of Basic Neurosciences, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| | - Stephanie Martin
- Department of Basic Neurosciences, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| | - Brian N Pasley
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, USA
| | - Robert T Knight
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, USA
- Department of Psychology, University of California, Berkeley, Berkeley, USA
| | - Xing Tian
- Division of Arts and Sciences, New York University Shanghai, Shanghai, China
- Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), School of Psychology and Cognitive Science, East China Normal University, Shanghai, China
- NYU-ECNU Institute of Brain and Cognitive Science at NYU Shanghai, Shanghai, China
| | - David Poeppel
- Department of Psychology, New York University, New York, NY, USA
- Ernst Strüngmann Institute for Neuroscience, Frankfurt, Germany
| | - Werner K Doyle
- Department of Neurology, New York University Grossman School of Medicine, New York, NY, USA
| | - Orrin Devinsky
- Department of Neurology, New York University Grossman School of Medicine, New York, NY, USA
| | - Luc H Arnal
- Institut de l'Audition, Institut Pasteur, INSERM, F-75012, Paris, France
| | - Pierre Mégevand
- Department of Basic Neurosciences, Faculty of Medicine, University of Geneva, Geneva, Switzerland
- Division of Neurology, Geneva University Hospitals, Geneva, Switzerland
| | - Anne-Lise Giraud
- Department of Basic Neurosciences, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| |
Collapse
|
15
|
Kiroy V, Bakhtin O, Krivko E, Lazurenko D, Aslanyan E, Shaposhnikov D, Shcherban I. Spoken and Inner Speech-related EEG Connectivity in Different Spatial Direction. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103224] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
|
16
|
Abstract
Implantable brain-computer interfaces (BCIs) are being developed to restore speech capacity for those who are unable to speak. Patients with locked-in syndrome or amyotrophic lateral sclerosis could be able to use covert speech – vividly imagining saying something without actual vocalisation – to trigger neural controlled systems capable of synthesising speech. User control has been identified as particularly pressing for this type of BCI. The incorporation of machine learning and statistical language models into the decoding process introduces a contribution to (or ‘shaping of’) the output that is beyond the user’s control. Whilst this type of ‘shared control’ of BCI action is not unique to speech BCIs, the automated shaping of what a user ‘says’ has a particularly acute ethical dimension, which may differ from parallel concerns surrounding automation in movement BCIs. This paper provides an analysis of the control afforded to the user of a speech BCI of the sort under development, as well as the relationships between accuracy, control, and the user’s ownership of the speech produced. Through comparing speech BCIs with BCIs for movement, we argue that, whilst goal selection is the more significant locus of control for the user of a movement BCI, control over process will be more significant for the user of the speech BCI. The design of the speech BCI may therefore have to trade off some possible efficiency gains afforded by automation in order to preserve sufficient guidance control necessary for users to express themselves in ways they prefer. We consider the implications for the speech BCI user’s responsibility for produced outputs and their ownership of token outputs. We argue that these are distinct assessments. Ownership of synthetic speech concerns whether the content of the output sufficiently represents the user, rather than their morally relevant, causal role in producing that output.
Collapse
|
17
|
Sheth J, Tankus A, Tran M, Pouratian N, Fried I, Speier W. Generalizing neural signal-to-text brain-computer interfaces. Biomed Phys Eng Express 2021; 7. [PMID: 33836507 DOI: 10.1088/2057-1976/abf6ab] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2020] [Accepted: 04/09/2021] [Indexed: 11/12/2022]
Abstract
Objective:Brain-Computer Interfaces (BCI) may help patients with faltering communication abilities due to neurodegenerative diseases produce text or speech by direct neural processing. However, their practical realization has proven difficult due to limitations in speed, accuracy, and generalizability of existing interfaces. The goal of this study is to evaluate the BCI performance of a robust speech decoding system that translates neural signals evoked by speech to a textual output. While previous studies have approached this problem by using neural signals to choose from a limited set of possible words, we employ a more general model that can type any word from a large corpus of English text.Approach:In this study, we create an end-to-end BCI that translates neural signals associated with overt speech into text output. Our decoding system first isolates frequency bands in the input depth-electrode signal encapsulating differential information regarding production of various phonemic classes. These bands form a feature set that then feeds into a Long Short-Term Memory (LSTM) model which discerns at each time point probability distributions across all phonemes uttered by a subject. Finally, a particle filtering algorithm temporally smooths these probabilities by incorporating prior knowledge of the English language to output text corresponding to the decoded word. The generalizability of our decoder is driven by the lack of a vocabulary constraint on this output word.Main result:This method was evaluated using a dataset of 6 neurosurgical patients implanted with intra-cranial depth electrodes to identify seizure foci for potential surgical treatment of epilepsy. We averaged 32% word accuracy and on the phoneme-level obtained 46% precision, 51% recall and 73.32% average phoneme error rate while also achieving significant increases in speed when compared to several other BCI approaches.Significance:Our study employs a more general neural signal-to-text model which could facilitate communication by patients in everyday environments.
Collapse
Affiliation(s)
- Janaki Sheth
- Department of Physics and Astronomy, UCLA, Los Angeles, CA, United States of America
| | - Ariel Tankus
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel.,Functional Neurosurgery Unit, Tel Aviv, Sourasky Medical Center, Tel Aviv, Israel.,Department of Neurology and Neurosurgery, Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel
| | - Michelle Tran
- Department of Neurosurgery, UCLA, Los Angeles, CA, United States of America
| | - Nader Pouratian
- Department of Neurosurgery, UCLA, Los Angeles, CA, United States of America
| | - Itzhak Fried
- Department of Neurosurgery, UCLA, Los Angeles, CA, United States of America
| | - William Speier
- Department of Radiology, UCLA, Los Angeles, CA, United States of America
| |
Collapse
|
18
|
Panachakel JT, Ramakrishnan AG. Decoding Covert Speech From EEG-A Comprehensive Review. Front Neurosci 2021; 15:642251. [PMID: 33994922 PMCID: PMC8116487 DOI: 10.3389/fnins.2021.642251] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2020] [Accepted: 03/18/2021] [Indexed: 11/13/2022] Open
Abstract
Over the past decade, many researchers have come up with different implementations of systems for decoding covert or imagined speech from EEG (electroencephalogram). They differ from each other in several aspects, from data acquisition to machine learning algorithms, due to which, a comparison between different implementations is often difficult. This review article puts together all the relevant works published in the last decade on decoding imagined speech from EEG into a single framework. Every important aspect of designing such a system, such as selection of words to be imagined, number of electrodes to be recorded, temporal and spatial filtering, feature extraction and classifier are reviewed. This helps a researcher to compare the relative merits and demerits of the different approaches and choose the one that is most optimal. Speech being the most natural form of communication which human beings acquire even without formal education, imagined speech is an ideal choice of prompt for evoking brain activity patterns for a BCI (brain-computer interface) system, although the research on developing real-time (online) speech imagery based BCI systems is still in its infancy. Covert speech based BCI can help people with disabilities to improve their quality of life. It can also be used for covert communication in environments that do not support vocal communication. This paper also discusses some future directions, which will aid the deployment of speech imagery based BCI for practical applications, rather than only for laboratory experiments.
Collapse
Affiliation(s)
- Jerrin Thomas Panachakel
- Medical Intelligence and Language Engineering Laboratory, Department of Electrical Engineering, Indian Institute of Science, Bangalore, India
| | | |
Collapse
|
19
|
Abstract
Technologies controlled directly by the brain are being developed, evolving based on insights gained from neuroscience, and rehabilitative medicine. Besides neuro-controlled prosthetics aimed at restoring function lost somehow, technologies controlled via brain-computer interfaces (BCIs) may also extend a user’s horizon of action, freed from the need for bodily movement. Whilst BCI-mediated action ought to be, on the whole, treated as conventional action, law and policy ought to be amended to accommodate BCI action by broadening the definition of action as “willed bodily movement”. Moreover, there are some dimensions of BCI mediated action that are significantly different to conventional cases. These relate to control. Specifically, to limits in both controllability of BCIs via neural states, and in foreseeability of outcomes from such actions. In some specific type of case, BCI-mediated action may be due to different ethical evaluation from conventional action.
Collapse
Affiliation(s)
- Stephen Rainey
- The Oxford Uehiro Centre for Practical Ethics, University of Oxford
| | - Hannah Maslen
- The Oxford Uehiro Centre for Practical Ethics, University of Oxford
| | - Julian Savulescu
- The Oxford Uehiro Centre for Practical Ethics, University of Oxford
| |
Collapse
|
20
|
Roussel P, Godais GL, Bocquelet F, Palma M, Hongjie J, Zhang S, Giraud AL, Mégevand P, Miller K, Gehrig J, Kell C, Kahane P, Chabardés S, Yvert B. Observation and assessment of acoustic contamination of electrophysiological brain signals during speech production and sound perception. J Neural Eng 2020; 17:056028. [PMID: 33055383 DOI: 10.1088/1741-2552/abb25e] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/28/2023]
Abstract
OBJECTIVE A current challenge of neurotechnologies is to develop speech brain-computer interfaces aiming at restoring communication in people unable to speak. To achieve a proof of concept of such system, neural activity of patients implanted for clinical reasons can be recorded while they speak. Using such simultaneously recorded audio and neural data, decoders can be built to predict speech features using features extracted from brain signals. A typical neural feature is the spectral power of field potentials in the high-gamma frequency band, which happens to overlap the frequency range of speech acoustic signals, especially the fundamental frequency of the voice. Here, we analyzed human electrocorticographic and intracortical recordings during speech production and perception as well as a rat microelectrocorticographic recording during sound perception. We observed that several datasets, recorded with different recording setups, contained spectrotemporal features highly correlated with those of the sound produced by or delivered to the participants, especially within the high-gamma band and above, strongly suggesting a contamination of electrophysiological recordings by the sound signal. This study investigated the presence of acoustic contamination and its possible source. APPROACH We developed analysis methods and a statistical criterion to objectively assess the presence or absence of contamination-specific correlations, which we used to screen several datasets from five centers worldwide. MAIN RESULTS Not all but several datasets, recorded in a variety of conditions, showed significant evidence of acoustic contamination. Three out of five centers were concerned by the phenomenon. In a recording showing high contamination, the use of high-gamma band features dramatically facilitated the performance of linear decoding of acoustic speech features, while such improvement was very limited for another recording showing no significant contamination. Further analysis and in vitro replication suggest that the contamination is caused by the mechanical action of the sound waves onto the cables and connectors along the recording chain, transforming sound vibrations into an undesired electrical noise affecting the biopotential measurements. SIGNIFICANCE Although this study does not per se question the presence of speech-relevant physiological information in the high-gamma range and above (multiunit activity), it alerts on the fact that acoustic contamination of neural signals should be proofed and eliminated before investigating the cortical dynamics of these processes. To this end, we make available a toolbox implementing the proposed statistical approach to quickly assess the extent of contamination in an electrophysiological recording (https://doi.org/10.5281/zenodo.3929296).
Collapse
Affiliation(s)
- Philémon Roussel
- Inserm, BrainTech Lab, U1205, Grenoble, France. University Grenoble Alpes, BrainTech Lab, U1205, Grenoble, France
| | | | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
21
|
Dash D, Wisler A, Ferrari P, Davenport EM, Maldjian J, Wang J. MEG Sensor Selection for Neural Speech Decoding. IEEE ACCESS : PRACTICAL INNOVATIONS, OPEN SOLUTIONS 2020; 8:182320-182337. [PMID: 33204579 PMCID: PMC7668411 DOI: 10.1109/access.2020.3028831] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/04/2023]
Abstract
Direct decoding of speech from the brain is a faster alternative to current electroencephalography (EEG) speller-based brain-computer interfaces (BCI) in providing communication assistance to locked-in patients. Magnetoencephalography (MEG) has recently shown great potential as a non-invasive neuroimaging modality for neural speech decoding, owing in part to its spatial selectivity over other high-temporal resolution devices. Standard MEG systems have a large number of cryogenically cooled channels/sensors (200 - 300) encapsulated within a fixed liquid helium dewar, precluding their use as wearable BCI devices. Fortunately, recently developed optically pumped magnetometers (OPM) do not require cryogens, and have the potential to be wearable and movable making them more suitable for BCI applications. This design is also modular allowing for customized montages to include only the sensors necessary for a particular task. As the number of sensors bears a heavy influence on the cost, size, and weight of MEG systems, minimizing the number of sensors is critical for designing practical MEG-based BCIs in the future. In this study, we sought to identify an optimal set of MEG channels to decode imagined and spoken phrases from the MEG signals. Using a forward selection algorithm with a support vector machine classifier we found that nine optimally located MEG gradiometers provided higher decoding accuracy compared to using all channels. Additionally, the forward selection algorithm achieved similar performance to dimensionality reduction using a stacked-sparse-autoencoder. Analysis of spatial dynamics of speech decoding suggested that both left and right hemisphere sensors contribute to speech decoding. Sensors approximately located near Broca's area were found to be commonly contributing among the higher-ranked sensors across all subjects.
Collapse
Affiliation(s)
- Debadatta Dash
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
- Department of Neurology, Dell Medical School, The University of Texas at Austin, Austin, TX 78712, USA
| | - Alan Wisler
- Department of Speech, Language, and Hearing Sciences, University of Texas at Austin, Austin, TX 78712, USA
| | - Paul Ferrari
- MEG Laboratory, Dell Children's Medical Center, Austin, TX 78723, USA
- Department of Psychology, The University of Texas at Austin, Austin, TX 78712, USA
| | | | - Joseph Maldjian
- Department of Radiology, University of Texas at Southwestern, Dallas, TX 75390, USA
| | - Jun Wang
- Department of Neurology, Dell Medical School, The University of Texas at Austin, Austin, TX 78712, USA
- Department of Speech, Language, and Hearing Sciences, University of Texas at Austin, Austin, TX 78712, USA
| |
Collapse
|
22
|
Neuroprosthetic Speech: The Ethical Significance of Accuracy, Control and Pragmatics. Camb Q Healthc Ethics 2020; 28:657-670. [PMID: 31475659 DOI: 10.1017/s0963180119000604] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2023]
Abstract
Neuroprosthetic speech devices are an emerging technology that can offer the possibility of communication to those who are unable to speak. Patients with 'locked in syndrome,' aphasia, or other such pathologies can use covert speech-vividly imagining saying something without actual vocalization-to trigger neural controlled systems capable of synthesizing the speech they would have spoken, but for their impairment.We provide an analysis of the mechanisms and outputs involved in speech mediated by neuroprosthetic devices. This analysis provides a framework for accounting for the ethical significance of accuracy, control, and pragmatic dimensions of prosthesis-mediated speech. We first examine what it means for the output of the device to be accurate, drawing a distinction between technical accuracy on the one hand and semantic accuracy on the other. These are conceptual notions of accuracy.Both technical and semantic accuracy of the device will be necessary (but not yet sufficient) for the user to have sufficient control over the device. Sufficient control is an ethical consideration: we place high value on being able to express ourselves when we want and how we want. Sufficient control of a neural speech prosthesis requires that a speaker can reliably use their speech apparatus as they want to, and can expect their speech to authentically represent them. We draw a distinction between two relevant features which bear on the question of whether the user has sufficient control: voluntariness of the speech and the authenticity of the speech. These can come apart: the user might involuntarily produce an authentic output (perhaps revealing private thoughts) or might voluntarily produce an inauthentic output (e.g., when the output is not semantically accurate). Finally, we consider the role of the interlocutor in interpreting the content and purpose of the communication.These three ethical dimensions raise philosophical questions about the nature of speech, the level of control required for communicative accuracy, and the nature of 'accuracy' with respect to both natural and prosthesis-mediated speech.
Collapse
|
23
|
Huggins JE, Guger C, Aarnoutse E, Allison B, Anderson CW, Bedrick S, Besio W, Chavarriaga R, Collinger JL, Do AH, Herff C, Hohmann M, Kinsella M, Lee K, Lotte F, Müller-Putz G, Nijholt A, Pels E, Peters B, Putze F, Rupp R, Schalk G, Scott S, Tangermann M, Tubig P, Zander T. Workshops of the Seventh International Brain-Computer Interface Meeting: Not Getting Lost in Translation. BRAIN-COMPUTER INTERFACES 2019; 6:71-101. [PMID: 33033729 PMCID: PMC7539697 DOI: 10.1080/2326263x.2019.1697163] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2019] [Accepted: 10/30/2019] [Indexed: 12/11/2022]
Abstract
The Seventh International Brain-Computer Interface (BCI) Meeting was held May 21-25th, 2018 at the Asilomar Conference Grounds, Pacific Grove, California, United States. The interactive nature of this conference was embodied by 25 workshops covering topics in BCI (also called brain-machine interface) research. Workshops covered foundational topics such as hardware development and signal analysis algorithms, new and imaginative topics such as BCI for virtual reality and multi-brain BCIs, and translational topics such as clinical applications and ethical assumptions of BCI development. BCI research is expanding in the diversity of applications and populations for whom those applications are being developed. BCI applications are moving toward clinical readiness as researchers struggle with the practical considerations to make sure that BCI translational efforts will be successful. This paper summarizes each workshop, providing an overview of the topic of discussion, references for additional information, and identifying future issues for research and development that resulted from the interactions and discussion at the workshop.
Collapse
Affiliation(s)
- Jane E Huggins
- Department of Physical Medicine and Rehabilitation, Department of Biomedical Engineering, Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan, United States, 325 East Eisenhower, Room 3017; Ann Arbor, Michigan 48108-5744
| | - Christoph Guger
- g.tec medical engineering GmbH/Guger Technologies OG, Austria, Sierningstrasse 14, 4521 Schiedlberg, Austria
| | - Erik Aarnoutse
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| | - Brendan Allison
- Dept. of Cognitive Science, Mail Code 0515, University of California at San Diego, La Jolla, United States
| | - Charles W Anderson
- Department of Computer Science, Molecular, Cellular and Integrative Neurosience Program, Colorado State University, Fort Collins, CO 80523
| | - Steven Bedrick
- Center for Spoken Language Understanding, Oregon Health & Science University, Portland, OR 97239
| | - Walter Besio
- Department of Electrical, Computer, & Biomedical Engineering and Interdisciplinary Neuroscience Program, University of Rhode Island, Kingston, Rhode Island, USA, CREmedical Corp. Kingston, Rhode Island, USA
| | - Ricardo Chavarriaga
- Defitech Chair in Brain-Machine Interface (CNBI), Center for Neuroprosthetics, Ecole Polytechnique Fédérale de Lausanne - EPFL, Switzerland
| | - Jennifer L Collinger
- University of Pittsburgh, Department of Physical Medicine and Rehabilitation, VA Pittsburgh Healthcare System, Department of Veterans Affairs, 3520 5th Ave, Pittsburgh, PA, 15213
| | - An H Do
- UC Irvine Brain Computer Interface Lab, Department of Neurology, University of California, Irvine
| | - Christian Herff
- School of Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | - Matthias Hohmann
- Max Planck Institute for Intelligent Systems, Department for Empirical Inference, Max-Planck-Ring 4, 72074 Tübingen, Germany
| | - Michelle Kinsella
- Oregon Health & Science University, Institute on Development & Disability, 707 SW Gaines St, #1290, Portland, OR 97239
| | - Kyuhwa Lee
- Swiss Federal Institute of Technology in Lausanne-EPFL
| | - Fabien Lotte
- Inria Bordeaux Sud-Ouest, LaBRI (Univ. Bordeaux/CNRS/Bordeaux INP), 200 avenue de la vieille tour, 33405, Talence Cedex, France
| | | | - Anton Nijholt
- Faculty EEMCS, University of Twente, Enschede, The Netherlands
| | - Elmar Pels
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| | - Betts Peters
- Oregon Health & Science University, Institute on Development & Disability, 707 SW Gaines St, #1290, Portland, OR 97239
| | - Felix Putze
- University of Bremen, Germany, Cognitive Systems Lab, University of Bremen, Enrique-Schmidt-Straße 5 (Cartesium), 28359 Bremen
| | - Rüdiger Rupp
- Spinal Cord Injury Center, Heidelberg University Hospital
| | - Gerwin Schalk
- National Center for Adaptive Neurotechnologies, Wadsworth Center, NYS Dept. of Health, Dept. of Neurology, Albany Medical College, Dept. of Biomed. Sci., State Univ. of New York at Albany, Center for Medical Sciences 2003, 150 New Scotland Avenue, Albany, New York 12208
| | - Stephanie Scott
- Department of Media Communications, Colorado State University, Fort Collins, CO 80523
| | - Michael Tangermann
- Brain State Decoding Lab, Cluster of Excellence BrainLinks-BrainTools, Computer Science Dept., University of Freiburg, Germany, Autonomous Intelligent Systems Lab, Computer Science Dept., University of Freiburg, Germany
| | - Paul Tubig
- Department of Philosophy, Center for Neurotechnology, University of Washington, Savery Hall, Room 361, Seattle, WA 98195
| | - Thorsten Zander
- Team PhyPA, Biological Psychology and Neuroergonomics, Technische Universität Berlin, Berlin, Germany, 7 Zander Laboratories B.V., Amsterdam, The Netherlands
| |
Collapse
|
24
|
Hendriks S, Grady C, Chiong W, Fins JJ, Ford P, Goering S, Greely HT, Hutchison K, Kelly ML, Kim SY, Klein E, Lisanby SH, Mayberg H, Maslen H, Miller FG, Ramos KM, Rommelfanger K, Sheth SA, Wexler A. Ethical Challenges of Risk, Informed Consent, and Posttrial Responsibilities in Human Research With Neural Devices: A Review. JAMA Neurol 2019; 76:1506-1514. [PMID: 31621797 PMCID: PMC9395156 DOI: 10.1001/jamaneurol.2019.3523] [Citation(s) in RCA: 42] [Impact Index Per Article: 8.4] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/21/2022]
Abstract
Importance Developing more and better diagnostic and therapeutic tools for central nervous system disorders is an ethical imperative. Human research with neural devices is important to this effort and a critical focus of the National Institutes of Health Brain Research Through Advancing Innovative Neurotechnologies (BRAIN) Initiative. Despite regulations and standard practices for conducting ethical research, researchers and others seek more guidance on how to ethically conduct neural device studies. This article draws on, reviews, specifies, and interprets existing ethical frameworks, literature, and subject matter expertise to address 3 specific ethical challenges in neural devices research: analysis of risk, informed consent, and posttrial responsibilities to research participants. Observations Research with humans proceeds after careful assessment of the risks and benefits. In assessing whether risks are justified by potential benefits in both invasive and noninvasive neural device research, the following categories of potential risks should be considered: those related to surgery, hardware, stimulation, research itself, privacy and security, and financial burdens. All 3 of the standard pillars of informed consent-disclosure, capacity, and voluntariness-raise challenges in neural device research. Among these challenges are the need to plan for appropriate disclosure of information about atypical and emerging risks of neural devices, a structured evaluation of capacity when that is in doubt, and preventing patients from feeling unduly pressured to participate. Researchers and funders should anticipate participants' posttrial needs linked to study participation and take reasonable steps to facilitate continued access to neural devices that benefit participants. Possible mechanisms for doing so are explored here. Depending on the study, researchers and funders may have further posttrial responsibilities. Conclusions and Relevance This ethical analysis and points to consider may assist researchers, institutional review boards, funders, and others engaged in human neural device research.
Collapse
Affiliation(s)
- Saskia Hendriks
- Department of Bioethics, Clinical Center, National Institutes of Health, Bethesda, MD, USA
- National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA
| | - Christine Grady
- Department of Bioethics, Clinical Center, National Institutes of Health, Bethesda, MD, USA
| | - Winston Chiong
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - Joseph J. Fins
- Division of Medical Ethics and CASBI, Weill Cornell Medical College, New York, NY, USA
| | - Paul Ford
- Center for Bioethics, Cleveland Clinic, Cleveland, OH, USA
| | - Sara Goering
- Department of Philosophy and Center for Neurotechnology, University of Washington, Seattle, WA, USA
| | | | - Katrina Hutchison
- Department of Philosophy, Macquarie University, Sydney, NSW, Australia
- Australian Research Council (ARC) Centre of Excellence for Electromaterials Science, Australia
| | - Michael L. Kelly
- Department of Neurosurgery, Case Western Reserve University School of Medicine, MetroHeath Medical Center, Cleveland, OH, USA
| | - Scott Y.H. Kim
- Department of Bioethics, Clinical Center, National Institutes of Health, Bethesda, MD, USA
| | - Eran Klein
- Department of Philosophy and Center for Neurotechnology, University of Washington, Seattle, WA, USA
- Department of Neurology, Oregon Health and Sciences, University Portland, Portland, OR, USA
| | - Sarah H. Lisanby
- Division of Translational Research, National Institute of Mental Health, Bethesda, MD, USA
| | - Helen Mayberg
- Neurology, Neurosurgery, Psychiatry and Neuroscience, Icahn School of Medicine at Mount Sinai, New York City, NY, USA
| | - Hannah Maslen
- The Oxford Uehiro Centre for Practical Ethics, University of Oxford, Oxford, UK
| | - Franklin G. Miller
- Division of Medical Ethics, Weill Cornell Medical College, New York, NY, USA
| | - Khara M. Ramos
- National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA
| | | | - Sameer A. Sheth
- Cognitive Science and Neuromodulation Program, Department of Neurological Surgery, Columbia University College of Physicians and Surgeons, New York, NY, USA
| | - Anna Wexler
- Department of Medical Ethics & Health Policy, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
25
|
Angrick M, Herff C, Mugler E, Tate MC, Slutzky MW, Krusienski DJ, Schultz T. Speech synthesis from ECoG using densely connected 3D convolutional neural networks. J Neural Eng 2019; 16:036019. [PMID: 30831567 PMCID: PMC6822609 DOI: 10.1088/1741-2552/ab0c59] [Citation(s) in RCA: 71] [Impact Index Per Article: 14.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
Abstract
OBJECTIVE Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. APPROACH Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. MAIN RESULTS In a study with six participants, we achieved correlations up to r = 0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. SIGNIFICANCE To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.
Collapse
Affiliation(s)
- Miguel Angrick
- Cognitive Systems Lab, University of Bremen, Bremen, Germany
| | | | | | | | | | | | | |
Collapse
|
26
|
Cooney C, Folli R, Coyle D. Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface. iScience 2018; 8:103-125. [PMID: 30296666 PMCID: PMC6174918 DOI: 10.1016/j.isci.2018.09.016] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2018] [Revised: 09/04/2018] [Accepted: 09/18/2018] [Indexed: 01/09/2023] Open
Abstract
A direct-speech brain-computer interface (DS-BCI) acquires neural signals corresponding to imagined speech, then processes and decodes these signals to produce a linguistic output in the form of phonemes, words, or sentences. Recent research has shown the potential of neurolinguistics to enhance decoding approaches to imagined speech with the inclusion of semantics and phonology in experimental procedures. As neurolinguistics research findings are beginning to be incorporated within the scope of DS-BCI research, it is our view that a thorough understanding of imagined speech, and its relationship with overt speech, must be considered an integral feature of research in this field. With a focus on imagined speech, we provide a review of the most important neurolinguistics research informing the field of DS-BCI and suggest how this research may be utilized to improve current experimental protocols and decoding techniques. Our review of the literature supports a cross-disciplinary approach to DS-BCI research, in which neurolinguistics concepts and methods are utilized to aid development of a naturalistic mode of communication.
Collapse
Affiliation(s)
- Ciaran Cooney
- Intelligent Systems Research Centre, Ulster University, Derry, UK.
| | - Raffaella Folli
- Institute for Research in Social Sciences, Ulster University, Jordanstown, UK
| | - Damien Coyle
- Intelligent Systems Research Centre, Ulster University, Derry, UK
| |
Collapse
|