1
|
Handelman DA, Osborn LE, Thomas TM, Badger AR, Thompson M, Nickl RW, Anaya MA, Wormley JM, Cantarero GL, McMullen D, Crone NE, Wester B, Celnik PA, Fifer MS, Tenore FV. Shared Control of Bimanual Robotic Limbs With a Brain-Machine Interface for Self-Feeding. Front Neurorobot 2022; 16:918001. [PMID: 35837250 PMCID: PMC9274256 DOI: 10.3389/fnbot.2022.918001] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2022] [Accepted: 05/25/2022] [Indexed: 11/15/2022] Open
Abstract
Advances in intelligent robotic systems and brain-machine interfaces (BMI) have helped restore functionality and independence to individuals living with sensorimotor deficits; however, tasks requiring bimanual coordination and fine manipulation continue to remain unsolved given the technical complexity of controlling multiple degrees of freedom (DOF) across multiple limbs in a coordinated way through a user input. To address this challenge, we implemented a collaborative shared control strategy to manipulate and coordinate two Modular Prosthetic Limbs (MPL) for performing a bimanual self-feeding task. A human participant with microelectrode arrays in sensorimotor brain regions provided commands to both MPLs to perform the self-feeding task, which included bimanual cutting. Motor commands were decoded from bilateral neural signals to control up to two DOFs on each MPL at a time. The shared control strategy enabled the participant to map his four-DOF control inputs, two per hand, to as many as 12 DOFs for specifying robot end effector position and orientation. Using neurally-driven shared control, the participant successfully and simultaneously controlled movements of both robotic limbs to cut and eat food in a complex bimanual self-feeding task. This demonstration of bimanual robotic system control via a BMI in collaboration with intelligent robot behavior has major implications for restoring complex movement behaviors for those living with sensorimotor deficits.
Collapse
Affiliation(s)
- David A. Handelman
- Department of Research and Exploratory Development, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, United States
| | - Luke E. Osborn
- Department of Research and Exploratory Development, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, United States
| | - Tessy M. Thomas
- Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD, United States
| | - Andrew R. Badger
- Department of Research and Exploratory Development, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, United States
| | - Margaret Thompson
- Department of Research and Exploratory Development, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, United States
| | - Robert W. Nickl
- Department of Physical Medicine and Rehabilition, Johns Hopkins Medicine, Baltimore, MD, United States
| | - Manuel A. Anaya
- Department of Physical Medicine and Rehabilition, Johns Hopkins Medicine, Baltimore, MD, United States
| | - Jared M. Wormley
- Department of Research and Exploratory Development, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, United States
| | - Gabriela L. Cantarero
- Department of Physical Medicine and Rehabilition, Johns Hopkins Medicine, Baltimore, MD, United States
| | - David McMullen
- National Institute of Mental Health, National Institutes of Health, Bethesda, MD, United States
| | - Nathan E. Crone
- Department of Neurology, Johns Hopkins Medicine, Baltimore, MD, United States
| | - Brock Wester
- Department of Research and Exploratory Development, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, United States
| | - Pablo A. Celnik
- Department of Physical Medicine and Rehabilition, Johns Hopkins Medicine, Baltimore, MD, United States
| | - Matthew S. Fifer
- Department of Research and Exploratory Development, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, United States
| | - Francesco V. Tenore
- Department of Research and Exploratory Development, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, United States
- *Correspondence: Francesco V. Tenore
| |
Collapse
|
2
|
Characteristics and stability of sensorimotor activity driven by isolated-muscle group activation in a human with tetraplegia. Sci Rep 2022; 12:10353. [PMID: 35725741 PMCID: PMC9209428 DOI: 10.1038/s41598-022-13436-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2022] [Accepted: 05/17/2022] [Indexed: 11/09/2022] Open
Abstract
Understanding the cortical representations of movements and their stability can shed light on improved brain-machine interface (BMI) approaches to decode these representations without frequent recalibration. Here, we characterize the spatial organization (somatotopy) and stability of the bilateral sensorimotor map of forearm muscles in an incomplete-high spinal-cord injury study participant implanted bilaterally in the primary motor and sensory cortices with Utah microelectrode arrays (MEAs). We built representation maps by recording bilateral multiunit activity (MUA) and surface electromyography (EMG) as the participant executed voluntary contractions of the extensor carpi radialis (ECR), and attempted motions in the flexor carpi radialis (FCR), which was paralytic. To assess stability, we repeatedly mapped and compared left- and right-wrist-extensor-related activity throughout several sessions, comparing somatotopy of active electrodes, as well as neural signals both at the within-electrode (multiunit) and cross-electrode (network) levels. Wrist motions showed significant activation in motor and sensory cortical electrodes. Within electrodes, firing strength stability diminished as the time increased between consecutive measurements (hours within a session, or days across sessions), with higher stability observed in sensory cortex than in motor, and in the contralateral hemisphere than in the ipsilateral. However, we observed no differences at network level, and no evidence of decoding instabilities for wrist EMG, either across timespans of hours or days, or across recording area. While map stability differs between brain area and hemisphere at multiunit/electrode level, these differences are nullified at ensemble level.
Collapse
|
4
|
Pandarinath C, Bensmaia SJ. The science and engineering behind sensitized brain-controlled bionic hands. Physiol Rev 2022; 102:551-604. [PMID: 34541898 PMCID: PMC8742729 DOI: 10.1152/physrev.00034.2020] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2020] [Revised: 09/07/2021] [Accepted: 09/13/2021] [Indexed: 12/13/2022] Open
Abstract
Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.
Collapse
Affiliation(s)
- Chethan Pandarinath
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, Georgia
- Department of Neurosurgery, Emory University, Atlanta, Georgia
| | - Sliman J Bensmaia
- Department of Organismal Biology and Anatomy, University of Chicago, Chicago, Illinois
- Committee on Computational Neuroscience, University of Chicago, Chicago, Illinois
- Grossman Institute for Neuroscience, Quantitative Biology, and Human Behavior, University of Chicago, Chicago, Illinois
| |
Collapse
|
5
|
Fernandez E. Selective Induction of Fingertip Sensations for Better Neuroprosthetic Control. Neurology 2021; 98:261-262. [PMID: 34880097 DOI: 10.1212/wnl.0000000000013177] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] Open
Affiliation(s)
- Eduardo Fernandez
- Bioengineering Institute and Cátedra Bidons Egara, University Miguel Hernández, 03202 Elche, Spain. .,CIBER Research Center on Bioengineering, Biomaterials and Nanomedicine (CIBER BBN), 28029 Madrid, Spain.,John Moran Eye Center and Biomedical Engineering, University of Utah, 84132 Salt Lake City, USA
| |
Collapse
|
6
|
Osborn LE, Christie BP, McMullen DP, Nickl RW, Thompson MC, Pawar AS, Thomas TM, Alejandro Anaya M, Crone NE, Wester BA, Bensmaia SJ, Celnik PA, Cantarero GL, Tenore FV, Fifer MS. Intracortical microstimulation of somatosensory cortex enables object identification through perceived sensations. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:6259-6262. [PMID: 34892544 DOI: 10.1109/embc46164.2021.9630450] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Advances in brain-machine interfaces have helped restore function and independence for individuals with sensorimotor deficits; however, providing efficient and effective sensory feedback remains challenging. Intracortical microstimulation (ICMS) of sensorimotor brain regions is a promising technique for providing bioinspired sensory feedback. In a human participant with chronically-implanted microelectrode arrays, we provided ICMS to the primary somatosensory cortex to generate tactile percepts in his hand. In a 3-choice object identification task, the participant identified virtual objects using tactile sensory feedback and no visual information. We evaluated three different stimulation paradigms, each with a different weighting of the grip force and its derivative, to explore the potential benefits of a more bioinspired stimulation strategy. In all paradigms, the participant's ability to identify the objects was above-chance, with object identification accuracy reaching 80% correct when using only sustained grip force feedback and 76.7% when using equal weighting of both sustained grip force and its derivative. These results demonstrate that bioinspired ICMS can provide sensory feedback that is functionally beneficial in sensorimotor tasks. Designing more efficient stimulation paradigms is important because it will allow us to 1) provide safer stimulation delivery methods that reduce overall injected charge without sacrificing function and 2) more effectively transmit sensory information to promote intuitive integration and usage by the human body.
Collapse
|
7
|
Chandrasekaran S, Fifer M, Bickel S, Osborn L, Herrero J, Christie B, Xu J, Murphy RKJ, Singh S, Glasser MF, Collinger JL, Gaunt R, Mehta AD, Schwartz A, Bouton CE. Historical perspectives, challenges, and future directions of implantable brain-computer interfaces for sensorimotor applications. Bioelectron Med 2021; 7:14. [PMID: 34548098 PMCID: PMC8456563 DOI: 10.1186/s42234-021-00076-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Accepted: 08/29/2021] [Indexed: 11/10/2022] Open
Abstract
Almost 100 years ago experiments involving electrically stimulating and recording from the brain and the body launched new discoveries and debates on how electricity, movement, and thoughts are related. Decades later the development of brain-computer interface technology began, which now targets a wide range of applications. Potential uses include augmentative communication for locked-in patients and restoring sensorimotor function in those who are battling disease or have suffered traumatic injury. Technical and surgical challenges still surround the development of brain-computer technology, however, before it can be widely deployed. In this review we explore these challenges, historical perspectives, and the remarkable achievements of clinical study participants who have bravely forged new paths for future beneficiaries.
Collapse
Affiliation(s)
- Santosh Chandrasekaran
- Neural Bypass and Brain Computer Interface Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
| | - Matthew Fifer
- Research and Exploratory Development Department, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, USA
| | - Stephan Bickel
- The Human Brain Mapping Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
- Department of Neurosurgery, Barbara Zucker School of Medicine at Hofstra/Northwell, Manhasset, NY, USA
- Department of Neurology, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Manhasset, NY, USA
| | - Luke Osborn
- Research and Exploratory Development Department, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, USA
| | - Jose Herrero
- The Human Brain Mapping Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
| | - Breanne Christie
- Research and Exploratory Development Department, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, USA
| | - Junqian Xu
- Departments of Radiology and Psychiatry, Baylor College of Medicine, Houston, TX, USA
| | - Rory K J Murphy
- Department of Neurosurgery, Barrow Neurological Institute, St. Joseph's Hospital and Medical Center, Phoenix, AZ, USA
| | - Sandeep Singh
- Good Shepherd Rehabilitation Hospital, Allentown, PA, USA
| | - Matthew F Glasser
- Departments of Radiology and Neuroscience, Washington University in St Louis, Saint Louis, MO, USA
| | - Jennifer L Collinger
- Rehabilitation Neural Engineering Labs, University of Pittsburgh, Pittsburgh, PA, USA
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA
- Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA
| | - Robert Gaunt
- Rehabilitation Neural Engineering Labs, University of Pittsburgh, Pittsburgh, PA, USA
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA
- Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA
| | - Ashesh D Mehta
- The Human Brain Mapping Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
- Department of Neurosurgery, Barbara Zucker School of Medicine at Hofstra/Northwell, Manhasset, NY, USA
| | - Andrew Schwartz
- McGowan Institute of Regenerative Medicine, University of Pittsburgh, Pittsburgh, PA, USA
| | - Chad E Bouton
- Neural Bypass and Brain Computer Interface Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA.
- Department of Molecular Medicine, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Manhasset, NY, USA.
| |
Collapse
|