1
|
Patwardhan S, Gladhill KA, Joiner WM, Schofield JS, Lee BS, Sikdar S. Using principles of motor control to analyze performance of human machine interfaces. Sci Rep 2023; 13:13273. [PMID: 37582852 PMCID: PMC10427694 DOI: 10.1038/s41598-023-40446-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Accepted: 08/10/2023] [Indexed: 08/17/2023] Open
Abstract
There have been significant advances in biosignal extraction techniques to drive external biomechatronic devices or to use as inputs to sophisticated human machine interfaces. The control signals are typically derived from biological signals such as myoelectric measurements made either from the surface of the skin or subcutaneously. Other biosignal sensing modalities are emerging. With improvements in sensing modalities and control algorithms, it is becoming possible to robustly control the target position of an end-effector. It remains largely unknown to what extent these improvements can lead to naturalistic human-like movement. In this paper, we sought to answer this question. We utilized a sensing paradigm called sonomyography based on continuous ultrasound imaging of forearm muscles. Unlike myoelectric control strategies which measure electrical activation and use the extracted signals to determine the velocity of an end-effector; sonomyography measures muscle deformation directly with ultrasound and uses the extracted signals to proportionally control the position of an end-effector. Previously, we showed that users were able to accurately and precisely perform a virtual target acquisition task using sonomyography. In this work, we investigate the time course of the control trajectories derived from sonomyography. We show that the time course of the sonomyography-derived trajectories that users take to reach virtual targets reflect the trajectories shown to be typical for kinematic characteristics observed in biological limbs. Specifically, during a target acquisition task, the velocity profiles followed a minimum jerk trajectory shown for point-to-point arm reaching movements, with similar time to target. In addition, the trajectories based on ultrasound imaging result in a systematic delay and scaling of peak movement velocity as the movement distance increased. We believe this is the first evaluation of similarities in control policies in coordinated movements in jointed limbs, and those based on position control signals extracted at the individual muscle level. These results have strong implications for the future development of control paradigms for assistive technologies.
Collapse
Affiliation(s)
| | - Keri Anne Gladhill
- Department of Psychology, George Mason University, Fairfax, VA, 22030, USA
| | - Wilsaan M Joiner
- Department of Neurobiology, Physiology and Behavior, University of California, Davis, Davis, CA, 95616, USA
| | - Jonathon S Schofield
- Mechanical and Aerospace Engineering Department, University of California, Davis, Davis, CA, 95616, USA
| | - Ben Seiyon Lee
- Department of Statistics, George Mason University, Fairfax, VA, 22030, USA
| | - Siddhartha Sikdar
- Department of Bioengineering, George Mason University, Fairfax, VA, 22030, USA.
- Center for Adaptive Systems of Brain-Body Interactions, Fairfax, VA, 22030, USA.
| |
Collapse
|
2
|
Kim C, Kim C, Kim H, Kwak H, Lee W, Im CH. Facial electromyogram-based facial gesture recognition for hands-free control of an AR/VR environment: optimal gesture set selection and validation of feasibility as an assistive technology. Biomed Eng Lett 2023; 13:465-473. [PMID: 37519877 PMCID: PMC10382369 DOI: 10.1007/s13534-023-00277-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2023] [Revised: 04/01/2023] [Accepted: 04/03/2023] [Indexed: 08/01/2023] Open
Abstract
The rapid expansion of virtual reality (VR) and augmented reality (AR) into various applications has increased the demand for hands-free input interfaces when traditional control methods are inapplicable (e.g., for paralyzed individuals who cannot move their hands). Facial electromyogram (fEMG), bioelectric signals generated from facial muscles, could solve this problem. Discriminating facial gestures using fEMG is possible because fEMG signals vary with these gestures. Thus, these signals can be used to generate discrete hands-free control commands. This study implemented an fEMG-based facial gesture recognition system for generating discrete commands to control an AR or VR environment. The fEMG signals around the eyes were recorded, assuming that the fEMG electrodes were embedded into the VR head-mounted display (HMD). Sixteen discrete facial gestures were classified using linear discriminant analysis (LDA) with Riemannian geometry features. Because the fEMG electrodes were far from the facial muscles associated with the facial gestures, some similar facial gestures were indistinguishable from each other. Therefore, this study determined the best facial gesture combinations with the highest classification accuracy for 3-15 commands. An analysis of the fEMG data acquired from 15 participants showed that the optimal facial gesture combinations increased the accuracy by 4.7%p compared with randomly selected facial gesture combinations. Moreover, this study is the first to investigate the feasibility of implementing a subject-independent facial gesture recognition system that does not require individual user training sessions. Lastly, our online hands-free control system was successfully applied to a media player to demonstrate the applicability of the proposed system. Supplementary Information The online version contains supplementary material available at 10.1007/s13534-023-00277-9.
Collapse
Affiliation(s)
- Chunghwan Kim
- Department of Electronic Engineering, Hanyang University, Seoul, 04763 Republic of Korea
| | - Chaeyoon Kim
- Department of HY-KIST Bio-Convergence, Hanyang University, Seoul, 04763 Republic of Korea
| | - HyunSub Kim
- Department of Electronic Engineering, Hanyang University, Seoul, 04763 Republic of Korea
| | - HwyKuen Kwak
- Hanwha Systems Co., Ltd., Seongnam, 13524 Republic of Korea
| | - WooJin Lee
- Korea Research Institute for Defense Technology Planning and Advancement, Jinju, 52851 Republic of Korea
| | - Chang-Hwan Im
- Department of Electronic Engineering, Hanyang University, Seoul, 04763 Republic of Korea
- Department of HY-KIST Bio-Convergence, Hanyang University, Seoul, 04763 Republic of Korea
- Department of Biomedical Engineering, Hanyang University, Seoul, 04763 Republic of Korea
| |
Collapse
|
3
|
Patwardhan S, Gladhill KA, Joiner WM, Schofield JS, Sikdar S. Using Principles of Motor Control to Analyze Performance of Human Machine Interfaces. RESEARCH SQUARE 2023:rs.3.rs-2763325. [PMID: 37292730 PMCID: PMC10246101 DOI: 10.21203/rs.3.rs-2763325/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
There have been significant advances in biosignal extraction techniques to drive external biomechatronic devices or to use as inputs to sophisticated human machine interfaces. The control signals are typically derived from biological signals such as myoelectric measurements made either from the surface of the skin or subcutaneously. Other biosignal sensing modalities are emerging. With improvements in sensing modalities and control algorithms, it is becoming possible to robustly control the target position of a end effector. It remains largely unknown to what extent these improvements can lead to naturalistic human-like movement. In this paper, we sought to answer this question. We utilized a sensing paradigm called sonomyography based on continuous ultrasound imaging of forearm muscles. Unlike myoelectric control strategies which measure electrical activation and use the extracted signals to determine the velocity of an end-effector; sonomyography measures muscle deformation directly with ultrasound and uses the extracted signals to proportionally control the position of an end-effector. Previously, we showed that users were able to accurately and precisely perform a virtual target acquisition task using sonomyography. In this work, we investigate the time course of the control trajectories derived from sonomyography. We show that the time course of the sonomyography-derived trajectories that users take to reach virtual targets reflect the trajectories shown to be typical for kinematic characteristics observed in biological limbs. Specifically, during a target acquisition task, the velocity profiles followed a minimum jerk trajectory shown for point-to-point arm reaching movements, with similar time to target. In addition, the trajectories based on ultrasound imaging result in a systematic delay and scaling of peak movement velocity as the movement distance increased. We believe this is the first evaluation of similarities in control policies in coordinated movements in jointed limbs, and those based on position control signals extracted at the individual muscle level. These results have strong implications for the future development of control paradigms for assistive technologies.
Collapse
Affiliation(s)
| | - Keri Anne Gladhill
- Department of Psychology, George Mason University, Fairfax, VA, 22030, USA
| | - Wilsaan M. Joiner
- Department of Neurobiology, Physiology and Behavior, University of California, Davis, Davis, CA, 95616, USA
| | - Jonathon S. Schofield
- Mechanical and Aerospace Engineering Department, University of California, Davis, Davis, CA, 95616, USA
| | - Siddhartha Sikdar
- Department of Bioengineering, George Mason University, Fairfax VA, 22030, USA
- Center for Adaptive Systems of Brain-Body Interactions, Fairfax VA, 22030, USA
| |
Collapse
|
4
|
Sato W, Kochiyama T. Crosstalk in Facial EMG and Its Reduction Using ICA. SENSORS (BASEL, SWITZERLAND) 2023; 23:2720. [PMID: 36904924 PMCID: PMC10007323 DOI: 10.3390/s23052720] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 02/24/2023] [Accepted: 02/28/2023] [Indexed: 06/18/2023]
Abstract
There is ample evidence that electromyography (EMG) signals from the corrugator supercilii and zygomatic major muscles can provide valuable information for the assessment of subjective emotional experiences. Although previous research suggested that facial EMG data could be affected by crosstalk from adjacent facial muscles, it remains unproven whether such crosstalk occurs and, if so, how it can be reduced. To investigate this, we instructed participants (n = 29) to perform the facial actions of frowning, smiling, chewing, and speaking, in isolation and combination. During these actions, we measured facial EMG signals from the corrugator supercilii, zygomatic major, masseter, and suprahyoid muscles. We performed an independent component analysis (ICA) of the EMG data and removed crosstalk components. Speaking and chewing induced EMG activity in the masseter and suprahyoid muscles, as well as the zygomatic major muscle. The ICA-reconstructed EMG signals reduced the effects of speaking and chewing on zygomatic major activity, compared with the original signals. These data suggest that: (1) mouth actions could induce crosstalk in zygomatic major EMG signals, and (2) ICA can reduce the effects of such crosstalk.
Collapse
Affiliation(s)
- Wataru Sato
- Psychological Process Research Team, Guardian Robot Project, RIKEN, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan
- Field Science Education and Research Center, Kyoto University, Oiwake-cho, Kitashirakawa, Kyoto 606-8502, Japan
| | - Takanori Kochiyama
- Brain Activity Imaging Center, ATR-Promotions, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan
| |
Collapse
|
5
|
Bouyam C, Punsawad Y. Human–machine interface-based wheelchair control using piezoelectric sensors based on face and tongue movements. Heliyon 2022; 8:e11679. [DOI: 10.1016/j.heliyon.2022.e11679] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Revised: 08/29/2022] [Accepted: 11/10/2022] [Indexed: 11/20/2022] Open
|
6
|
Zhu B, Zhang D, Chu Y, Gu Y, Zhao X. SeNic: An Open Source Dataset for sEMG-Based Gesture Recognition in Non-ideal Conditions. IEEE Trans Neural Syst Rehabil Eng 2022; 30:1252-1260. [PMID: 35533170 DOI: 10.1109/tnsre.2022.3173708] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
In order to reduce the gap between the laboratory environment and actual use in daily life of human-machine interaction based on surface electromyogram (sEMG) intent recognition, this paper presents a benchmark dataset of sEMG in non-ideal conditions (SeNic). The dataset mainly consists of 8-channel sEMG signals, and electrode shifts from an 3D-printed annular ruler. A total of 36 subjects participate in our data acquisition experiments of 7 gestures in non-ideal conditions, where non-ideal factors of 1) electrode shifts, 2) individual difference, 3) muscle fatigue, 4) inter-day difference, and 5) arm postures are elaborately involved. Signals of sEMG are validated first in temporal and frequency domains. Results of recognizing gestures in ideal conditions indicate the high quality of the dataset. Adverse impacts in non-ideal conditions are further revealed in the amplitudes of these data and recognition accuracies. To be concluded, SeNic is a benchmark dataset that introduces several non-ideal factors which often degrade the robustness of sEMG-based systems. It could be used as a freely available dataset and a common platform for researchers in the sEMG-based recognition community. The benchmark dataset SeNic are available online via the website3.
Collapse
|