Predicting 3D lip shapes using facial surface EMG

Correlation of 3D Morphometric Changes, Kinematics, and Muscle Activity During Smile

OBJECTIVE

Knowing the morphological, kinematic, and electrophysiological parameters of the smile in healthy individuals may contribute to evaluating, planning, and monitoring the smile reanimation. This study aimed to determine the correlation between 3D morphometric changes, movement kinematics, and muscle activity in the facial soft tissue of healthy individuals.

METHOD

In this cohort study, 20 volunteers were selected from healthy individuals with no facial disorders. During smiling, three-dimensional face scanning, facial motion capture, and surface electromyography (sEMG) were performed. The average displacement, velocity, and acceleration during facial movements were measured. The mean change in 3D surface morphometry and activation of the zygomaticus major were determined.

RESULTS

The volunteers, comprising 10 males and 10 females, had a mean age of 24 ± 10 years; for female, mean age was 23 ± 5 years and for men 26 ± 13 years. Significant correlations were found between kinematic and morphometric data (r = 0.51, p < 0.001), sEMG and morphometric (r = 0.50, p < 0.001) data, and sEMG and kinematic data (r = 0.49, p < 0.002). The maximum acceleration occurred during approximately 65% of the muscle activation time and 64% of the peak muscle activation value. Additionally, the maximum velocity was reached at around 73% of the muscle activation time and 67% of the peak muscle activation value. Furthermore, the maximum displacement values were observed at approximately 88% of the muscle activation time and 76% of the peak muscle activation value.

CONCLUSION

The findings may provide insights into the smile's functional parameters, contribute to understanding facial muscle-related disorders, and aid in improving the diagnosis and treatment of the smile.

LEVEL OF EVIDENCE

NA Laryngoscope, 134:3112-3119, 2024.

Prediction of Voice Fundamental Frequency and Intensity from Surface Electromyographic Signals of the Face and Neck

Silent speech interfaces (SSIs) enable speech recognition and synthesis in the absence of an acoustic signal. Yet, the archetypal SSI fails to convey the expressive attributes of prosody such as pitch and loudness, leading to lexical ambiguities. The aim of this study was to determine the efficacy of using surface electromyography (sEMG) as an approach for predicting continuous acoustic estimates of prosody. Ten participants performed a series of vocal tasks including sustained vowels, phrases, and monologues while acoustic data was recorded simultaneously with sEMG activity from muscles of the face and neck. A battery of time-, frequency-, and cepstral-domain features extracted from the sEMG signals were used to train deep regression neural networks to predict fundamental frequency and intensity contours from the acoustic signals. We achieved an average accuracy of 0.01 ST and precision of 0.56 ST for the estimation of fundamental frequency, and an average accuracy of 0.21 dB SPL and precision of 3.25 dB SPL for the estimation of intensity. This work highlights the importance of using sEMG as an alternative means of detecting prosody and shows promise for improving SSIs in future development.

Surface Electromyography-Based Recognition, Synthesis, and Perception of Prosodic Subvocal Speech

Purpose This study aimed to evaluate a novel communication system designed to translate surface electromyographic (sEMG) signals from articulatory muscles into speech using a personalized, digital voice. The system was evaluated for word recognition, prosodic classification, and listener perception of synthesized speech. Method sEMG signals were recorded from the face and neck as speakers with (n = 4) and without (n = 4) laryngectomy subvocally recited (silently mouthed) a speech corpus comprising 750 phrases (150 phrases with variable phrase-level stress). Corpus tokens were then translated into speech via personalized voice synthesis (n = 8 synthetic voices) and compared against phrases produced by each speaker when using their typical mode of communication (n = 4 natural voices, n = 4 electrolaryngeal [EL] voices). Naïve listeners (n = 12) evaluated synthetic, natural, and EL speech for acceptability and intelligibility in a visual sort-and-rate task, as well as phrasal stress discriminability via a classification mechanism. Results Recorded sEMG signals were processed to translate sEMG muscle activity into lexical content and categorize variations in phrase-level stress, achieving a mean accuracy of 96.3% (SD = 3.10%) and 91.2% (SD = 4.46%), respectively. Synthetic speech was significantly higher in acceptability and intelligibility than EL speech, also leading to greater phrasal stress classification accuracy, whereas natural speech was rated as the most acceptable and intelligible, with the greatest phrasal stress classification accuracy. Conclusion This proof-of-concept study establishes the feasibility of using subvocal sEMG-based alternative communication not only for lexical recognition but also for prosodic communication in healthy individuals, as well as those living with vocal impairments and residual articulatory function. Supplemental Material https://doi.org/10.23641/asha.14558481.

Can we decode phonetic features in inner speech using surface electromyography?

Although having a long history of scrutiny in experimental psychology, it is still controversial whether wilful inner speech (covert speech) production is accompanied by specific activity in speech muscles. We present the results of a preregistered experiment looking at the electromyographic correlates of both overt speech and inner speech production of two phonetic classes of nonwords. An automatic classification approach was undertaken to discriminate between two articulatory features contained in nonwords uttered in both overt and covert speech. Although this approach led to reasonable accuracy rates during overt speech production, it failed to discriminate inner speech phonetic content based on surface electromyography signals. However, exploratory analyses conducted at the individual level revealed that it seemed possible to distinguish between rounded and spread nonwords covertly produced, in two participants. We discuss these results in relation to the existing literature and suggest alternative ways of testing the engagement of the speech motor system during wilful inner speech production.

IMAGE-BASED SKELETAL MUSCLE COORDINATION: CASE STUDY ON A SUBJECT SPECIFIC FACIAL MIMIC SIMULATION

Facial muscle coordination is a fundamental mechanism for facial mimics and expressions. The understanding of this complex mechanism leads to better diagnosis and treatment of facial disorders like facial palsy or disfigurement. The objective of this work was to use magnetic resonance imaging (MRI) technique to characterize the activation behavior of facial muscles and then simulate their coordination mechanism using a subject specific finite element model. MRI data of lower head of a healthy subject were acquired in neutral and in the pronunciation of the sound [o] positions. Then, a finite element model was derived directly from acquired MRI images in neutral position. Transversely-isotropic, hyperelastic, quasi-incompressible behavior law was implemented for modeling facial muscles. The simulation to produce the pronunciation of the sound [o] was performed by the cumulative coordination between three pairs of facial mimic muscles (Zygomaticus Major (ZM), Levator Labii Superioris (LLS), Levator Anguli Oris (LAO)). Mean displacement amplitude showed a good agreement with a relative deviation of 15% between numerical outcome and MRI-based measurement when all three muscles are involved. This study elucidates, for the first time, the facial muscle coordination using in vivo data leading to improve the model understanding and simulation outcomes.

sEMG-assisted inverse modelling of 3D lip movement: a feasibility study towards person-specific modelling

We propose a surface-electromyographic (sEMG) assisted inverse-modelling (IM) approach for a biomechanical model of the face to obtain realistic person-specific muscle activations (MA) by tracking movements as well as innervation trajectories. We obtained sEMG data of facial muscles and 3D positions of lip markers in six volunteers and, using a generic finite element (FE) face model in ArtiSynth, performed inverse static optimisation with and without sEMG tracking on both simulation data and experimental data. IM with simulated data and experimental data without sEMG data showed good correlations of tracked positions (0.93 and 0.67) and poor correlations of MA (0.27 and 0.20). When utilising the sEMG-assisted IM approach, MA correlations increased drastically (0.83 and 0.59) without sacrificing performance in position correlations (0.92 and 0.70). RMS errors show similar trends with an error of 0.15 in MA and of 1.10 mm in position. Therefore, we conclude that we were able to demonstrate the feasibility of an sEMG-assisted inverse modelling algorithm for the perioral region. This approach may help to solve the ambiguity problem in inverse modelling and may be useful, for instance, in future applications for preoperatively predicting treatment-related function loss.

Simulation of facial expressions using person-specific sEMG signals controlling a biomechanical face model

PURPOSE

Functional inoperability in advanced oral cancer is difficult to assess preoperatively. To assess functions of lips and tongue, biomechanical models are required. Apart from adjusting generic models to individual anatomy, muscle activation patterns (MAPs) driving patient-specific functional movements are necessary to predict remaining functional outcome. We aim to evaluate how volunteer-specific MAPs derived from surface electromyographic (sEMG) signals control a biomechanical face model.

METHODS

Muscle activity of seven facial muscles in six volunteers was measured bilaterally with sEMG. A triple camera set-up recorded 3D lip movement. The generic face model in ArtiSynth was adapted to our needs. We controlled the model using the volunteer-specific MAPs. Three activation strategies were tested: activating all muscles [Formula: see text], selecting the three muscles showing highest muscle activity bilaterally [Formula: see text]-this was calculated by taking the mean of left and right muscles and then selecting the three with highest variance-and activating the muscles considered most relevant per instruction [Formula: see text], bilaterally. The model's lip movement was compared to the actual lip movement performed by the volunteers, using 3D correlation coefficients [Formula: see text].

RESULTS

The correlation coefficient between simulations and measurements with [Formula: see text] resulted in a median [Formula: see text] of 0.77. [Formula: see text] had a median [Formula: see text] of 0.78, whereas with [Formula: see text] the median [Formula: see text] decreased to 0.45.

CONCLUSION

We demonstrated that MAPs derived from noninvasive sEMG measurements can control movement of the lips in a generic finite element face model with a median [Formula: see text] of 0.78. Ultimately, this is important to show the patient-specific residual movement using the patient's own MAPs. When the required treatment tools and personalisation techniques for geometry and anatomy become available, this may enable surgeons to test the functional results of wedge excisions for lip cancer in a virtual environment and to weigh surgery versus organ-sparing radiotherapy or photodynamic therapy.