MRI-based finite element modeling of facial mimics: a case study on the paired zygomaticus major muscles

Novel Baseline Facial Muscle Database Using Statistical Shape Modeling and In Silico Trials toward Decision Support for Facial Rehabilitation

Backgrounds and Objective: Facial palsy is a complex pathophysiological condition affecting the personal and professional lives of the involved patients. Sudden muscle weakness or paralysis needs to be rehabilitated to recover a symmetric and expressive face. Computer-aided decision support systems for facial rehabilitation have been developed. However, there is a lack of facial muscle baseline data to evaluate the patient states and guide as well as optimize the rehabilitation strategy. In this present study, we aimed to develop a novel baseline facial muscle database (static and dynamic behaviors) using the coupling between statistical shape modeling and in-silico trial approaches. Methods: 10,000 virtual subjects (5000 males and 5000 females) were generated from a statistical shape modeling (SSM) head model. Skull and muscle networks were defined so that they statistically fit with the head shapes. Two standard mimics: smiling and kissing were generated. The muscle strains of the lengths in neutral and mimic positions were computed and recorded thanks to the muscle insertion and attachment points on the animated head and skull meshes. For validation, five head and skull meshes were reconstructed from the five computed tomography (CT) image sets. Skull and muscle networks were then predicted from the reconstructed head meshes. The predicted skull meshes were compared with the reconstructed skull meshes based on the mesh-to-mesh distance metrics. The predicted muscle lengths were also compared with those manually defined on the reconstructed head and skull meshes. Moreover, the computed muscle lengths and strains were compared with those in our previous studies and the literature. Results: The skull prediction's median deviations from the CT-based models were 2.2236 mm, 2.1371 mm, and 2.1277 mm for the skull shape, skull mesh, and muscle attachment point regions, respectively. The median deviation of the muscle lengths was 4.8940 mm. The computed muscle strains were compatible with the reported values in our previous Kinect-based method and the literature. Conclusions: The development of our novel facial muscle database opens new avenues to accurately evaluate the facial muscle states of facial palsy patients. Based on the evaluated results, specific types of facial mimic rehabilitation exercises can also be selected optimally to train the target muscles. In perspective, the database of the computed muscle lengths and strains will be integrated into our available clinical decision support system for automatically detecting malfunctioning muscles and proposing patient-specific rehabilitation serious games.

A biophysically guided constitutive law of the musculotendon-complex: modelling and numerical implementation in Abaqus

BACKGROUND AND OBJECTIVE

Many biomedical, clinical, and industrial applications may benefit from musculoskeletal simulations. Three-dimensional macroscopic muscle models (3D models) can more accurately represent muscle architecture than their 1D (line-segment) counterparts. Nevertheless, 3D models remain underutilised in academic, clinical, and commercial environments. Among the reasons for this is a lack of modelling and simulation standardisation, verification, and validation. Here, we strive towards a solution by providing an open-access, characterised, constitutive relation (CR) for 3D musculotendon models.

METHODS

The musculotendon complex is modelled following the state-of-the-art active stress approach and is treated as hyperelastic, transversely isotropic, and nearly incompressible. Furthermore, force-length and -velocity relationships are incorporated, and muscle activation is derived from motor-unit information. The CR was implemented within the commercial finite-element software package Abaqus as a user-subroutine. A masticatory system model with left and right masseters was used to demonstrate active and passive movement.

RESULTS

The CR was characterised by various experimental data sets and was able to capture a wide variety of passive and active behaviours. Furthermore, the masticatory simulations revealed that joint movement was sensitive to the muscle's in-fibre passive response.

CONCLUSIONS

This user-material provides a "plug and play" template for 3D neuro-musculoskeletal finite element modelling. We hope that this reduces modelling effort, fosters exchange, and contributes to the standardisation of such models.

Reinforcement learning coupled with finite element modeling for facial motion learning

BACKGROUND AND OBJECTIVE

Facial palsy patients or patients with facial transplantation have abnormal facial motion due to altered facial muscle functions and nerve damage. Computer-aided system and physics-based models have been developed to provide objective and quantitative information. However, the predictive capacity of these solutions is still limited to explore the facial motion patterns with emerging properties. The present study aims to couple the reinforcement learning and the finite element modeling for facial motion learning and prediction.

METHODS

A novel modeling workflow for learning facial motion was developed. A physically-based model of the face within the Artisynth modeling platform was used. Information exchange protocol was proposed to link reinforcement learning and rigid multi-bodies dynamics outcomes. Two reinforcement learning algorithms (deep deterministic policy gradient (DDPG) and Twin-delayed DDPG (TD3)) were used and implemented to drive the simulations of symmetry-oriented and smile movements. Numerical outcomes were compared to experimental observations (Bosphorus database) for evaluation and validation purposes.

RESULTS

As result, after more than 100 episodes of exploring the environment, the agent starts to learn from previous trials and can find the optimal policy after more than 300 episodes of training. Regarding the symmetry-oriented motion, the muscle excitations predicted by the trained agent help to increase the value of reward from R = -2.06 to R = -0.23, which counts for ∼89% improvement of the symmetry value of the face. For smile-oriented motion, two points at the edge of the mouth move up 0.35 cm, which is within the range of movements estimated from the Bosphorus database (0.4 ± 0.32 cm).

CONCLUSIONS

The present study explored the muscle excitation patterns by coupling reinforcement learning with a detailed finite element model of the face. We developed, for the first time, a novel coupling scheme to integrate the finite element simulation into the reinforcement learning process for facial motion learning. As perspectives, this present workflow will be applied for facial palsy and facial transplantation patients to guide and optimize the functional rehabilitation program.

Enhanced head-skull shape learning using statistical modeling and topological features

Skull prediction from the head is a challenging issue toward a cost-effective therapeutic solution for facial disorders. This issue was initially studied in our previous work using full head-to-skull relationship learning. However, the head-skull thickness topology is locally shaped, especially in the face region. Thus, the objective of the present study was to enhance our head-to-skull prediction problem by using local topological features for training and predicting. Head and skull feature points were sampled on 329 head and skull models from computed tomography (CT) images. These feature points were classified into the back and facial topologies. Head-to-skull relations were trained using the partial least square regression (PLSR) models separately in the two topologies. A hyperparameter tuning process was also conducted for selecting optimal parameters for each training model. Thus, a new skull could be generated so that its shape was statistically fitted with the target head. Mean errors of the predicted skulls using the topology-based learning method were better than those using the non-topology-based learning method. After tenfold cross-validation, the mean error was enhanced 36.96% for the skull shapes and 14.17% for the skull models. Mean error in the facial skull region was especially improved with 4.98%. The mean errors were also improved 11.71% and 25.74% in the muscle attachment regions and the back skull regions respectively. Moreover, using the enhanced learning strategy, the errors (mean ± SD) for the best and worst prediction cases are from 1.1994 ± 1.1225 mm (median: 0.9036, coefficient of multiple determination (R²): 0.997274) to 3.6972 ± 2.4118 mm (median: 3.9089, R²: 0.999614) and from 2.0172 ± 2.0454 mm (median: 1.2999, R²: 0.995959) to 4.0227 ± 2.6098 mm (median: 3.9998, R²: 0.998577) for the predicted skull shapes and the predicted skull models respectively. This present study showed that more detailed information on the head-skull shape leads to a better accuracy level for the skull prediction from the head. In particular, local topological features on the back and face regions of interest should be considered toward a better learning strategy for the head-to-skull prediction problem. In perspective, this enhanced learning strategy was used to update our developed clinical decision support system for facial disorders. Furthermore, a new class of learning methods, called geometric deep learning will be studied.

Operation Safety of a 2-DoF Planar Mechanism for Arm Rehabilitation

The operation safety of rehabilitation devices must be addressed early in the development process and before being tested on people. In this paper, the operation safety of a 2-DoF (degrees of freedom) planar mechanism for arm rehabilitation is addressed. Then, the safety and efficiency of the device operation is assessed through the Transmission Index (TI) distribution in its workspace. Furthermore, the produced stresses on the human arm are assessed via the FEM (finite element method) when the rehabilitation device reaches five critical positions within its workspace. The TI distribution showed that the proposed design has a proper behaviour from a force transmission point of view, avoiding any singular configuration that might cause a control failure and subsequent risk for the user and supporting the user’s motion with a good efficiency throughout its operational workspace. The FEM analysis showed that Nurse operation is safe for the human arm since a negligible maximum stress of 6.55 × 103 N/m2 is achieved by the human arm when the device is located on the evaluated critical positions.

Kinect-driven Patient-specific Head, Skull, and Muscle Network Modelling for Facial Palsy Patients

BACKGROUND AND OBJECTIVE

Facial palsy negatively affects both professional and personal life qualities of involved patients. Classical facial rehabilitation strategies can recover facial mimics into their normal and symmetrical movements and appearances. However, there is a lack of objective, quantitative, and in-vivo facial texture and muscle activation bio-feedbacks for personalizing rehabilitation programs and diagnosing recovering progresses. Consequently, this study proposed a novel patient-specific modelling method for generating a full patient specific head model from a visual sensor and then computing the facial texture and muscle activation in real-time for further clinical decision making.

METHODS

The modeling workflow includes (1) Kinect-to-head, (2) head-to-skull, and (3) muscle network definition & generation processes. In the Kinect-to-head process, subject-specific data acquired from a new user in neutral mimic were used for generating his/her geometrical head model with facial texture. In particular, a template head model was deformed to optimally fit with high-definition facial points acquired by the Kinect sensor. Moreover, the facial texture was also merged from his/her facial images in left, right, and center points of view. In the head-to-skull process, a generic skull model was deformed so that its shape was statistically fitted with his/her geometrical head model. In the muscle network definition & generation process, a muscle network was defined from the head and skull models for computing muscle strains during facial movements. Muscle insertion points and muscle attachment points were defined as vertex positions on the head model and the skull model respectively based on the standard facial anatomy. Three healthy subjects and two facial palsy patients were selected for validating the proposed method. In neutral positions, magnetic resonance imaging (MRI)-based head and skull models were compared with Kinect-based head and skull models. In mimic positions, infrared depth-based head models in smiling and [u]-pronouncing mimics were compared with appropriate animated Kinect-driven head models. The Hausdorff distance metric was used for these comparisons. Moreover, computed muscle lengths and strains in the tested facial mimics were validated with reported values in literature.

RESULTS

With the current hardware configuration, the patient-specific head model with skull and muscle network could be fast generated within 17.16±0.37s and animated in real-time with the framerate of 40 fps. In neutral positions, the best mean error was 1.91 mm for the head models and 3.21 mm for the skull models. On facial regions, the best mean errors were 1.53 mm and 2.82 mm for head and skull models respectively. On muscle insertion/attachment point regions, the best mean errors were 1.09 mm and 2.16 mm for head and skull models respectively. In mimic positions, these errors were 2.02 mm in smiling mimics and 2.00 mm in [u]-pronouncing mimics for the head models on facial regions. All above error values were computed on a one-time validation procedure. Facial muscles exhibited muscle shortening and muscle elongating for smiling and pronunciation of sound [u] respectively. Extracted muscle features (i.e. muscle length and strain) are in agreement with experimental and literature data.

CONCLUSIONS

This study proposed a novel modeling method for fast generating and animating patient-specific biomechanical head model with facial texture and muscle activation bio-feedbacks. The Kinect-driven muscle strains could be applied for further real-time muscle-oriented facial paralysis grading and other facial analysis applications.

Real-time computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements using a contactless sensor and system of systems approach

BACKGROUND AND OBJECTIVE

Head and facial mimic animations play important roles in various fields such as human-machine interactions, internet communications, multimedia applications, and facial mimic analysis. Numerous studies have been trying to simulate these animations. However, they hardly achieved all requirements of full rigid head and non-rigid facial mimic animations in a subject-specific manner with real-time framerates. Consequently, this present study aimed to develop a real-time computer vision system for tracking simultaneously rigid head and non-rigid facial mimic movements.

METHODS

Our system was developed using the system of systems approach. A data acquisition sub-system was implemented using a contactless Kinect sensor. A subject-specific model generation sub-system was designed to create the geometrical model from the Kinect sensor without texture information. A subject-specific texture generation sub-system was designed for enhancing the reality of the generated model with texture information. A head animation sub-system with graphical user interfaces was also developed. Model accuracy and system performances were analyzed.

RESULTS

The comparison with MRI-based model shows a very good accuracy level (distance deviation of ~1 mm in neutral position and an error range of [2-3 mm] for different facial mimic positions) for the generated model from our system. Moreover, the system speed can be optimized to reach a high framerate (up to 60 fps) during different head and facial mimic animations.

CONCLUSIONS

This study presents a novel computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements in real time. In perspectives, serious game technology will be integrated into this system towards a full computer-aided decision support system for facial rehabilitation.

Real-time Subject-specific Head and Facial Mimic Animation System using a Contactless Kinect Sensor and System of Systems Approach

Facial palsies due to stroke, accidental and sportive injuries or sometimes without etiology, affect the professional and personal lives of involved patients. These disorders are not only a functional handicap but also a social integration impairment. The recovery of facial mimics with a normal and symmetrical facial expression allows involved patients to improve their living conditions and social identity. Current approaches lack of visual feedbacks. To monitor facial mimics and head movements in a quantitative and objective manners, a computer-aided animation system needs to be developed. Numerous systems have been proposed using single camera, stereo camera, 3-D scanner, and Kinect approaches. In particular, Kinect contactless sensor has been proven to be very suitable for 3-D facial simulation applications. However, little studies have employed the Kinect sensor for real-time head animation applications. Consequently, this study developed a real-time head and facial mimic animation system using the contactless Kinect sensor and the system of systems approach. To evaluate the accuracy of the subject-specific Kinect-based geometrical models, magnetic resonance imaging (MRI) data were used. As results, the mean distance deviation between generated Kinect-based and reconstructed MRI-based geometrical head models are approximately 1 mm for two tested subjects. The generation times are 9.7 s ± 0.3 and 0.046 s ± 0.005 by using the full facial landmarks and MPEG-4 facial landmarks respectively. Real-time head and facial mimic animations were illustrated. Particularly, the system could be executed at a very high framerate (60 fps). Further developments relate to the integration of texture information and internal structures such as a skull and muscle network to develop a full subject specific head and facial mimic animation system for facial mimic rehabilitation.

A Systematic Review of Continuum Modeling of Skeletal Muscles: Current Trends, Limitations, and Recommendations

Finite elasticity theory has been commonly used to model skeletal muscle. A very large range of heterogeneous constitutive laws has been proposed. In this review, the most widely used continuum models of skeletal muscles were synthetized and discussed. Trends and limitations of these laws were highlighted to propose new recommendations for future researches. A systematic review process was performed using two reliable search engines as PubMed and ScienceDirect. 40 representative studies (13 passive muscle materials and 27 active muscle materials) were included into this review. Note that exclusion criteria include tendon models, analytical models, 1D geometrical models, supplement papers, and indexed conference papers. Trends of current skeletal muscle modeling relate to 3D accurate muscle representation, parameter identification in passive muscle modeling, and the integration of coupled biophysical phenomena. Parameter identification for active materials, assumed fiber distribution, data assumption, and model validation are current drawbacks. New recommendations deal with the incorporation of multimodal data derived from medical imaging, the integration of more biophysical phenomena, and model reproducibility. Accounting for data uncertainty in skeletal muscle modeling will be also a challenging issue. This review provides, for the first time, a holistic view of current continuum models of skeletal muscles to identify potential gaps of current models according to the physiology of skeletal muscle. This opens new avenues for improving skeletal muscle modeling in the framework of in silico medicine.

IMAGE-BASED SKELETAL MUSCLE COORDINATION: CASE STUDY ON A SUBJECT SPECIFIC FACIAL MIMIC SIMULATION

Facial muscle coordination is a fundamental mechanism for facial mimics and expressions. The understanding of this complex mechanism leads to better diagnosis and treatment of facial disorders like facial palsy or disfigurement. The objective of this work was to use magnetic resonance imaging (MRI) technique to characterize the activation behavior of facial muscles and then simulate their coordination mechanism using a subject specific finite element model. MRI data of lower head of a healthy subject were acquired in neutral and in the pronunciation of the sound [o] positions. Then, a finite element model was derived directly from acquired MRI images in neutral position. Transversely-isotropic, hyperelastic, quasi-incompressible behavior law was implemented for modeling facial muscles. The simulation to produce the pronunciation of the sound [o] was performed by the cumulative coordination between three pairs of facial mimic muscles (Zygomaticus Major (ZM), Levator Labii Superioris (LLS), Levator Anguli Oris (LAO)). Mean displacement amplitude showed a good agreement with a relative deviation of 15% between numerical outcome and MRI-based measurement when all three muscles are involved. This study elucidates, for the first time, the facial muscle coordination using in vivo data leading to improve the model understanding and simulation outcomes.