Using Artificial Intelligence to Measure Facial Expression following Facial Reanimation Surgery

Cross-Facial Nerve Grafting Used Independently in Facial Reanimation: A Narrative Review

Importance: Cross-Facial Nerve Grafting (CFNG) for facial palsy offers potential to restore spontaneous facial expression, but specific indications and associated outcomes are limited. Updates to this technique have aided in its successful employment in select cases. This review aims to explore the context in which CFNG has been successfully utilized as a primary modality. Observations: Literature review was performed auditing all studies investigating CFNG as a primary modality, which reported outcomes. A total of 326 cases reporting outcomes for primary CFNG were included. Eye closure outcomes were 83.3% successful at ages 0-18, 77.3% successful at ages 19-40, and 57.1% successful at ages 41+. Smile outcomes were 73.7% successful at ages 0-18, 81.5% successful at ages 19-40, and 52.8% successful at ages 41+. For synkinesis, 89% of cases were considered successful; 100% successful at ages 0-18, and 78.4% successful in adults. Conclusions and Relevance: CFNG may offer return of spontaneous facial function in select cases. Higher percentages of successful outcomes are observed in younger patients, when performed in two stages, and when performed earlier from the onset of FP in cases of eye closure restoration. In the modern era, CFNG has been more commonly employed as an adjunctive procedure to other reanimation techniques.

Functional Outcome of Dual Reinnervation with Cross-Facial Nerve Graft and Masseteric Nerve Transfer for Facial Paralysis

BACKGROUND

The combination of cross-facial nerve graft (CFNG) and masseteric nerve transfer (MNT) for reinnervation of facial paralysis may provide advantages of both neural sources. However, quantitative functional outcome reports with a larger number of patients are lacking in the literature. Here, the authors describe their 8-year experience with this surgical technique.

METHODS

Twenty patients who presented with complete facial paralysis (duration, <12 months) received dual reinnervation with CFNG and MNT. The functional outcome of the procedure was evaluated with the physician-graded outcome metric eFACE scale. The objective artificial intelligence-driven software Emotrics and FaceReader were used for oral commissure measurements and emotional expression assessment, respectively.

RESULTS

The mean follow-up was 31.75 ± 23.32 months. In the eFACE score, the nasolabial fold depth and oral commissure at rest improved significantly ( P < 0.05) toward a more balanced state after surgery. Postoperatively, there was a significant decrease in oral commissure asymmetry while smiling (from 19.22 ± 6.1 mm to 12.19 ± 7.52 mm). For emotional expression, the median intensity score of happiness, as measured by the FaceReader software, increased significantly while smiling (0.28; interquartile range, 0.13 to 0.64). In five patients (25%), a secondary static midface suspension with fascia lata strip had to be performed because of unsatisfactory resting symmetry. Older patients and patients with greater preoperative resting asymmetry were more likely to receive static midface suspension.

CONCLUSION

The authors' results suggest that the combination of MNT and CFNG for reinnervation of facial paralysis provides good voluntary motion and may lessen the use of static midface suspension in the majority of patients.

CLINICAL QUESTION/LEVEL OF EVIDENCE

Therapeutic, IV.

Defining Standard Values for FaceReader Facial Expression Software Output

BACKGROUND

FaceReader is a validated software package that uses computer vision technology for facial expression recognition which has become increasingly popular in academic research to expedite, scale, and decrease the cost of facial emotion analysis. In this study, we compare FaceReader analysis to human evaluator interpretation in order to define standard values for the software output.

METHODS

Randomly generated facial images produced by generative adversarial networks were analyzed using FaceReader and by survey participants (n=496). The age, facial emotion, and intensity of emotion as determined by the software and survey participants were recorded. Results were analyzed and compared.

RESULTS

80 randomly generated images (20 children, 20 young adult, 20 middle aged, and 20 elderly; 38 male and 42 female) were included. Analysis of correlation between most common expression identified by FaceReader and the primary emotion detected by surveyors showed strong correlation (κ = 0.77, 95% CI = 0.64-0.91). On analyzing this correlation by age group, there was fair correlation in children (κ = 0.40, 95% CI = 0.078-0.72), perfect correlation in young adults (κ = 1.0, 95% CI = 1.0-1.0), strong correlation in middle aged adults (κ = 0.79, 95% CI = 0.53-1) and near perfect in elderly adults(κ = 0.9 , 95% CI = 0.7-1.0).

CONCLUSIONS

We provided the first study defining the expected average values generated by FaceReader in generally smiling images. This can be used as a standard in future studies.

LEVEL OF EVIDENCE IV

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors   www.springer.com/00266 .

Artificial Intelligence in Facial Plastic and Reconstructive Surgery: A Systematic Review

Artificial intelligence (AI) is a technology that is evolving rapidly and is changing the world and medicine as we know it. After reviewing the PROSPERO database of systematic reviews, there is no article related to this topic in facial plastic and reconstructive surgery. The objective of this article was to review the literature regarding AI applications in facial plastic and reconstructive surgery.A systematic review of the literature about AI in facial plastic and reconstructive surgery using the following keywords: Artificial Intelligence, robotics, plastic surgery procedures, and surgery plastic and the following databases: PubMed, SCOPUS, Embase, BVS, and LILACS. The inclusion criteria were articles about AI in facial plastic and reconstructive surgery. Articles written in a language other than English and Spanish were excluded. In total, 17 articles about AI in facial plastic met the inclusion criteria; after eliminating the duplicated papers and applying the exclusion criteria, these articles were reviewed thoroughly. The leading type of AI used in these articles was computer vision, explicitly using models of convolutional neural networks to objectively compare the preoperative with the postoperative state in multiple interventions such as facial lifting and facial transgender surgery.In conclusion, AI is a rapidly evolving technology, and it could significantly impact the treatment of patients in facial plastic and reconstructive surgery. Legislation and regulations are developing slower than this technology. It is imperative to learn about this topic as soon as possible and that all stakeholders proactively promote discussions about ethical and regulatory dilemmas.

Artificial Intelligence: Singularity Approaches

SUMMARY

Artificial intelligence (AI) has been a disruptive technology within health care, from the development of simple care algorithms to complex deep-learning models. AI has the potential to reduce the burden of administrative tasks, advance clinical decision-making, and improve patient outcomes. Unlocking the full potential of AI requires the analysis of vast quantities of clinical information. Although AI holds tremendous promise, widespread adoption within plastic surgery remains limited. Understanding the basics is essential for plastic surgeons to evaluate the potential uses of AI. This review provides an introduction of AI, including the history of AI, key concepts, applications of AI in plastic surgery, and future implications.

How Brow Rotation Affects Emotional Expression Utilizing Artificial Intelligence

BACKGROUND

It is well known that brow position affects emotional expression. However, there is little literature on how and to what degree this change in emotional expression happens. Previous studies on this topic have utilized manual rating; this method of study remains small and labor intensive. Our objective is to correlate manual brow rotations with emotional outcomes using artificial intelligence to objectively determine how specific brow manipulations affected human expression.

METHODS

We included 53 brow-lift patients in this study. Pre-operative patients' brows were rotated to - 20, - 10, +10, and +20 degrees in respect to the central axis of their existing brow using PIXLR, a cloud-based set of image editing tools and utilities. These images were analyzed using FaceReader, a validated software package that uses computer vision technology for facial expression recognition. The primary facial emotion and intensity of facial action units (0 = no action unit detected to 4 = most intense action unit detected) generated by the software were recorded.

RESULTS

265 total images [5 images (pre-operative, - 20 degree brow rotation, - 10, +10, and +20) per patient] were analyzed using FaceReader. The primary emotion detected in the majority of images was neutral. The percentage of disgust in patients' expressions, as detected by FaceReader, increased with increased positive brow rotation (1.76% disgust detected at - 20 degrees, 2.09% at - 10 degrees, 2.65% at neutral, 2.61% at +10 degrees, and 2.95% at +20 degrees). In contrast, the percentage of sadness in patients' expressions decreased with increased positive brow rotation (29.92% sadness detected at - 20 degrees, 21.5% at - 10 degrees, 11.42% at neutral, 15.75% at +10 degrees, and 12.86% at +20 degrees). Our facial action unit analysis corresponded with primary emotion analysis. The intensity of the inner brow raiser decreased with increased positive brow rotation 8.54% at - 20 degrees, 4.21% at - 10 degrees, 1.48% at neutral, 0.84% at +10 degrees, and 0.76% at +20 degrees). The intensity of the outer brow raiser increased with increased positive brow rotation (0.97% at - 20 degrees, 0.45% at - 10 degrees, 1.12% at neutral, 5.45% at +10 degrees, and 11.19% at +20 degrees).

CONCLUSION

We demonstrated that increasing the degree of brow rotation correlated positively with the percentage of disgust and inversely with the percentage of sadness detected by FaceReader. This study demonstrated how different manipulated brow positions affected emotional outcomes using artificial intelligence. Physicians can use these findings to better understand how brow-lifts can affect the perceived emotion of their patients.

LEVEL OF EVIDENCE III

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

A Narrative Review of Artificial Intelligence (AI) for Objective Assessment of Aesthetic Endpoints in Plastic Surgery

Notoriously characterized by subjectivity and lack of solid scientific validation, reporting aesthetic outcome in plastic surgery is usually based on ill-defined end points and subjective measures very often from the patients' and/or providers' perspective. With the tremendous increase in demand for all types of aesthetic procedures, there is an urgent need for better understanding of aesthetics and beauty in addition to reliable and objective outcome measures to quantitate what is perceived as beautiful and attractive. In an era of evidence-based medicine, recognition of the importance of science with evidence-based approach to aesthetic surgery is long overdue. View the many limitations of conventional outcome evaluation tools of aesthetic interventions, objective outcome analysis provided by tools described to be reliable is being investigated such as advanced artificial intelligence (AI). The current review is intended to analyze available evidence regarding advantages as well as limitations of this technology in objectively documenting outcome of aesthetic interventions. It has shown that some AI applications such as facial emotions recognition systems are capable of objectively measuring and quantitating patients' reported outcomes and defining aesthetic interventions success from the patients' perspective. Though not reported yet, observers' satisfaction with the results and their appreciation of aesthetic attributes may also be measured in the same manner.Level of Evidence III This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

Applying artificial intelligence algorithm in the design of a guide plate for mandibular angle ostectomy

PURPOSE

Surgical guide plates can improve the accuracy of surgery, although their design process is complex and time-consuming. This study aimed to use artificial intelligence (AI) to design standardized mandibular angle ostectomy guide plates and reduce clinician workload.

METHODS

An intelligence algorithm was designed and trained to design guide plates, with a safety-ensuring penalty factor added. A single-center retrospective cohort study was conducted to test the algorithm among patients who had visited our hospital between 2020 and 2021 for mandibular angle ostectomy. We included patients diagnosed with mandibular angle hypertrophy and excluded those combined with other facial malformations. The guide plate design method acted as the primary predictor, which was AI algorithm vs. experienced residents. Moreover, the symmetry of plate-guided ostectomy was chosen as the primary outcome. The safety, shape, location, effectiveness, and design duration of the guide plate were also recorded. The independent samples t-test and Pearson's chi-squared test were used and P-values < 0.05 were considered significant.

RESULTS

Fifty patients (7 men, 43 women; 27 ± 4 years) were included. The two groups differed significantly in terms of safety (7.02 vs. 5.25, P < 0.05) and design duration (24.98 vs. 1685.08, P ＜ 0.05). The ostectomy symmetry and shape, location, and effectiveness of the guide plates did not differ significantly between the two groups.

CONCLUSIONS

The intelligent algorithm can improve safety and save time for guide plate design, ensuring other quality of the guide plates. It has good potential applicability in accurate mandibular angle ostectomy.

Assessing facial weakness in myasthenia gravis with facial recognition software and deep learning

OBJECTIVE

Myasthenia gravis (MG) is an autoimmune disease leading to fatigable muscle weakness. Extra-ocular and bulbar muscles are most commonly affected. We aimed to investigate whether facial weakness can be quantified automatically and used for diagnosis and disease monitoring.

METHODS

In this cross-sectional study, we analyzed video recordings of 70 MG patients and 69 healthy controls (HC) with two different methods. Facial weakness was first quantified with facial expression recognition software. Subsequently, a deep learning (DL) computer model was trained for the classification of diagnosis and disease severity using multiple cross-validations on videos of 50 patients and 50 controls. Results were validated using unseen videos of 20 MG patients and 19 HC.

RESULTS

Expression of anger (p = 0.026), fear (p = 0.003), and happiness (p < 0.001) was significantly decreased in MG compared to HC. Specific patterns of decreased facial movement were detectable in each emotion. Results of the DL model for diagnosis were as follows: area under the curve (AUC) of the receiver operator curve 0.75 (95% CI 0.65-0.85), sensitivity 0.76, specificity 0.76, and accuracy 76%. For disease severity: AUC 0.75 (95% CI 0.60-0.90), sensitivity 0.93, specificity 0.63, and accuracy 80%. Results of validation, diagnosis: AUC 0.82 (95% CI: 0.67-0.97), sensitivity 1.0, specificity 0.74, and accuracy 87%. For disease severity: AUC 0.88 (95% CI: 0.67-1.0), sensitivity 1.0, specificity 0.86, and accuracy 94%.

INTERPRETATION

Patterns of facial weakness can be detected with facial recognition software. Second, this study delivers a 'proof of concept' for a DL model that can distinguish MG from HC and classifies disease severity.

Artificial intelligence-based numerical analysis of the quality of facial reanimation: A comparative retrospective cohort study between one-stage dual innervation and single innervation

This study aimed to investigate the difference in facial reanimation surgery using functional gracilis muscle transfer between the masseteric nerve alone and its combined use with cross face nerve graft (CFNG), which has not been explored before. A novel analysis method based on artificial intelligence (AI) was employed to compare the outcomes of the two approaches. Using AI, 3-dimensional facial landmarks were extracted from 2-dimensional photographs, and distance and angular symmetry scores were calculated. The patients were divided into two groups, with Group 1 undergoing one-stage CFNG and masseteric nerve dual innervation, and Group 2 receiving only masseteric nerve. The symmetry scores were obtained before and 1 year after surgery to assess the degree of change. Of the 35 patients, Group 1 included 13 patients, and Group 2 included 22 patients. The analysis revealed that, in the resting state, the change in the symmetry score of the mouth corner showed distance symmetry (2.55 ± 2.94, 0.52 ± 2.75 for Group 1 and Group 2, respectively, p = 0.048) and angle symmetry (1.21 ± 1.43, 0.02 ± 0.22 for Group 1 and Group 2, respectively, p = 0.001), which were significantly improved in Group 1, indicating a more symmetric pattern after surgery. In the smile state, only the angle symmetry was improved more symmetrically in Group 1 (3.20 ± 2.38, 1.49 ± 2.22 for Group 1 and Group 2, respectively, p = 0.041). Within the limitations of the study it seems that this new analysis method enabled a more accurate numerical symmetry score to be obtained, and while the degree of mouth corner excursion was sufficient with only the masseteric nerve, accompanying CFNG led to further improvement in symmetry in the resting state.

Artificial Intelligence in Ophthalmology - Status Quo and Future Perspectives

Artificial intelligence (AI) is an emerging technology in healthcare and holds the potential to disrupt many arms in medical care. In particular, disciplines using medical imaging modalities, including e.g. radiology but ophthalmology as well, are already confronted with a wide variety of AI implications. In ophthalmologic research, AI has demonstrated promising results limited to specific diseases and imaging tools, respectively. Yet, implementation of AI in clinical routine is not widely spread due to availability, heterogeneity in imaging techniques and AI methods. In order to describe the status quo, this narrational review provides a brief introduction to AI ("what the ophthalmologist needs to know"), followed by an overview of different AI-based applications in ophthalmology and a discussion on future challenges.Abbreviations: Age-related macular degeneration, AMD; Artificial intelligence, AI; Anterior segment OCT, AS-OCT; Coronary artery calcium score, CACS; Convolutional neural network, CNN; Deep convolutional neural network, DCNN; Diabetic retinopathy, DR; Machine learning, ML; Optical coherence tomography, OCT; Retinopathy of prematurity, ROP; Support vector machine, SVM; Thyroid-associated ophthalmopathy, TAO.

Artificial Intelligence in Surgical Evaluation: A Study of Facial Rejuvenation Techniques

Background

Aesthetic facial surgeries historically rely on subjective analysis in determining success; this limits objective comparison of surgical outcomes.

Objectives

This case study exemplifies the use of an artificial intelligence software on objectively analyzing facial rejuvenation techniques with the aim of reducing subjective bias.

Methods

Retrospectively, all patients who underwent facial rejuvenation surgery with concomitant procedures from 2015 to 2017 were included (n = 32). Patients were categorized into Groups A to C: Group A-10 superficial musculoaponeurotic system (SMAS) plication facelift (n = 10), Group B-SMASectomy facelift (n = 7), and Group C-high SMAS facelift (n = 15). Neutral repose images preoperatively and postoperatively (average >3 months) were analyzed using artificial intelligence for emotion and action unit alterations.

Results

Postoperatively, Group A experienced a decrease in happiness by 0.84% and a decrease in anger by 6.87% (P >> .1). Group B had an increase in happiness by 0.77% and an increase in anger by 1.91% (P >> .1). Both Group A and Group B did not show any discernable action unit patterns. In Group C, the lip corner puller AU increased in average intensity from 0% to 18.7%. This correlated with an average increase in detected happiness from 1.03% to 13.17% (P = .008). Conversely, the average detected anger decreased from 14.66% to 0.63% (P = .032).

Conclusions

This study provides the first proof of concept for the use of a machine learning software application to objectively assess various aesthetic surgical outcomes in facial rejuvenation. Due to limitations in patient heterogeneity, this study does not claim one technique's superiority but serves as a conceptual foundation for future investigation.

Level of Evidence 4

Facial Recognition Software Use on Surgically Altered Faces: A Systematic Review

OBJECTIVE

Facial recognition software (FRS) is becoming pervasive in society for commercial use, security systems, and entertainment. Alteration of the facial appearance with surgery poses a challenge to these algorithms, but several methods are being studied to overcome this issue. This study systematically reviews methods used in facial recognition of surgically altered faces.

MATERIALS AND METHODS

A systematic review was performed by searching PubMed and Institute of Electrical and Electronics Engineers (IEEE) databases to identify studies addressing FRS and surgery. On initial review, 178 manuscripts were identified relating to FRS and surgery and allowed division into multiple subgroups. The decision was made to focus on the recognition of surgically altered faces.

RESULTS

Eligible studies included those reports in English on FRS of surgically altered faces, and 39 papers were included. Surgical procedures range from affecting skin surface, such as skin peeling, to altering facial features, such as rhinoplasty, mentoplasty, malar augmentation, brow lift, facelift, orthognathic surgery, facial reanimation, and facial feminization. Methods were classified into appearance-based, feature-based, and texture-based. Descriptive versus experimental protocols were characterized by different reporting outcomes and controls. Accuracy ranged from 19.1% to 85.35% using various analysis methods.

CONCLUSIONS

Knowledge of available limitations and advantages can aid in counseling patients regarding personal technology use, security, and quell fears about surgery to evade authorities. Surgical knowledge can be utilized to improve FRS algorithms for postsurgical recognition.

Review on Facial-Recognition-Based Applications in Disease Diagnosis

Diseases not only manifest as internal structural and functional abnormalities, but also have facial characteristics and appearance deformities. Specific facial phenotypes are potential diagnostic markers, especially for endocrine and metabolic syndromes, genetic disorders, facial neuromuscular diseases, etc. The technology of facial recognition (FR) has been developed for more than a half century, but research in automated identification applied in clinical medicine has exploded only in the last decade. Artificial-intelligence-based FR has been found to have superior performance in diagnosis of diseases. This interdisciplinary field is promising for the optimization of the screening and diagnosis process and assisting in clinical evaluation and decision-making. However, only a few instances have been translated to practical use, and there is need of an overview for integration and future perspectives. This review mainly focuses on the leading edge of technology and applications in varieties of disease, and discusses implications for further exploration.

Toward a Universal Measure of Facial Difference Using Two Novel Machine Learning Models

A sensitive, objective, and universally accepted method of measuring facial deformity does not currently exist. Two distinct machine learning methods are described here that produce numerical scores reflecting the level of deformity of a wide variety of facial conditions.

METHODS

The first proposed technique utilizes an object detector based on a cascade function of Haar features. The model was trained using a dataset of 200,000 normal faces, as well as a collection of images devoid of faces. With the model trained to detect normal faces, the face detector confidence score was shown to function as a reliable gauge of facial abnormality. The second technique developed is based on a deep learning architecture of a convolutional autoencoder trained with the same rich dataset of normal faces. Because the convolutional autoencoder regenerates images disposed toward their training dataset (ie, normal faces), we utilized its reconstruction error as an indicator of facial abnormality. Scores generated by both methods were compared with human ratings obtained using a survey of 80 subjects evaluating 60 images depicting a range of facial deformities [rating from 1 (abnormal) to 7 (normal)].

RESULTS

The machine scores were highly correlated to the average human score, with overall Pearson's correlation coefficient exceeding 0.96 (P < 0.00001). Both methods were computationally efficient, reporting results within 3 seconds.

CONCLUSIONS

These models show promise for adaptation into a clinically accessible handheld tool. It is anticipated that ongoing development of this technology will facilitate multicenter collaboration and comparison of outcomes between conditions, techniques, operators, and institutions.

Seminal Studies in Facial Reanimation Surgery: Consensus and Controversies in the Top 50 Most Cited Articles

ABSTRACT

Facial paralysis can impair one's ability to form facial expressions that are congruent with internal emotion. This hinders communication and the cognitive processing of emotional experience. Facial reanimation surgery, which aims to restore full facial expressivity is a relatively recent undertaking which is still evolving. Due in large part to published techniques, refinements, and clinical outcomes in the scientific literature, consensus on best practice is gradually emerging, whereas controversies still exist.Taking stock of how the discipline reached its current state can help delineate areas of agreement and debate, and more clearly reveal a path forward. To do this, the authors have analyzed the 50 seminal publications pertaining to facial reanimation surgery. In longstanding cases, the free gracilis transfer emerges as a clear muscle of choice but the nerve selection remains controversial with prevailing philosophies advocating cross facial nerve grafts (with or without the support of an ipsilateral motor donor) or an ipsilateral motor donor only, of which the hypoglossal and nerve to masseter predominate. The alternative orthodoxy has refined the approach popularized by Gillies in 1934 and does not require the deployment of microsurgical principles. Although this citation analysis does not tell the whole story, surgeons with an interest in facial reanimation will find that this is a good place to start.