Recognizing Emotional Expression as an Outcome Measure After Face Transplant

Defining Standard Values for FaceReader Facial Expression Software Output

BACKGROUND

FaceReader is a validated software package that uses computer vision technology for facial expression recognition which has become increasingly popular in academic research to expedite, scale, and decrease the cost of facial emotion analysis. In this study, we compare FaceReader analysis to human evaluator interpretation in order to define standard values for the software output.

METHODS

Randomly generated facial images produced by generative adversarial networks were analyzed using FaceReader and by survey participants (n=496). The age, facial emotion, and intensity of emotion as determined by the software and survey participants were recorded. Results were analyzed and compared.

RESULTS

80 randomly generated images (20 children, 20 young adult, 20 middle aged, and 20 elderly; 38 male and 42 female) were included. Analysis of correlation between most common expression identified by FaceReader and the primary emotion detected by surveyors showed strong correlation (κ = 0.77, 95% CI = 0.64-0.91). On analyzing this correlation by age group, there was fair correlation in children (κ = 0.40, 95% CI = 0.078-0.72), perfect correlation in young adults (κ = 1.0, 95% CI = 1.0-1.0), strong correlation in middle aged adults (κ = 0.79, 95% CI = 0.53-1) and near perfect in elderly adults(κ = 0.9 , 95% CI = 0.7-1.0).

CONCLUSIONS

We provided the first study defining the expected average values generated by FaceReader in generally smiling images. This can be used as a standard in future studies.

LEVEL OF EVIDENCE IV

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors   www.springer.com/00266 .

Vascularized Composite Allograft Versus Prosthetic for Reconstruction After Facial and Hand Trauma: Comparing Cost, Complications, and Long-term Outcome

ABSTRACT

In the past decade, vascularized composite allotransplantation (VCA) has become clinical reality for reconstruction after face and hand trauma. It offers patients the unique opportunity to regain form and function in a way that had only been achieved with traditional reconstruction or with the use of prostheses. On the other hand, prostheses for facial and hand reconstruction have continued to evolve over the years and, in many cases, represent the primary option for patients after hand and face trauma. We compared the cost, associated complications, and long-term outcomes of VCA with prostheses for reconstruction of the face and hand/upper extremity. Ultimately, VCA and prostheses represent 2 different reconstructive options with distinct benefit profiles and associated limitations and should ideally not be perceived as competing choices. Our work adds a valuable component to the general framework guiding the decision to offer VCA or prostheses for reconstruction after face and upper extremity trauma.

How Brow Rotation Affects Emotional Expression Utilizing Artificial Intelligence

BACKGROUND

It is well known that brow position affects emotional expression. However, there is little literature on how and to what degree this change in emotional expression happens. Previous studies on this topic have utilized manual rating; this method of study remains small and labor intensive. Our objective is to correlate manual brow rotations with emotional outcomes using artificial intelligence to objectively determine how specific brow manipulations affected human expression.

METHODS

We included 53 brow-lift patients in this study. Pre-operative patients' brows were rotated to - 20, - 10, +10, and +20 degrees in respect to the central axis of their existing brow using PIXLR, a cloud-based set of image editing tools and utilities. These images were analyzed using FaceReader, a validated software package that uses computer vision technology for facial expression recognition. The primary facial emotion and intensity of facial action units (0 = no action unit detected to 4 = most intense action unit detected) generated by the software were recorded.

RESULTS

265 total images [5 images (pre-operative, - 20 degree brow rotation, - 10, +10, and +20) per patient] were analyzed using FaceReader. The primary emotion detected in the majority of images was neutral. The percentage of disgust in patients' expressions, as detected by FaceReader, increased with increased positive brow rotation (1.76% disgust detected at - 20 degrees, 2.09% at - 10 degrees, 2.65% at neutral, 2.61% at +10 degrees, and 2.95% at +20 degrees). In contrast, the percentage of sadness in patients' expressions decreased with increased positive brow rotation (29.92% sadness detected at - 20 degrees, 21.5% at - 10 degrees, 11.42% at neutral, 15.75% at +10 degrees, and 12.86% at +20 degrees). Our facial action unit analysis corresponded with primary emotion analysis. The intensity of the inner brow raiser decreased with increased positive brow rotation 8.54% at - 20 degrees, 4.21% at - 10 degrees, 1.48% at neutral, 0.84% at +10 degrees, and 0.76% at +20 degrees). The intensity of the outer brow raiser increased with increased positive brow rotation (0.97% at - 20 degrees, 0.45% at - 10 degrees, 1.12% at neutral, 5.45% at +10 degrees, and 11.19% at +20 degrees).

CONCLUSION

We demonstrated that increasing the degree of brow rotation correlated positively with the percentage of disgust and inversely with the percentage of sadness detected by FaceReader. This study demonstrated how different manipulated brow positions affected emotional outcomes using artificial intelligence. Physicians can use these findings to better understand how brow-lifts can affect the perceived emotion of their patients.

LEVEL OF EVIDENCE III

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

Implications of Vascularized Composite Allotransplantation in Plastic Surgery on Legal Medicine

Background: When a patient receives a transplant—be it classically an organ or bone marrow or, more recently, composite allotransplantations of the limb or face—it can result in artificial chimerism. Such chimerism raises considerations in forensic medicine, a field that relies on the collection and identification of biological samples from crime scenes. Beyond this chimerism, composite allotransplantations create further challenges. Methods: After screening the literature and press releases, we provide a brief history and summary of some of the technologies used in forensic identification, explaining their advantages and pitfalls in the light of transplantation and cautioning against misidentifying those who evade justice by taking advantage of such considerations. Results: With face transplantation, patients can receive the skin, hair, salivary glands, teeth, and oral and nasal mucosa of their donors, components which hold great importance in forensic science. Modern technologies such as computer-assisted facial recognition, although gradually becoming more accurate over time, also face new challenges in this post-transplantation era as facial recognition software can be misled by surgical alterations of the face or face transplantation. With limb transplantation, there is an impact on fingerprint identification. Conclusions: Both surgical transplantation techniques and forensic technologies have seen incomprehensibly great innovation in the past century. Given the growing rate of successful composite transplantation in the USA and worldwide, it is now important for law enforcement agents to be aware of the new possibility of having two sets of genetic material, hair, saliva, fingerprints, or even facial recognition data for the same individual.

Quantification of Facial Allograft Edema During Acute Rejection: A Software-Based 3-Dimensional Analysis

BACKGROUND

Acute rejection (AR) is a common complication in facial transplant (FT) patients associated with allograft edema and erythema. Our study aims to demonstrate the feasibility of using software-based 3-dimensional (3D) facial analysis to quantify edema as it resolves during/after AR treatment in an FT patient.

METHODS

Our patient is a 23-year-old man who underwent a face and bilateral hand allotransplant in August 2020. The Vectra H1 (Canfield, Fairfield, NJ) portable scanner was used to capture 3D facial images at 8 time points between postoperative day (POD) 392 and 539. The images were analyzed with the Vectra Software using a rejection-free image (POD 539) as a control.

RESULTS

Edema increased in the periorbital, lower third, and submandibular regions before AR treatment (POD 392-415). At POD 448, total facial edema was reduced to near baseline values in response to plasmapheresis and thymoglobulin (+156.94 to +28.2 mL). The fastest and most notable response to treatment was seen in the periorbital region, while some edema remained in the submandibular (+19.79 mL) and right lower third (+8.65 mL) regions. On POD 465, after the initial improvement, the edema increased but was resolved with steroid use. Facial edema did not correlate with the histopathological evaluation in our patient.

CONCLUSIONS

We demonstrated the feasibility of analyzing 3D facial images to quantify edema during/after AR treatment in an FT patient. Our analysis detected edema changes consistent with AR followed by an improvement after treatment. This technology shows promise for noninvasive monitoring of FT patients.

Software-based Detection of Acute Rejection Changes in Face Transplant

BACKGROUND

 An objective, non-invasive method for redness detection during acute allograft rejection in face transplantation (FT) is lacking.

METHODS

 A retrospective cohort study was performed with 688 images of 7 patients with face transplant (range, 1 to 108 months post-transplant). Healthy controls were matched to donor age, sex, and had no prior facial procedures. Rejection state was confirmed via tissue biopsy. An image-analysis software developed alongside VicarVision (Amsterdam, Netherlands) was used to produce R, a measure of differences between detectable color and absolute red. R is inversely proportional to redness, where lower R values correspond to increased redness. Linear mixed models were used to study fixed effect of rejection state on R values. Estimated marginal means of fitted models were calculated for pairwise comparisons.

RESULTS

 Of 688 images, 175, 170, 202, and 141 images were attributable to Banff Grade 0,1,2, and 3, respectively. Estimated change in R value of facial allografts decreased with increasing Banff Grade (p = 0.0001). The mean R value of clinical rejection (Banff Grade ⅔) (16.67, 95% Confidence Interval [CI] 14.79-18.58) was lower (p = 0.005) than non-rejection (Banff Grade 0/1) (19.38, 95%CI 17.43-21.33). Both clinical and non-rejection mean R values were lower (p = 0.0001) than healthy controls (24.12, 95%CI 20.96-27.28).

CONCLUSION

 This proof-of-concept study demonstrates that software-based analysis can detect and monitor acute rejection changes in FT. Future studies should expand on this tool's potential application in telehealth and as a screening tool for allograft rejection.

Long-term Outcomes After Facial Allotransplantation: Systematic Review of the Literature

BACKGROUND

Facial vascularized composite allotransplantation (fVCA) represents a reconstructive approach that enables superior improvements in functional and esthetic restoration compared with conventional craniomaxillofacial reconstruction. Outcome reports of fVCA are usually limited to short-term follow-up or single-center experiences. We merge scientific literature on reported long-term outcome data to better define the risks and benefits of fVCA.

METHODS

We conducted a systematic review of PubMed/MEDLINE databases in accordance with PRISMA guidelines. English full-text articles providing data on at least 1 unique fVCA patient, with ≥3 years follow-up, were included.

RESULTS

The search yielded 1812 articles, of which 28 were ultimately included. We retrieved data on 23 fVCA patients with mean follow-up of 5.3 years. More than half of the patients showed improved quality of life, eating, speech, and motor and sensory function following fVCA. On average, the patients had 1 acute cell-mediated rejection and infectious episode per year. The incidence rates of acute rejection and infectious complications were high within first-year posttransplant but declined thereafter. Sixty-five percent of the patients developed at least 1 neoplastic or metabolic complication after transplantation. Chronic vascular rejection was confirmed in 2 patients, leading to allograft loss after 8 and 9 years. Two patient deaths occurred 3.5 and 10.5 years after transplant due to suicide and lung cancer, respectively.

CONCLUSIONS

Allograft functionality and improvements in quality of life suggest a positive risk-benefit ratio for fVCA. Recurrent acute rejection episodes, chronic rejection, immunosuppression-related complications, and heterogeneity in outcome reporting present ongoing challenges in this field.

Changes in Computer-Analyzed Facial Expressions with Age

Facial expressions are well known to change with age, but the quantitative properties of facial aging remain unclear. In the present study, we investigated the differences in the intensity of facial expressions between older (n = 56) and younger adults (n = 113). In laboratory experiments, the posed facial expressions of the participants were obtained based on six basic emotions and neutral facial expression stimuli, and the intensities of their faces were analyzed using a computer vision tool, OpenFace software. Our results showed that the older adults expressed strong expressions for some negative emotions and neutral faces. Furthermore, when making facial expressions, older adults used more face muscles than younger adults across the emotions. These results may help to understand the characteristics of facial expressions in aging and can provide empirical evidence for other fields regarding facial recognition.

Artificial Intelligence-Driven Video Analysis for Novel Outcome Measures After Smile Reanimation Surgery

Background: Since facial paralysis is a dynamic condition, the analysis of still photographs is not sufficient for measurement of facial reanimation outcomes. This study aimed at evaluating an artificial intelligence (AI)-driven software as a novel video assessment tool for smile reanimation surgery and at comparing it with the Terzis score. Methods: Patients with facial paralysis undergoing smile reanimation surgery between January 2008 and April 2020 were eligible for this retrospective study. Inclusion criteria were at least 6 months of follow-up and availability of both pre- and post-operative video documentation. The software output was given as intensity score (IS) values between 0 and 1, representing emotions/action units (AUs) that are absent or fully present, respectively. Results: During the study period, 240 patients underwent facial reanimation surgery, of whom 63 patients met the inclusion criteria. Postoperatively, the median IS of the happiness emotion and lip corner puller AU increased significantly (p < 0.001). There was a positive correlation of Terzis score with the IS of happiness emotion (r = 0.8) and lip corner puller AU (r = 0.74). Conclusions: The novel AI-driven video analysis is strongly correlated with the Terzis score and shows promise for objective functional outcome evaluation after smile reanimation surgery.

Limitations and limits and of vascularized composite allotransplantations: can we reach the holy grail?

PURPOSE OF REVIEW

In recent times, vascularized composite allotransplantation (VCA) have been gaining more attention and applications. Currently, VCA are at the highest level of the reconstruction pyramid, and thus the effects expected after them are intended to outweigh what the 'classical' reconstructive surgery can offer us, including even the most advanced microsurgical techniques.

RECENT FINDINGS

Over 40 patients have received a partial or full-face transplant. Others have received penis, uterus, larynx, abdominal wall, and lower extremity transplants. Each type of VCA has its own problems and limitations. However, resolving the limits defined by immunosuppression and improved donor selection would revolutionize all of them.

SUMMARY

Defining the limits and limitations of given procedures will not only allow for better preparation of transplant teams but will also help in determining the direction of future research.

Accelerated chronic skin changes without allograft vasculopathy: A 10-year outcome report after face transplantation

BACKGROUND

Long-term outcomes after face transplantation are rarely reported in the scientific literature. Here we present outcome data of a partial face allograft recipient 10 years after transplantation.

METHODS

Medical records were reviewed for functional and psychosocial outcomes as well as complications. Histopathologic analyses of autopsy tissues and characterization of skin immune cells were performed.

RESULTS

The patient retained long-term motor and sensory function, though with a noticeable drop in sensory function after year 5. Social reintegration of the patient was marked by reconnection with his family and participation in public social activities. Immunosuppressive therapy consisted of tacrolimus (target levels 6-8 ng/mL after the first year), mycophenolate, and prednisone, while steroids were completely weaned between years 1 and 7. One acute cellular rejection episode of grade II or higher occurred on average per year and led to chronic skin changes (papillary dermal sclerosis with superficial hyalinization, epidermal thinning with loss of rete ridges, perieccrine fibrosis), but the allograft vessels, muscles, adipose tissue, and bone were spared. Allograft skin was characterized by increased number of CD4+ TNF-α/IL17A producing T-cells as compared with native skin. Long-term kidney function was maintained at 60 mL/min estimated glomerular filtration rate. Unfortunately, the preexisting hepatitis C virus infection with liver cirrhosis was resistant to 3 treatments with new direct-acting antivirals and eventually hepatocellular carcinoma developed, causing the patient's death 10 years after transplantation.

CONCLUSION

This report suggests that face transplants can maintain their function for at least 10 years. Chronic skin changes can occur independently of allograft vasculopathy.