Voicing Your Emotion: Integrating Emotion and Identity in Cross-Modal 3D Facial Animations

Speech + Identity Prompt + Emotion expressive 3D talking face animation

Abstract

Recent advances in speech-driven 3D facial animation have yielded promising results, yet capturing intricate expressiveness, particularly in emotion and identity, remains an intricate challenge. While conventional studies often prioritize lip synchronization, they frequently overlook the full spectrum of emotional nuances and the uniqueness of individual identity.

Addressing this deficiency, we unveil a pioneering method tailored to produce 3D facial expressions that resonate deeply with emotion and identity, guided by speech and user-provided prompt words.

Our key insight is an emotion-identity fusion mechanism, a pre-trained self-reconstruction codebook, meticulously crafted from a wide array of emotional facial movements, serving as an expressive motion benchmark. Using this foundation, prompt words are seamlessly transformed into facial representations that capture both emotion and identity, and are then adeptly projected onto 3D templates.

When synergized with speech audio and a user-specified emotion, our algorithm breathes life into a 3D avatar, reflecting the desired emotion and identity. The model's efficacy is bolstered by an advanced autoregression technique, marrying emotion and identity through an innovative feature fusion mechanism and a purpose-built loss function. Consequently, our approach emerges as a robust tool for crafting 3D talking avatars, rich in emotional depth and distinctive identity.

Method Overview

The figure shows the pipeline: speech is encoded by Wav2Vec2.0, prompt text is converted to a personalized template via Stable Diffusion + Img2FLAME, emotion and identity are fused in temporal features, and the autoregressive generator predicts final 3D facial motion.

Method framework — Framework of our method

Core Modules

Text2Avatar: Generates a customizable 3D face template from textual prompts via Stable Diffusion, EMOCA, and FLAME topology.
Emotion-Identity Fusion: Integrates emotion and identity features into sequential facial motion representations using linear self-attention and stylization blocks.
Avatar Generator: Extracts speech features with Wav2Vec 2.0, employs dual Transformer decoders, and leverages pre-trained VQ-VAE for quantized motion priors to autoregressively synthesize realistic and temporally coherent 3D facial animations.
Key Points Loss: Applied during training to focus on highly active facial vertices, enhancing dynamic expression and improving lip synchronization accuracy.

Core Experimental Results

The method proposed in this paper outperforms existing approaches in three aspects: lip-sync accuracy, emotional expressiveness, and identity customization. It effectively solves the problems of stiff expressions, uncontrollable identities, and insufficient dynamics in traditional 3D facial animation.

Quantitative metrics: Ours (LME 1.2070 / AVE 1.2528), compared with FaceFormer (1.2071 / 1.2857) and CodeTalker (1.3039 / 1.2637).

User study reports strong preference for our method: +84.6% and +52.4% on facial dynamism against FaceFormer and CodeTalker.

Cross-Modal generation of lip-synced 3D talking avatars with different expressions using prompt-guided identity. Demonstrate the capability to generate realistic avatars that accurately reflect specified identities and emotions.

The t-SNE diagram of the same sentence under different emotion types. Demonstrate the Emotion-Identity Fusion module enables clear differentiation of facial features across different emotions, making expressions more distinguishable.

Visualization of lip distance in conversation sequences of different models. Proves larger and more natural lip movements while maintaining accurate lip synchronization, effectively avoiding over-smoothing.

Ablation experiment on the quality of emotion-identity fusion module. Ours enhance muscle movements in areas such as the periocular region and cheeks, resulting in richer emotional cues.

BibTeX

@inproceedings{song2024voicing,
      title={Voicing Your Emotion: Integrating Emotion and Identity in Cross-Modal 3D Facial Animations},
      author={Song, Wenfeng and Lv, Zhenyu and Wang, Xuan and Hou, Xia},
      booktitle={2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
      pages={717--718},
      year={2024},
      organization={IEEE}
    }