Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent

Alice Delbosc (1, 2, 3) , Magalie Ochs (2, 3, 1) , Nicolas Sabouret (4, 5) , Brian Ravenet (4, 5) , Stéphane Ayache (3, 2, 6)

1 TALEP - Traitement Automatique du Langage Ecrit et Parlé
2 LIS - Laboratoire d'Informatique et des Systèmes (LIS) (Marseille, Toulon)
3 AMU - Aix Marseille Université
4 Université Paris-Saclay
5 LISN - Laboratoire Interdisciplinaire des Sciences du Numérique
6 QARMA - éQuipe d'AppRentissage de MArseille

This paper introduces a new model to generate rhythmically relevant non-verbal facial behaviors for virtual agents while they speak. The model demonstrates perceived performance comparable to behaviors directly extracted from the data and replayed on a virtual agent, in terms of synchronization with speech and believability. Interestingly, we found that training the model with two different sets of data, instead of one, did not necessarily improve its performance. The expressiveness of the people in the dataset and the shooting conditions are key elements. We also show that employing an adversarial model, in which fabricated fake examples are introduced during the training phase, increases the perception of synchronization with speech. A collection of videos demonstrating the results and code can be accessed at: https://github.com/aldelb/non_verbal_facial_animation.

Keywords: Non-verbal behavior; behavior generation; embodied conversational agent; neural networks; adversarial learning; encoder-decoder

Domains: Machine Learning [cs.LG]; Artificial Intelligence [cs.AI]; Neural and Evolutionary Computing [cs.NE]

Alice Delbosc, Magalie Ochs, Nicolas Sabouret, Brian Ravenet, Stéphane Ayache. Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent. GENEA : GENERATION AND EVALUATION OF NON-VERBAL BEHAVIOUR FOR EMBODIED AGENTS during ICMI ’23 Companion, Oct 2023, Paris, France. ⟨hal-04206768⟩