Skip to Main content Skip to Navigation
New interface
Conference papers

Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures

Abstract : Communicative gestures and speech prosody are tightly linked. Our aim is to predict when gestures are performed based on prosody. We develop a model based on a seq2seq recurrent neural network with attention mechanism. The model is trained on a corpus of natural dyadic interaction where the speech prosody and the gestures have been annotated. Because the output of the model is a sequence, we use a sequence comparison technique to evaluate the model performance. We find that the model can predict certain gesture classes. In our experiment, we also replace some input features with random values to find which prosody features are pertinent. We find that the F0 is pertinent. Lastly, we also train the model on one speaker and test it with the other speaker to find whether the model is generalisable. We find that the models which we train on one speaker also works for another speaker of the same conversation.
Complete list of metadata

Cited literature [26 references]  Display  Hide  Download
Contributor : CCSD Connect in order to contact the contributor
Submitted on : Tuesday, September 8, 2020 - 2:35:49 PM
Last modification on : Sunday, June 26, 2022 - 2:53:29 AM
Long-term archiving on: : Saturday, December 5, 2020 - 3:13:14 AM


Files produced by the author(s)


  • HAL Id : hal-02933487, version 1


Fajrian Yunus, Chloé Clavel, Catherine I Pelachaud. Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures. Workshop sur les Affects, Compagnons artificiels et Interactions, CNRS, Université Toulouse Jean Jaurès, Université de Bordeaux, Jun 2020, Saint Pierre d'Oléron, France. ⟨hal-02933487⟩



Record views


Files downloads