Skip to Main content Skip to Navigation
Conference papers

Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures

Abstract : Communicative gestures and speech prosody are tightly linked. Our aim is to predict when gestures are performed based on prosody. We develop a model based on a seq2seq recurrent neural network with attention mechanism. The model is trained on a corpus of natural dyadic interaction where the speech prosody and the gestures have been annotated. Because the output of the model is a sequence, we use a sequence comparison technique to evaluate the model performance. We find that the model can predict certain gesture classes. In our experiment, we also replace some input features with random values to find which prosody features are pertinent. We find that the F0 is pertinent. Lastly, we also train the model on one speaker and test it with the other speaker to find whether the model is generalisable. We find that the models which we train on one speaker also works for another speaker of the same conversation.
Complete list of metadatas

Cited literature [26 references]  Display  Hide  Download

https://hal.inria.fr/hal-02933487
Contributor : Ccsd Sciencesconf.Org <>
Submitted on : Tuesday, September 8, 2020 - 2:35:49 PM
Last modification on : Monday, October 12, 2020 - 5:55:45 PM

File

wacai_2020_7_.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02933487, version 1

Citation

Fajrian Yunus, Chloé Clavel, Catherine Pelachaud. Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures. Workshop sur les Affects, Compagnons artificiels et Interactions, CNRS, Université Toulouse Jean Jaurès, Université de Bordeaux, Jun 2020, Saint Pierre d'Oléron, France. ⟨hal-02933487⟩

Share

Metrics

Record views

28

Files downloads

102