Skip to Main content Skip to Navigation
Conference papers

Using Natural Language Feedback in a Neuro-inspired Integrated Multimodal Robotic Architecture

Johannes Twiefel 1 Xavier Hinaut 2, 1 Marcelo Borghetti 1 Erik Strahl 1 Stefan Wermter 1 
1 KT - Knowledge Technology group [Hamburg]
Department of Informatics [Hamburg]
2 Mnemosyne - Mnemonic Synergy
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest, IMN - Institut des Maladies Neurodégénératives [Bordeaux]
Abstract : In this paper we present a multi-modal human robot interaction architecture which is able to combine information coming from different sensory inputs, and can generate feedback for the user which helps to teach him/her implicitly how to interact with the robot. The system combines vision, speech and language with inference and feedback. The system environment consists of a Nao robot which has to learn objects situated on a table only by understanding absolute and relative object locations uttered by the user and afterwards points on a desired object to show what it has learned. The results of a user study and performance test show the usefulness of the feedback produced by the system and also justify the usage of the system in a real-world applications, as its classification accuracy of multi-modal input is around 80.8%. In the experiments, the system was able to detect inconsistent input coming from different sensory modules in all cases and could generate useful feedback for the user from this information.
Complete list of metadata

Cited literature [14 references]  Display  Hide  Download
Contributor : Xavier Hinaut Connect in order to contact the contributor
Submitted on : Thursday, December 15, 2016 - 7:21:50 PM
Last modification on : Saturday, June 25, 2022 - 7:44:17 PM


Files produced by the author(s)




Johannes Twiefel, Xavier Hinaut, Marcelo Borghetti, Erik Strahl, Stefan Wermter. Using Natural Language Feedback in a Neuro-inspired Integrated Multimodal Robotic Architecture. 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Aug 2016, New York City, United States. pp.52 - 57, ⟨10.1109/ROMAN.2016.7745090⟩. ⟨hal-01417706⟩



Record views


Files downloads