Learning to Manipulate Tools by Aligning Simulation to Video Demonstration - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue IEEE Robotics and Automation Letters Année : 2022

Learning to Manipulate Tools by Aligning Simulation to Video Demonstration

Résumé

A seamless integration of robots into human environments requires robots to learn how to use existing human tools. Current approaches for learning tool manipulation skills mostly rely on expert demonstrations provided in the target robot environment, for example, by manually guiding the robot manipulator or by teleoperation. In this work, we introduce an automated approach that replaces an expert demonstration with a Youtube video for learning a tool manipulation strategy. The main contributions are twofold. First, we design an alignment procedure that aligns the simulated environment with the realworld scene observed in the video. This is formulated as an optimization problem that finds a spatial alignment of the tool trajectory to maximize the sparse goal reward given by the environment. Second, we devise an imitation learning approach that focuses on the trajectory of the tool and how it interacts with the environment, rather than the motion of the human. We demonstrate the proposed approach on spade, scythe and hammer tools in simulation, and show the effectiveness of the trained policy for the spade on a real Franka Emika Panda robot demonstration.
Fichier principal
Vignette du fichier
Learning to Manipulate Tools byAligning Simulation to Video Demonstration.pdf (8.42 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03478117 , version 1 (13-12-2021)

Identifiants

Citer

Kateryna Zorina, Justin Carpentier, Josef Sivic, Vladimír Petrík. Learning to Manipulate Tools by Aligning Simulation to Video Demonstration. IEEE Robotics and Automation Letters, In press, ⟨10.48550/arXiv.2111.03088⟩. ⟨hal-03478117⟩
86 Consultations
58 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More