Audiovisual 3d rendering as a tool for multimodal interfaces

George Drettakis 1
1 REVES - Rendering and virtual environments with sound
CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : In this talk, we will start with a short overview of 3D audiovisual rendering and its applicability to multimodal interfaces. In recent years, we have seen the generalization of 3D applications, ranging from computer games, which involve a high level of realism, to applications such as SecondLife, in which the visual and auditory quality of the 3D environment leaves much to be desired. In our introduction will attempt to examine the relationship between the audiovisual rendering of the environment and the interface. We will then review some of the audio-visual rendering algorithms we have developed in the last few years. We will discuss four main challenges we have addressed. The first is the development of realistic illumination and shadow algorithms which contribute greatly to the realism of 3D scenes, but could also be important for interfaces. The second involves the application of these illumination algorithms to augmented reality settings. The third concerns the development of perceptually-based techniques, and in particular using audio-visual cross-modal perception. The fourth challenge has been the development of approximate but "plausible", interactive solutions to more advanced rendering effects, both for graphics and audio. On the audio side, our review will include the introduction of clustering, masking and perceptual rendering for 3D spatialized audio and our recently developed solution for the treatment of contact sounds. On the graphics side, our discussion will include a quick overview of our illumination and shadow work, its application to augmented reality, our work on interactive rendering approximations and perceptually driven algorithms. For all these techniques we will discuss their relevance to multimodal interfaces, including our experience in a urban design case-study and attempt to relate them to recent interface research. We will close with a broad reflection on the potential for closer collaboration between 3D audiovisual rendering and multimodal interfaces.
Type de document :
Communication dans un congrès
Vassilis Digalakis and Alex Potamianos and Matthew Turk and Roberto Pieraccini and Yuri Ivanov. 10th international conference on Multimodal interfaces (ICMI'08), Jan 2008, Chania, Greece. ACM, 2008, 〈10.1145/1452392.1452394〉
Liste complète des métadonnées

https://hal.inria.fr/inria-00606818
Contributeur : Team Reves <>
Soumis le : jeudi 7 juillet 2011 - 11:14:32
Dernière modification le : jeudi 11 janvier 2018 - 16:20:43

Identifiants

Collections

Citation

George Drettakis. Audiovisual 3d rendering as a tool for multimodal interfaces. Vassilis Digalakis and Alex Potamianos and Matthew Turk and Roberto Pieraccini and Yuri Ivanov. 10th international conference on Multimodal interfaces (ICMI'08), Jan 2008, Chania, Greece. ACM, 2008, 〈10.1145/1452392.1452394〉. 〈inria-00606818〉

Partager

Métriques

Consultations de la notice

86