Is ATIS too shallow to go deeper for benchmarking Spoken Language Understanding models?

Abstract : The ATIS (Air Travel Information Service) corpus will be soon celebrating its 30th birthday. Designed originally to benchmark spoken language systems, it still represents the most well-known corpus for benchmarking Spoken Language Understanding (SLU) systems. In 2010, in a paper titled "What is left to be understood in ATIS?" [1], Tur et al. discussed the relevance of this corpus after more than 10 years of research on statistical models for performing SLU tasks. Nowadays, in the Deep Neural Network (DNN) era, ATIS is still used as the main benchmark corpus for evaluating all kinds of DNN models, leading to further improvements, although rather limited, in SLU accuracy compared to previous state-of-the-art models. We propose in this paper to investigate these results obtained on ATIS from a qualitative point of view rather than just a quantitative point of view and answer the two following questions: what kind of qualitative improvement brought DNN models to SLU on the ATIS corpus? Is there anything left, from a qualitative point of view, in the remaining 5% of errors made by current state-of-the-art models?
Type de document :
Communication dans un congrès
InterSpeech 2018, Sep 2018, Hyderabad, India. pp.1-5
Liste complète des métadonnées

https://hal.inria.fr/hal-01835425
Contributeur : Christian Raymond <>
Soumis le : mercredi 11 juillet 2018 - 13:29:04
Dernière modification le : jeudi 6 septembre 2018 - 15:49:39
Document(s) archivé(s) le : samedi 13 octobre 2018 - 01:32:06

Fichier

Interspeech2018(1).pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01835425, version 1

Citation

Frédéric Béchet, Christian Raymond. Is ATIS too shallow to go deeper for benchmarking Spoken Language Understanding models?. InterSpeech 2018, Sep 2018, Hyderabad, India. pp.1-5. 〈hal-01835425〉

Partager

Métriques

Consultations de la notice

262

Téléchargements de fichiers

117