Structured Penalties for Log-linear Language Models

Anil Nelakanti 1 Cédric Archambeau 1 Julien Mairal 2 Francis Bach 3, 4 Guillaume Bouchard 1, *
* Auteur correspondant
2 LEAR - Learning and recognition in vision
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann, INPG - Institut National Polytechnique de Grenoble
4 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, ENS Paris - École normale supérieure - Paris, Inria Paris-Rocquencourt, CNRS - Centre National de la Recherche Scientifique : UMR8548
Abstract : Language models can be formalized as loglinear regression models where the input features represent previously observed contexts up to a certain length m. The complexity of existing algorithms to learn the parameters by maximum likelihood scale linearly in nd, where n is the length of the training corpus and d is the number of observed features. We present a model that grows logarithmically in d, making it possible to efficiently leverage longer contexts. We account for the sequential structure of natural language using treestructured penalized objectives to avoid overfitting and achieve better generalization.
Type de document :
Communication dans un congrès
EMNLP - Empirical Methods in Natural Language Processing, Oct 2013, Seattle, United States. Association for Computational Linguistics, pp.233-243, 2013, 〈http://aclweb.org/anthology//D/D13/D13-1024.pdf〉
Liste complète des métadonnées

Littérature citée [29 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-00904820
Contributeur : Julien Mairal <>
Soumis le : vendredi 15 novembre 2013 - 12:00:56
Dernière modification le : vendredi 25 mai 2018 - 12:02:06
Document(s) archivé(s) le : dimanche 16 février 2014 - 04:31:00

Fichier

anil_emnlp.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00904820, version 1

Collections

Citation

Anil Nelakanti, Cédric Archambeau, Julien Mairal, Francis Bach, Guillaume Bouchard. Structured Penalties for Log-linear Language Models. EMNLP - Empirical Methods in Natural Language Processing, Oct 2013, Seattle, United States. Association for Computational Linguistics, pp.233-243, 2013, 〈http://aclweb.org/anthology//D/D13/D13-1024.pdf〉. 〈hal-00904820〉

Partager

Métriques

Consultations de la notice

880

Téléchargements de fichiers

285