An alternative scheme for perplexity estimation and its assessment for the evaluation of language models

Abstract : Language models are usually evaluated on test texts using the perplexity derived from the likelihood function computed on these texts (test set perplexity). In order to use this measure in the framework of a comparative evaluation campaign, we have developed an alternative scheme for estimating the test set perplexity. The method is derived from the Shannon game and based on a gambling approach on the next word to come in a truncated sentence. We also study the entropy bounds proposed by Shannon and based on the rank of the correct answer, in order to estimate a perplexity interval for non-probabilistic language models. The relevance of the approach is validated on an example. We then report the results of a preliminary comparative evaluation using the proposed scheme
Document type :
Journal articles
Complete list of metadatas

https://hal.inria.fr/inria-00100687
Contributor : Publications Loria <>
Submitted on : Tuesday, September 26, 2006 - 2:49:19 PM
Last modification on : Saturday, May 4, 2019 - 1:20:26 AM

Identifiers

  • HAL Id : inria-00100687, version 1

Citation

Frédéric Bimbot, Marc El Bèze, Stéphane Igounet, Michèle Jardino, Kamel Smaïli, et al.. An alternative scheme for perplexity estimation and its assessment for the evaluation of language models. Computer Speech and Language, Elsevier, 2001, 15 (1), pp.1-13. ⟨inria-00100687⟩

Share

Metrics

Record views

500