Skip to Main content Skip to Navigation
Conference papers

Reference-less Quality Estimation of Text Simplification Systems

Abstract : The evaluation of text simplification (TS) systems remains an open challenge. As the task has common points with machine translation (MT), TS is often evaluated using MT metrics such as BLEU. However, such metrics require high quality reference data, which is rarely available for TS. TS has the advantage over MT of being a monolingual task, which allows for direct comparisons to be made between the simplified text and its original version. In this paper, we compare multiple approaches to reference-less quality estimation of sentence-level text simplification systems, based on the dataset used for the QATS 2016 shared task. We distinguish three different dimensions: gram-maticality, meaning preservation and simplicity. We show that n-gram-based MT metrics such as BLEU and METEOR correlate the most with human judgment of grammaticality and meaning preservation, whereas simplicity is best evaluated by basic length-based metrics.
Document type :
Conference papers
Complete list of metadata

Cited literature [39 references]  Display  Hide  Download

https://hal.inria.fr/hal-01959054
Contributor : Louis Martin <>
Submitted on : Tuesday, January 29, 2019 - 4:50:51 PM
Last modification on : Thursday, August 29, 2019 - 2:24:03 PM
Long-term archiving on: : Tuesday, April 30, 2019 - 5:17:57 PM

Files

main.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01959054, version 2
  • ARXIV : 1901.10746

Collections

Citation

Louis Martin, Samuel Humeau, Pierre-Emmanuel Mazaré, Antoine Bordes, Éric Villemonte de la Clergerie, et al.. Reference-less Quality Estimation of Text Simplification Systems. 1st Workshop on Automatic Text Adaptation (ATA), Nov 2018, Tilburg, Netherlands. ⟨hal-01959054v2⟩

Share

Metrics

Record views

93

Files downloads

327