Skip to Main content Skip to Navigation
Conference papers

Quality of syntactic implication of RL-based sentence summarization

Hoa Le 1 Christophe Cerisara 1 Claire Gardent 1
1 SYNALP - Natural Language Processing : representations, inference and semantics
LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : Work on summarization has explored both reinforcement learning (RL) optimization using ROUGE as a reward and syntax-aware models, such as models whose input is enriched with part-of-speech (POS)-tags and/or dependency information. However, it is not clear what is the respective impact of these approaches beyond the standard ROUGE evaluation metric, which arguably fails to capture several important qualitative aspects of texts. Especially, RL-based for summa-rization is becoming more and more popular. In this paper, we provide a detailed comparison of these two approaches and of their combination along several dimensions that relate to the perceived quality of the generated summaries: how many words are repeated in the output ? How close to the ground truth is the generated distribution of part-of-speech tags ? What is the impact of sentence length ? How good are relevance and grammaticality ? Using the standard Gigaword sentence summarization task, we compare an RL self-critical sequence training (SCST) method with syntax-aware models that leverage POS tags and/or Dependency information. We show that on all qualitative evaluations, the combined model gives the best results, but also that only training with RL and without any syntactic information already gives nearly as good results as syntax-aware models with less parameters and faster training convergence.
Document type :
Conference papers
Complete list of metadatas

Cited literature [31 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02883327
Contributor : Christophe Cerisara <>
Submitted on : Monday, June 29, 2020 - 9:03:27 AM
Last modification on : Wednesday, July 1, 2020 - 3:08:17 AM

File

1912.05493.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02883327, version 1
  • ARXIV : 1912.05493

Citation

Hoa Le, Christophe Cerisara, Claire Gardent. Quality of syntactic implication of RL-based sentence summarization. AAAI Workshop on Engineering Dependable and Secure Machine Learning Systems 2020, Feb 2020, New York, United States. ⟨hal-02883327⟩

Share

Metrics

Record views

28

Files downloads

5