MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2022

MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification

Résumé

In text summarization and simplification, system outputs must be evaluated along multiple dimensions such as relevance, factual consistency, fluency, and grammaticality, and a wide range of possible outputs could be of high quality. These properties make the development of an adaptable, reference-less evaluation metric both necessary and challenging. We introduce MaskEval, a reference-less metric for text summarization and simplification that operates by performing masked language modeling (MLM) on the concatenation of the candidate and the source texts. It features an attention-like weighting mechanism to modulate the relative importance of each MLM step, which crucially allows it to be adapted to evaluate different quality dimensions. We demonstrate its effectiveness on English summarization and simplification in terms of correlations with human judgments, and explore transfer scenarios between the two tasks.

Dates et versions

hal-03834733 , version 1 (30-10-2022)

Identifiants

Citer

Yu Lu Liu, Rachel Bawden, Thomas Scialom, Benoît Sagot, Jackie Chi Kit Cheung. MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification. 2022. ⟨hal-03834733⟩

Collections

INRIA INRIA2
41 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More