Back to Search Start Over

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text.

Authors :
Gehrmann, Sebastian
Clark, Elizabeth
Sellam, Thibault
Source :
Journal of Artificial Intelligence Research; 2023, Vol. 77, p103-166, 64p
Publication Year :
2023

Abstract

Evaluation practices in natural language generation (NLG) have many known flaws, but improved evaluation approaches are rarely widely adopted. This issue has become more urgent, since neural generation models have improved to the point where their outputs can often no longer be distinguished based on the surface-level features that older metrics rely on. This paper surveys the issues with human and automatic model evaluations and with commonly used datasets in NLG that have been pointed out over the past 20 years. We summarize, categorize, and discuss how researchers have been addressing these issues and what their findings mean for the current state of model evaluations. Building on those insights, we lay out a long-term vision for evaluation research and propose concrete steps for researchers to improve their evaluation processes. Finally, we analyze 66 generation papers from recent NLP conferences in how well they already follow these suggestions and identify which areas require more drastic changes to the status quo. [ABSTRACT FROM AUTHOR]

Details

Language :
English
ISSN :
10769757
Volume :
77
Database :
Supplemental Index
Journal :
Journal of Artificial Intelligence Research
Publication Type :
Academic Journal
Accession number :
173083465
Full Text :
https://doi.org/10.1613/jair.1.13715