1. Face the Facts! Evaluating RAG-based Fact-checking Pipelines in Realistic Settings
- Author
-
Russo, Daniel, Menini, Stefano, Staiano, Jacopo, and Guerini, Marco
- Subjects
Computer Science - Computation and Language ,Computer Science - Computers and Society - Abstract
Natural Language Processing and Generation systems have recently shown the potential to complement and streamline the costly and time-consuming job of professional fact-checkers. In this work, we lift several constraints of current state-of-the-art pipelines for automated fact-checking based on the Retrieval-Augmented Generation (RAG) paradigm. Our goal is to benchmark, under more realistic scenarios, RAG-based methods for the generation of verdicts - i.e., short texts discussing the veracity of a claim - evaluating them on stylistically complex claims and heterogeneous, yet reliable, knowledge bases. Our findings show a complex landscape, where, for example, LLM-based retrievers outperform other retrieval techniques, though they still struggle with heterogeneous knowledge bases; larger models excel in verdict faithfulness, while smaller models provide better context adherence, with human evaluations favouring zero-shot and one-shot approaches for informativeness, and fine-tuned models for emotional alignment., Comment: Code and data at https://github.com/drusso98/face-the-facts
- Published
- 2024