Back to Search
Start Over
LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLI.
- Source :
-
Language Resources & Evaluation . Jun2024, Vol. 58 Issue 2, p427-458. 32p. - Publication Year :
- 2024
-
Abstract
- Natural Language Inference (NLI) is considered a representative task to test natural language understanding (NLU). In this work, we propose an extensible framework to collectively yet categorically test diverse Logical reasoning capabilities required for NLI (and, by extension, NLU). Motivated by behavioral testing, we create a semi-synthetic large test bench (363 templates, 363k examples) and an associated framework that offers the following utilities: (1) individually test and analyze reasoning capabilities along 17 reasoning dimensions (including pragmatic reasoning); (2) design experiments to study cross-capability information content (leave one out or bring one in); and (3) the synthetic nature enables us to control for artifacts and biases. We extend a publicly available framework of automated test case instantiation from free-form natural language templates (CheckList) and a well-defined taxonomy of capabilities to cover a wide range of increasingly harder test cases while varying the complexity of natural language. Through our analysis of state-of-the-art NLI systems, we observe that our benchmark is indeed hard (and non-trivial even with training on additional resources). Some capabilities stand out as harder. Further, fine-grained analysis and fine-tuning experiments reveal more insights about these capabilities and the models – supporting and extending previous observations; thus showing the utility of the proposed testbench. [ABSTRACT FROM AUTHOR]
- Subjects :
- *NATURAL languages
*LINGUISTIC complexity
*LANGUAGE ability testing
Subjects
Details
- Language :
- English
- ISSN :
- 1574020X
- Volume :
- 58
- Issue :
- 2
- Database :
- Academic Search Index
- Journal :
- Language Resources & Evaluation
- Publication Type :
- Academic Journal
- Accession number :
- 178064689
- Full Text :
- https://doi.org/10.1007/s10579-023-09691-y