1. Evaluating the Robustness of Question-Answering Models to Paraphrased Questions
- Author
-
Alting von Geusau, Paulo, Bloem, Peter, Baratchi, Mitra, Cao, Lu, Kosters, Walter A., Lijffijt, Jefrey, van Rijn, Jan N., Takes, Frank W., Baratchi, Mitra, Cao, Lu, Kosters, Walter A., Lijffijt, Jefrey, van Rijn, Jan N., Takes, Frank W., Artificial intelligence, Network Institute, and Knowledge Representation and Reasoning
- Subjects
Computer science ,business.industry ,Space (commercial competition) ,computer.software_genre ,Embeddings ,Natural language ,Robustness (computer science) ,Transformers ,Question answering ,Unsupervised learning ,Artificial intelligence ,Set (psychology) ,business ,computer ,Natural language processing ,Sentence - Abstract
Understanding questions expressed in natural language is a fundamental challenge studied under different applications such as question answering (QA). We explore whether recent state-of-the-art models are capable of recognizing two paraphrased questions using unsupervised learning. Firstly, we test QA models’ performance on an existing paraphrased dataset (Dev-Para). Secondly, we create a new paraphrased evaluation set (Para-SQuAD) containing multiple paraphrased question pairs from the SQuAD dataset. We describe qualitative investigations on these models and how they present paraphrased questions in continuous space. The results demonstrate that the paraphrased dataset confuses the QA models and decreases their performance. Visualizing the sentence embeddings of Para-SQuAD by the QA models suggests that all models, except BERT, struggle to recognize paraphrased questions effectively.
- Published
- 2021
- Full Text
- View/download PDF