Neural networks and conditional random fields based approach for effective question processing.

Abstract Question processing is a natural language processing task which is more complex and requires proper understanding of the question. In this work, we aimed to carry out a deep analysis to determine what types of information can be extracted from various question types. The extracted information is of two types: directly available information like entity types and indirectly inferred information like question type, intent, and domain. Named Entity Recognition (NER) methods can be applied for extracting direct entities. There are predominantly two approaches for NER, one is based on Conditional Random Field (CRF) and other is based on Bi-directional Long Short-Term Memory (Bi-LSTM) network. For identifying intent, domain and type, a Convolutional Neural Network (CNN) based model and a Bi-LSTM based model were used. Global Vector Representation (GloVe) is uesd for word embedding. In addition, the initial GloVe embeddings are enriched with synonyms and antonyms by using the counter fitting method. Experiments were conducted over Quora question collection dataset which is annotated manually. Results are promising for both tasks and the word embedding enrichment using counter-fitting method indicate significant importance in capturing semantics. [ABSTRACT FROM AUTHOR]