1. LGDE: Local Graph-based Dictionary Expansion
- Author
-
Schindler, Dominik J., Jha, Sneha, Zhang, Xixuan, Buehling, Kilian, Heft, Annett, and Barahona, Mauricio
- Subjects
Computer Science - Computation and Language ,Computer Science - Social and Information Networks ,Physics - Physics and Society - Abstract
We present Local Graph-based Dictionary Expansion (LGDE), a method for data-driven discovery of the semantic neighbourhood of words using tools from manifold learning and network science. At the heart of LGDE lies the creation of a word similarity graph from the geometry of word embeddings followed by local community detection based on graph diffusion. The diffusion in the local graph manifold allows the exploration of the complex nonlinear geometry of word embeddings to capture word similarities based on paths of semantic association, over and above direct pairwise similarities. Exploiting such semantic neighbourhoods enables the expansion of dictionaries of pre-selected keywords, an important step for tasks in information retrieval, such as database queries and online data collection. We validate LGDE on a corpus of English-language hate speech-related posts from Reddit and Gab and show that LGDE enriches the list of keywords with significantly better performance than threshold methods based on direct word similarities. We further demonstrate our method through a real-world use case from communication science, where LGDE is evaluated quantitatively on the expansion of a conspiracy-related dictionary from online data collected and analysed by domain experts. Our empirical results and expert user assessment indicate that LGDE expands the seed dictionary with more useful keywords due to the manifold-learning-based similarity network., Comment: Python code available at: https://github.com/barahona-research-group/LGDE
- Published
- 2024