1. Soft Measures for Extracting Causal Collective Intelligence
- Author
-
Berijanian, Maryam, Dork, Spencer, Singh, Kuldeep, Millikan, Michael Riley, Riggs, Ashlin, Swaminathan, Aadarsh, Gibbs, Sarah L., Friedman, Scott E., and Brugnone, Nathan
- Subjects
Computer Science - Computation and Language ,Computer Science - Artificial Intelligence ,Computer Science - Computers and Society ,Computer Science - Social and Information Networks - Abstract
Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP., Comment: Camera-ready version accepted for publication in the EMNLP 2024 Workshop NLP4Science
- Published
- 2024