1. The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?
- Author
-
Choi, Alexander S., Akter, Syeda Sabrina, Singh, JP, and Anastasopoulos, Antonios
- Subjects
Computer Science - Computation and Language ,Computer Science - Human-Computer Interaction - Abstract
Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks, leading researchers to use them for time and labor-intensive analyses. However, their capability to handle highly specialized and open-ended tasks in domains like policy studies remains in question. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership. The study, conducted in two stages-Topic Discovery and Topic Assignment-integrates LLMs with expert annotators to observe the impact of LLM suggestions on what is usually human-only analysis. Results indicate that LLM-generated topic lists have significant overlap with human generated topic lists, with minor hiccups in missing document-specific topics. However, LLM suggestions may significantly improve task completion speed, but at the same time introduce anchoring bias, potentially affecting the depth and nuance of the analysis, raising a critical question about the trade-off between increased efficiency and the risk of biased analysis., Comment: Accepted to EMNLP Main 2024. First two authors contributed equally
- Published
- 2024