Can distributional semantics explain performance on the false belief task?

In a previous pre-registration (https://osf.io/agqwv), we asked whether GPT-3, a neural language model, showed sensitivity to implied belief states. We found that it did: in a false belief task paradigm, GPT-3 assigned significantly different probability estimates to different search locations as a function of a character's belief states. In other words, GPT-3 was "sensitive" to false belief information, in the sense that its predictions about upcoming words/locations changed according to previous language in the passage that indicated whether the character knew or didn't know about an object's location change. The goal of the present work is to ask whether, and to what extent, human behavior on this task is explained by GPT-3's output, as opposed to the key experimental manipulation of Condition. The theoretical motivation for this work is to test the hypothesis that human behavior on the false belief task requires some sort of *additional* cognitive resource or information source that cannot be obtained from distributional statistics alone. Thus, GPT-3's output serves as a kind of "baseline" to ask about the *sufficiency* (in principle) of distributional statistics, as captured by GPT-3, in explaining sensitivity to false belief on this task. The hypothesis space is as follows: H0 (null): Human behavior is not sensitive to the false belief manipulation. H1 (baseline): Human sensitivity to false belief does not exceed GPT-3’s sensitivity. In other words, distributional statistics are sufficient to explain human behavior on the task, and Condition does not explain additional variance. H2 (>baseline): Human behavior is sensitive to false belief, above and beyond variance in GPT-3 log-odds. Distributional statistics are *insufficient* to explain human behavior on the task, because Condition explains additional variance.