Grounded physical language understanding with probabilistic programs and simulated worlds

Human language richly invokes our intuitive physical knowledge. We talk about physical objects, scenes, properties, and events; and we can ask questions and answer them with predictions and inferences about physical worlds described entirely in language. How does language construct meanings that connect to our general physical reasoning? In this paper, we propose PiLoT, a computational model that maps language into a probabilistic language of thought—meanings are constructed as probabilistic programs, which provide a formal basis for probabilistic and physical reasoning. Our model uses a large language model (LLM) to map from language to meanings and a probabilistic physics engine to support inferences over scenes described in language. We conduct a linguistic reasoning experiment based on prior psychophysics studies that requires reasoning about physical outcomes based on linguistic descriptions. We show that PiLoT well predicts human judgments across this experiment and outperforms baseline models which use the LLM to directly perform the same task.