Search

Cognitive science has evolved since early disputes between radical empiricism and radical nativism. The authors are reacting to the revival of radical empiricism spurred by recent successes in deep neural network (NN) models. We agree that language-like mental representations (language-of-thoughts [LoTs]) are part of the best game in town, but they cannot be understood independent of the other players.

While affordance detection and Human-Object interaction (HOI) detection tasks are related, the theoretical foundation of affordances makes it clear that the two are distinct. In particular, researchers in affordances make distinctions between J. J. Gibson's traditional definition of an affordance, "the action possibilities" of the object within the environment, and the definition of a telic affordance, or one defined by conventionalized purpose or use. We augment the HICO-DET dataset with annotations for Gibsonian and telic affordances and a subset of the dataset with annotations for the orientation of the humans and objects involved. We then train an adapted Human-Object Interaction (HOI) model and evaluate a pre-trained viewpoint estimation system on this augmented dataset. Our model, AffordanceUPT, is based on a two-stage adaptation of the Unary-Pairwise Transformer (UPT), which we modularize to make affordance detection independent of object detection. Our approach exhibits generalization to new objects and actions, can effectively make the Gibsonian/telic distinction, and shows that this distinction is correlated with features in the data that are not captured by the HOI annotations of the HICO-DET dataset., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2023 Henlein, Gopinath, Krishnaswamy, Mehler and Pustejovsky.)

Much progress in AI over the last decade has been driven by advances in natural language processing technology, in turn facilitated by large datasets and increased computation power used to train large neural language models. These systems demonstrate apparently sophisticated linguistic understanding or generation capabilities, but often fail to transfer their skills to situations they have not encountered before. We argue that computational situated grounding of linguistic information to real or simulated scenarios provide a solution to some of these learning challenges by creating situational representations that both serve as a formal model of the salient phenomena, and contain rich amounts of exploitable, task-appropriate data for training new, flexible computational models. We approach this problem from a neurosymbolic perspective, using multimodal contextual modeling of interactive situations, events, and object properties, particularly afforded behaviors, and habitats , the situations that condition them. These properties are tightly coupled to processes of situated grounding, and herein we discuss we combine neural and symbolic methods with multimodal simulations to create a platform, VoxWorld, for modeling communication in context, and we demonstrate how neural embedding vectors of symbolically-encoded object affordances facilitate transferring knowledge of objects and situations to novel entities, and learning how to recognize and generate linguistic and gestural denotations., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2022 Krishnaswamy and Pustejovsky.)

Computational lexical resources such as WordNet, PropBank, VerbNet, and FrameNet are in regular use in various NLP applications, assisting in the never-ending quest for richer, more precise semantic representations. Coherent class-based organization of lexical units in VerbNet and FrameNet can improve the efficiency of processing by clustering similar items together and sharing descriptions. However, class members are sometimes quite different, and the clustering in both can gloss over useful fine-grained semantic distinctions. FrameNet officially eschews syntactic considerations and focuses primarily on semantic coherence, associating nouns, verbs and adjectives with the same semantic frame, while VerbNet considers both syntactic and semantic factors in defining a class of verbs, relying heavily on meaning-preserving diathesis alternations. Many VerbNet classes significantly overlap in membership with similar FrameNet Frames, e.g., VerbNet Cooking-45.3 and FrameNet Apply_heat, but some VerbNet classes are so heterogeneous as to be difficult to characterize semantically, e.g., Other_cos-45.4 . We discuss a recent addition to the VerbNet class semantics, verb-specific semantic features, that provides significant enrichment to the information associated with verbs in each VerbNet class. They also implicitly group together verbs sharing semantic features within a class, forming more semantically coherent subclasses. These efforts began with introspection and dictionary lookup, and progressed to automatic techniques, such as using NLTK sentiment analysis on verb members of VerbNet classes with an Experiencer argument role, to assign positive, negative or neutral labels to them. More recently we found the Brandeis Semantic Ontology (BSO) to be an invaluable source of rich semantic information and were able to use a VerbNet-BSO mapping to find fine-grained distinctions in the semantic features of verb members of 25 VerbNet classes. This not only confirmed the assignments previously made to classes such as Admire-31.2 , but also gave a more fine-grained semantic decomposition for the members. Also, for the Judgment-31.1 class, the new method revealed new, more fine-grained existing semantic features for the verbs. Overall, the BSO mapping produced promising results, and as a manually curated resource, we have confidence the results are reliable and need little (if any) further hand-correction. We discuss our various techniques, illustrating the results with specific classes., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2022 Kazeminejad, Palmer, Brown and Pustejovsky.)