Search

Modern computing is generally taken to consist primarily of symbol manipulation. But symbols are abstract, and computers are physical. How can a physical device manipulate abstract symbols? Neither Church nor Turing considered this question. My answer is that the bit, as a hardware-implemented abstract data type, serves as a bridge between materiality and abstraction. Computing also relies on three other primitive—but more straightforward—abstractions: Sequentiality, State, and Transition. These physically-implemented abstractions define the borderline between hardware and software and between physicality and abstraction. At a deeper level, asking how a physical device can interact with abstract symbols is the wrong question. The relationship between symbols and physical devices begins with the realization that human beings already know what it means to manipulate symbols. We build and program computers to do what we understand to be symbol manipulation. To understand what that means, consider a light switch. A light switch doesn’t turn a light on or off. Those are abstractions. Light switches don’t operate with abstractions. We build light switches (and their associated circuitry), so that when flipped, the world is changed in such a way that we understand the light to be on or off. Similarly, we build computers to perform operations that we understand as manipulating symbols.

In this paper, we propose an HISNav VQA dataset – a challenging dataset for a Visual Question Answering task that is aimed at the needs of Visual Navigation in human-centered environments. The dataset consists of images of various room scenes that were captured using the Habitat virtual environment and of questions important for navigation tasks using only visual information. We also propose a baseline for a HISNav VQA dataset, a Vector Semiotic Architecture, and demonstrate its performance. The Vector Semiotic Architecture is a combination of a Sign-Based World Model and Vector Symbolic Architectures. The Sign-Based World Model allows representing various aspects of an agent’s knowledge, and Vector Symbolic Architectures serve on a low computational level. The Vector Semiotic Architecture addresses the symbol grounding problem that plays an important role in the Visual Question Answering Task.

We present a neural network based system capable of learning a multimodal representation of images and words. This representation allows for bidirectional grounding of the meaning of words and the visual attributes that they represent, such as colour, size and object name. We also present a new dataset captured specifically for this task.

Allowing humans to communicate through natural language with robots requires connections between words and percepts. The process of creating these connections is called symbol grounding and has been studied for nearly three decades. Although many studies have been conducted, not many considered grounding of synonyms and the employed algorithms either work only offline or in a supervised manner. In this paper, a cross-situational learning based grounding framework is proposed that allows grounding of words and phrases through corresponding percepts without human supervision and online, i.e. it does not require any explicit training phase, but instead updates the obtained mappings for every new encountered situation. The proposed framework is evaluated through an interaction experiment between a human tutor and a robot, and compared to an existing unsupervised grounding framework. The results show that the proposed framework is able to ground words through their corresponding percepts online and in an unsupervised manner, while outperforming the baseline framework., Comment: 11 pages, 6 figures, 3 tables; Published in Proceedings of the Second Grand Challenge and Workshop on Multimodal Language (Challenge-HML) in the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)

Even with the relative ascendancy of sub-symbolic approaches to AI, the symbol grounding problem presents an ongoing challenge to work in artificial general intelligence. Prevailing ontology design practices typically presuppose the transparency of the relation between semantics and syntax by transcendently stipulating it extrinsic to the system, rather than providing a platform for the internal development of this relation through agents’ interactions endogenous to the system. Drawing on theoretical resources from ecological psychology, dynamical systems theory, and interactive computation, this work suggests an inversion of the symbol grounding problem in order to analyze how the symbolic regime can emerge out of causally embedded dynamical interactions within a system of autonomous intelligent agents. Under this view, syntactic-symbols come to be stabilized from other signs as constraints harnessing the dynamics of agents’ inter-actions, where the functional effects generated by such constrained dynamics give rise to an internal characterization of semantics broadly aligned with Brandom’s semantic pragmatism. Finally, ludics—a protological framework based on interactive computation—provides a formal model to concretely describe this continuity between syntax and semantics arising within and through the regulative dynamics of interactions in a multi-agent system. Accordingly, this bottom-up approach to grounding the symbolic order in dynamics could provide the conditions for artificial agents to engage in autonomous language design, thus equipping themselves with a powerful cognitive technology for intersubjective coordination and (re)structuring ontologies within a community of agents.

Machine learning is making substantial progress in diverse applications. The success is mostly due to advances in deep learning. However, deep learning can make mistakes and its generalization abilities to new tasks are questionable. We ask when and how one can combine network outputs, when (i) details of the observations are evaluated by learned deep components and (ii) facts and rules are available. The Deep Consistence Seeking (DCS) machine seeks for consistent and deterministic event descriptions and improves the representation accordingly. The machine has an anomaly detection component that may trigger coherence seeking. Coherence seeking resolves conflicts between computational modules by preferring components with higher scores. We illustrate that context can help in correcting recognitions and in deriving training samples for self-training. We put these concepts into a general framework of cognition, by distinguishing creativity, rule extraction, verification, and symbol grounding. We demonstrate our approach in a driving scenario.

Theories of embodied cognition assume that concepts are grounded in non-linguistic, sensorimotor experience. In support of this assumption, previous studies have shown that upwards response movements are faster than downwards movements after participants have been presented with words whose referents are typically located in the upper vertical space (and vice versa for downwards responses). This is taken as evidence that processing these words reactivates sensorimotor experiential traces. This congruency effect was also found for novel words, after participants learned these words as labels for novel objects that they encountered either in their upper or lower visual field. While this indicates that direct experience with a word's referent is sufficient to evoke said congruency effects, the present study investigates whether this direct experience is also a necessary condition. To this end, we conducted five experiments in which participants learned novel words from purely linguistic input: Novel words were presented in pairs with real up- or down-words (Experiment 1); they were presented in natural sentences where they replaced these real words (Experiment 2); they were presented as new labels for these real words (Experiment 3); and they were presented as labels for novel combined concepts based on these real words (Experiment 4 and 5). In all five experiments, we did not find any congruency effects elicited by the novel words; however, participants were always able to make correct explicit judgements about the vertical dimension associated to the novel words. These results suggest that direct experience is necessary for reactivating experiential traces, but this reactivation is not a necessary condition for understanding (in the sense of storing and accessing) the corresponding aspects of word meaning.

A probabilistic stance is proposed for tackling the symbol grounding problem.The outcome of such grounding is accommodated in a novel semantic map representation.This semantic map considers different interpretations of the robot workspace.A more coherent robot operation is achieved by exploring such interpretations.Our proposal has been assessed employing the Robot@Home dataset. Semantic maps augment metric-topological maps with meta-information, i.e. l semantic knowledge aimed at the planning and execution of high-level robotic tasks. Semantic knowledge typically encodes human-like concepts, like types of objects and rooms, which are connected to sensory data when symbolic representations of percepts from the robot workspace are grounded to those concepts. Such a symbol grounding is usually carried out by algorithms that individually categorize each symbol and provide a crispy outcome a symbol is either a member of a category or not. Such approach is valid for a variety of tasks, but it fails at: (i) dealing with the uncertainty inherent to the grounding process, and (ii) jointly exploiting the contextual relations among concepts (e.g. microwaves are usually in kitchens). This work provides a solution for probabilistic symbol grounding that overcomes these limitations. Concretely, we rely on Conditional Random Fields (CRFs) to model and exploit contextual relations, and to provide measurements about the uncertainty coming from the possible groundings in the form of beliefs (e.g. an object can be categorized (grounded) as a microwave or as a nightstand with beliefs 0.6 and 0.4, respectively). Our solution is integrated into a novel semantic map representation called Multiversal Semantic Map (MvSmap), which keeps the sets of different groundings, or universes, as instances of ontologies annotated with the obtained beliefs for their posterior exploitation. The suitability of our proposal has been proven with the Robot@Home dataset, a repository that contains challenging multi-modal sensory information gathered by a mobile robot in home environments.

Behavior planning is an important function of any complex technical facility intelligent control system. Presently, a symbol paradigm of artificial intelligence offers a variety of planning algorithms, including those that use precedent information, i.e. algorithms based on acquired knowledge. A symbol grounding problem within the exiting approaches of knowledge representation does not allow effective use the developed algorithms together with learning mechanisms for the purpose of solving a wide variety of applied problems by actual intelligent agents (robotics systems). This article presents the original planning algorithm (MAP Planner), which uses a sign world model as the basis for acquisition and maintenance of knowledge for future use in behavior planning. the sign problem approach describes planning as a cognitive function actualized by the world model of a subject of activity. Apart from solving symbol grounding problems and ensuring psychological and biological plausibility, a sign planning process model allows interaction of an intelligent agent with other participants in solving a cooperative task. The article presents the description of the knowledge representation method used, a MAP planning algorithm, and a model experiment in a “block world”.

Integrating computer vision and natural language processing is a novel interdisciplinary field that has received a lot of attention recently. In this survey, we provide a comprehensive introduction of the integration of computer vision and natural language processing in multimedia and robotics applications with more than 200 key references. The tasks that we survey include visual attributes, image captioning, video captioning, visual question answering, visual retrieval, human-robot interaction, robotic actions, and robot navigation. We also emphasize strategies to integrate computer vision and natural language processing models as a unified theme of distributional semantics. We make an analog of distributional semantics in computer vision and natural language processing as image embedding and word embedding, respectively. We also present a unified view for the field and propose possible future directions.

We address one-shot imitation learning, where the goal is to execute a previously unseen task based on a single demonstration. While there has been exciting progress in this direction, most of the approaches still require a few hundred tasks for meta-training, which limits the scalability of the approaches. Our main contribution is to formulate one-shot imitation learning as a symbolic planning problem along with the symbol grounding problem. This formulation disentangles the policy execution from the inter-task generalization and leads to better data efficiency. The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures. We address this challenge by proposing a continuous relaxation of the discrete symbolic planner that directly plans on the probabilistic outputs of the symbol grounding model. Our continuous relaxation of the planner can still leverage the information contained in the probabilistic symbol grounding and significantly improve over the baseline planner for the one-shot imitation learning tasks without using large training data., IROS 2019

Human language is capable of communicating mental models between speakers of a language. The question why and how this works is closely tied to a specific variant of the symbol grounding problem that still leaves many open questions. This paper presents the core mechanism of the TextMap system, a logic-based system for generating visuospatial representations from textual input. The system leverages a recent discovery linking logical truth tables of formulae to images: a simple model counting mechanism that automatically extracts coordinate information from propositional Horn-logic knowledge bases encoding spatial predications. The system is based on a biologically inspired low-level bit vector mechanism, the activation bit vector machine (ABVM). It does not require an ontology apart from a list of which tokens indicate relations. Its minimalism and simplicity make TextMap a general purpose visualization or imagery tool. This paper demonstrates the core model counting mechanism and the results of a larger case study of a geographic layout of 13 cities.

The speed and accuracy with which robots are able to interpret natural language is fundamental to realizing effective human-robot interaction. A great deal of attention has been paid to developing models and approximate inference algorithms that improve the efficiency of language understanding. However, existing methods still attempt to reason over a representation of the environment that is flat and unnecessarily detailed, which limits scalability. An open problem is then to develop methods capable of producing the most compact environment model sufficient for accurate and efficient natural language understanding. We propose a model that leverages environment-related information encoded within instructions to identify the subset of observations and perceptual classifiers necessary to perceive a succinct, instruction-specific environment representation. The framework uses three probabilistic graphical models trained from a corpus of annotated instructions to infer salient scene semantics, perceptual classifiers, and grounded symbols. Experimental results on two robots operating in different environments demonstrate that by exploiting the content and the structure of the instructions, our method learns compact environment representations that significantly improve the efficiency of natural language symbol grounding., Accepted to ICRA 2019

The cognitive science research aims to make the machines to understand the human (natural) languages such as English and Tamil as the way humans do. As a result, machines can be programmed and instructed by human languages. As a starting point, the machine should understand the word meanings before it starts to understand the sentences. In the last fifty years, research community has put lots of effort to understand how to represent the meanings of words so that machine can manipulate the meanings in order to understand the sentences. Current research works represent the meaning of the word (i.e. symbol) in terms of other words like in dictionaries and this is useful only for humans to understand the meanings of the symbol and this approach is not suitable for machines. To overcome this issue, the concept of symbol grounding is proposed. Symbol grounding is the task of assigning meaning(s) to the word and physical symbol grounding is the task of assigning meaning to the word that refers to physical world object such as apple, cup, bottle, etc. The physical symbol grounding research focuses more on how to model or represent physical world object. Many research works have been done in the physical symbol grounding and they have the following limitations: a) the feature values can be seen as another way of representing the entity. Just capturing the color and two-dimensional shape parameters does not constitute the complete representation or modeling of the entity. So, the feature values extracted from the signal are incomplete. b) The feature values are helpful for the robot only to recognize the entity in the signals. The above systems assume that recognition is understanding, but actually it is not so. c) They use preprogrammed semantic rules to understand the meaning of the set of words (i.e. sentence) and also preprogrammed action rules to execute the behaviors in robot. Because of these limitations, machines using the above computational models for physical symbol grounding cannot become an autonomous or intelligent machine to understand the human language. To overcome the above limitations, a new architecture based on 3D object’s geometrical properties is proposed for the physical symbol grounding.

How many words (and which ones) are sufficient to define all other words? When dictionaries are analyzed as directed graphs with links from defining words to defined words, they reveal a latent structure. Recursively removing all words that are reachable by definition but that do not define any further words reduces the dictionary to a Kernel of about 10%. This is still not the smallest number of words that can define all the rest. About 75% of the Kernel turns out to be its Core, a Strongly Connected Subset of words with a definitional path to and from any pair of its words and no word's definition depending on a word outside the set. But the Core cannot define all the rest of the dictionary. The 25% of the Kernel surrounding the Core consists of small strongly connected subsets of words: the Satellites. The size of the smallest set of words that can define all the rest (the graph's Minimum Feedback Vertex Set or MinSet) is about 1% of the dictionary, 15% of the Kernel, and half-Core, half-Satellite. But every dictionary has a huge number of MinSets. The Core words are learned earlier, more frequent, and less concrete than the Satellites, which in turn are learned earlier and more frequent but more concrete than the rest of the Dictionary. In principle, only one MinSet's words would need to be grounded through the sensorimotor capacity to recognize and categorize their referents. In a dual-code sensorimotor-symbolic model of the mental lexicon, the symbolic code could do all the rest via re-combinatory definition., 38 pages, 10 figures, 2 tables, 73 references

A natural language interface exploits the conceptual simplicity and naturalness of the language to create a high-level user-friendly communication channel between humans and machines. One of the promising applications of such interfaces is generating visual interpretations of semantic content of a given natural language that can be then visualized either as a static scene or a dynamic animation. This survey discusses requirements and challenges of developing such systems and reports 26 graphical systems that exploit natural language interfaces and addresses both artificial intelligence and visualization aspects. This work serves as a frame of reference to researchers and to enable further advances in the field., Due to copyright most of the figures only appear in the journal version

Intelligent systems need to understand and respond to human words to enable them to interact with humans in a natural way. Several studies attempted to realize these abilities by investigating the symbol grounding problem. For example, we proposed multilayered multimodal latent Dirichlet allocation (mMLDA) to enable the formation of various concepts and inference using grounded concepts. We previously reported on the issue of connecting words to various hierarchical concepts and also proposed a simple preliminary algorithm for generating sentences. This paper proposes a novel method that enables a sensing system to verbalize an everyday scene it observes. The method uses mMLDA and Bayesian hidden Markov models (BHMM) and the proposed algorithm improves the word inference of our previous work. The advantage of our approach is that grammar learning based on BHMM not only boosts concept selection results but enables our method to process functional words. The proposed verbalization algorithm produces results...

Cognitive science is a form of "reverse engineering" (as Dennett has dubbed it). We are trying to explain the mind by building (or explaining the functional principles of) systems that have minds. A "Turing" hierarchy of empirical constraints can be applied to this task, from t1, toy models that capture only an arbitrary fragment of our performance capacity, to T2, the standard "pen-pal" Turing Test (total symbolic capacity), to T3, the Total Turing Test (total symbolic plus robotic capacity), to T4 (T3 plus internal [neuromolecular] indistinguishability). All scientific theories are underdetermined by data. What is the right level of empirical constraint for cognitive theory? I will argue that T2 is underconstrained (because of the Symbol Grounding Problem and Searle's Chinese Room Argument) and that T4 is overconstrained (because we don't know what neural data, if any, are relevant). T3 is the level at which we solve the "other minds" problem in everyday life, the one at which evolution operates (the Blind Watchmaker is no mind-reader either) and the one at which symbol systems can be grounded in the robotic capacity to name and manipulate the objects their symbols are about. I will illustrate this with a toy model for an important component of T3 -- categorization -- using neural nets that learn category invariance by "warping" similarity space the way it is warped in human categorical perception: within-category similarities are amplified and between-category similarities are attenuated. This analog "shape" constraint is the grounding inherited by the arbitrarily shaped symbol that names the category and by all the symbol combinations it enters into. No matter how tightly one constrains any such model, however, it will always be more underdetermined than normal scientific and engineering theory. This will remain the ineliminable legacy of the mind/body problem.

An important characteristic of human language is compositionality. We can efficiently express a wide variety of real-world situations, events, and behaviors by compositionally constructing the meaning of a complex expression from a finite number of elements. Previous studies have analyzed how machine-learning models, particularly neural networks, can learn from experience to represent compositional relationships between language and robot actions with the aim of understanding the symbol grounding structure and achieving intelligent communicative agents. Such studies have mainly dealt with the words (nouns, adjectives, and verbs) that directly refer to real-world matters. In addition to these words, the current study deals with logic words, such as “not,” “and,” and “or” simultaneously. These words are not directly referring to the real world, but are logical operators that contribute to the construction of meaning in sentences. In human–robot communication, these words may be used often. The current study builds a recurrent neural network model with long short-term memory units and trains it to learn to translate sentences including logic words into robot actions. We investigate what kind of compositional representations, which mediate sentences and robot actions, emerge as the network's internal states via the learning process. Analysis after learning shows that referential words are merged with visual information and the robot's own current state, and the logical words are represented by the model in accordance with their functions as logical operators. Words such as “true,” “false,” and “not” work as non-linear transformations to encode orthogonal phrases into the same area in a memory cell state space. The word “and,” which required a robot to lift up both its hands, worked as if it was a universal quantifier. The word “or,” which required action generation that looked apparently random, was represented as an unstable space of the network's dynamical system.

From the start, Kurt Godel observed that computer and brain paradigms were considered on a par by researchers and that researchers had misunderstood his theorems. He hailed with displeasure that the brain transcends computers. In this brief article, we point out that Artificial Intelligence (AI) comprises multitudes of human-made methodologies, systems, and languages, and implemented with computer technology. These advances enhance development in the electron and quantum realms. In the biological realm, animal neurons function, also utilizing electron flow, and are products of evolution. Mirror neurons are an important paradigm in neuroscience research. Moreover, the paradigm shift proposed here - 'hall of mirror neurons' - is a potentially further productive research tactic. These concepts further expand AI and brain research.

In this paper, we propose a Bayesian generative model that can form multiple categories based on each sensory-channel and can associate words with any of the four sensory-channels (action, position, object, and color). This paper focuses on cross-situational learning using the co-occurrence between words and information of sensory-channels in complex situations rather than conventional situations of cross-situational learning. We conducted a learning scenario using a simulator and a real humanoid iCub robot. In the scenario, a human tutor provided a sentence that describes an object of visual attention and an accompanying action to the robot. The scenario was set as follows: the number of words per sensory-channel was three or four, and the number of trials for learning was 20 and 40 for the simulator and 25 and 40 for the real robot. The experimental results showed that the proposed method was able to estimate the multiple categorizations and to learn the relationships between multiple sensory-channels and words accurately. In addition, we conducted an action generation task and an action description task based on word meanings learned in the cross-situational learning scenario. The experimental results showed that the robot could successfully use the word meanings learned by using the proposed method.

A standard approach in the simulation of language evolution is the use of Language Games to model communicative interactions between intelligent agents. Usually, in such language games, an agent uses the results from its perceptual layer to categorize and to conceptualize world objects, a process named categorization. In this paper, we develop an approach to the categorization process, where the decomposition of reality in meaningful experiences is co-evolved with the lexicon formation in the language game. This approach brings some insights on how meaning might be assigned to symbols, in a dynamic and continuously changing environment being experienced by an agent. In order to do that, we use Barsalou’s notion of mental simulation and Gardenfors’ notion of conceptual spaces such that, together with ESOM neural networks, a cognitive architecture can be developed, where mental concepts formation and lexicon formation are able to co-evolve during a language game. The performance of our cognitive architecture is evaluated and the results show that the architecture is able to fulfill its semantics function, by allowing a population of agents to exchange the meaning of linguistic symbols during a naming game, without relying on “a priori” categorization scheme provided by an external expert or a set of examples for training a neural network in a previous discrimination game. These results, beyond bringing evidence on potential ways for symbols to get meaning on a biologically realistic way, open a set of possibilities for further uses of conceptual spaces on a much more complex problem: the grounding of a grammatical language.

How do abstract words get meaning? And how could sensorimotor experience based representation be used for abstract words? This is a very important problem for cognitive science, neuroscience and cognitive robotics because these words are far from perception and are perceived through the mind, not through the senses.. In this paper a cognitive robotic model is proposed to obtain abstract words semantic representation through indirect grounding in sensorimotor experience. The model employs the symbolic knowledge representation method (graph-based semantic network) for the robot's conceptual knowledge. The robot perceives abstract words in this model by referring to already grounded concrete words with sensorimotor experience, so it is not subject to the symbol grounding problem. An algorithm is written to obtain semantic referents of abstract words through transitive inference method. A simulation experiment has been designed for DARwIn-OP robot model for verification.

This chapter presents the Cognitive Symbol Grounding framework for the grounding of language into perception, cognition, and action. This approach is characterized by the hypothesis that symbols are directly grounded into internal categorical representations, whilst at the same time having logical (e.g., syntactic) relationships with other symbols. The internal categorical representations, constituting the meanings upon which symbols are grounded, include perceptual, sensorimotor, and social categories, as well as internal state representations. Three main modeling approaches to the symbol grounding are presented: (1) the connectionist approach, based on artificial neural networks for category learning and naming tasks, (2) computational models of the emergence of language in simulated embodied agents, and (3) developmental robotics models of language learning. These models provide an integrative view of the cognitive systems and help our understanding of the relationships between vision, action, and language.

Abstract. In two experiments, we showed that irrelevant numerical information influenced the speed of sentence-picture verification. Participants were asked to verify whether the concept mentioned in a sentence matched the object presented in a subsequent picture. Concurrently, the number word attached to the concept in the sentence and the quantity of objects presented in the picture were manipulated (numerical congruency). The number of objects varied from one to four. In Experiment 1, participants read statements such as three dogs. In Experiment 2, they read sentences such as three dogs were wandering in the street. In both experiments, the verification speed revealed the interaction between response and numerical congruency. The verification times for concept-object match were faster when there was also numerical congruence (compared with incongruence) between the number word and quantity. On the other hand, there was no difference between numerical congruence and incongruence when the concept and object mismatched. The results are interpreted as evidence for the symbol grounding of number words in perceptual representation of small quantities, that is, quantities falling in the subitization range.

In Jones and Goel (2012), we describe a meta-reasoning architecture that uses abstraction networks (ANs) and empirical verification procedures (EVPs) to ground self-diagnosis and self-repair of domain knowledge in perception. In particular, we showed that when a hierarchical classifier organized as an AN makes an incorrect prediction, then meta-reasoning can help diagnose and repair the semantics of the concepts in the network. Further, we demonstrated that if an EVP associated with each concept in the network can verify the semantics of that concept at diagnosis time, then the meta-reasoner can perform knowledge diagnosis and repair tractably. In this article, we report on three additional results on the use of perceptually grounded meta-reasoning for correcting prediction errors. Firstly, a new theoretical analysis indicates that the meta-reasoning diagnostic procedure is optimal and establishes the knowledge conditions under which the learning converges. Secondly, an empirical study indicates that the EVPs themselves can be adapted through refining the conceptual semantics. Thirdly, another empirical study shows that if EVPs cannot be defined for all concepts in a hierarchy, the computational technique degrades gracefully. While the theoretical analysis provides a deeper explanation of the sources of power in ANs, the two empirical studies demonstrate ways in which the strong assumptions made by ANs in their most basic form can be relaxed.

Each natural language phrase is evidence for a particular strategy of construing reality. One domain where this has been extensively studied is spatial language, which reveals an enormous amount of variation of conceptualization strategies both within a particular language and cross-culturally. This paper proposes a computational formalism for representing conceptualization strategies and shows how the formalism can be used to study and explain the evolution and emergence of spatial conceptualization strategies and their impact on shared grounded communication systems.

As Alan Turing showed in his epochal work, computation is formal symbol-manipulation—reading and writing symbols based on rules that operate only on the arbitrary shapes of the symbols (whether 0’s and 1’s or the words of a natural language) not on their meanings. So where does the meaning of symbols come from? My version of this impasse was the Chinese-Chinese dictionary-go-round: How can one learn the meanings of Chinese words from a Chinese-Chinese dictionary alone? Everything is defined there, but the definitions are meaningless unless you already know what some of them, at least, mean. How to “ground” the meanings of those symbols, so the rest of them can be defined and described through words alone? That is what I dubbed the “symbol grounding problem”. The natural candidate for grounding symbols is direct sensorimotor interaction with the things the symbols refer to, so that the grounded symbol system can recognize manipulate and name them, autonomously.... These are sensorimotor and robotic capacities... To categorize is to do the right thing with the right kind of thing. The focus of my research has since been on how we acquire categories... Symbol grounding is also the necessary condition for the evolution of language, but language has to be grounded in the capacity to acquire some categories, at least, directly, through the senses, rather than indirectly, through language...

We propose a bio-inspired unsupervised connectionist architecture and apply it to grounding the spatial phrases. The two-layer architecture combines by concatenation the information from the visual and the phonological inputs. In the first layer, the visual pathway employs separate ‘what’ and ‘where’ subsystems that represent the identity and spatial relations of two objects in 2D space, respectively. The bitmap images are presented to an artificial retina and the phonologically encoded five-word sentences describing the image serve as the phonological input. The visual scene is hence represented by several self-organizing maps (SOMs) and the phonological description is processed by the Recursive SOM that learns to topographically represent the spatial phrases, represented as five-word sentences (e.g., ‘blue ball above red cup’). Primary representations from the first-layer modules are unambiguously integrated in a multimodal second-layer module, implemented by the SOM or the ‘neural gas’ algorithms. The system learns to bind proper lexical and visual features without any prior knowledge. The simulations reveal that separate processing and representation of the spatial location and the object shape significantly improve the performance of the model. We provide quantitative experimental results comparing three models in terms of their accuracy.

Recently, a lot of studies in the domain of numerical cognition have been published demonstrating a robust association between numerical symbol processing and individual differences in mathematics achievement. Because numerical symbols are so important for mathematics achievement, many researchers want to provide an answer on the 'symbol grounding problem,' i.e., how does a symbol acquires its numerical meaning? The most popular account, the approximate number system (ANS) mapping account, assumes that a symbol acquires its numerical meaning by being mapped on a non-verbal and ANS. Here, we critically evaluate four arguments that are supposed to support this account, i.e., (1) there is an evolutionary system for approximate number processing, (2) non-symbolic and symbolic number processing show the same behavioral effects, (3) non-symbolic and symbolic numbers activate the same brain regions which are also involved in more advanced calculation and (4) non-symbolic comparison is related to the performance on symbolic mathematics achievement tasks. Based on this evaluation, we conclude that all of these arguments and consequently also the mapping account are questionable. Next we explored less popular alternative, where small numerical symbols are initially mapped on a precise representation and then, in combination with increasing knowledge of the counting list result in an independent and exact symbolic system based on order relations between symbols. We evaluate this account by reviewing evidence on order judgment tasks following the same four arguments. Although further research is necessary, the available evidence so far suggests that this symbol-symbol association account should be considered as a worthy alternative of how symbols acquire their meaning. ispartof: FRONTIERS IN PSYCHOLOGY vol:7 issue:OCT ispartof: location:Switzerland status: published

One long term goal of artificial intelligence research is the development of robot systems, which have approximately the same cognitive, communicational, and handling abilities like humans. For instance, a human-like communication requires an intuitive and symbolic user interface. Since actual robot control systems typically consist of subsymbolic interfaces, the robot must be able to extract subsymbolic information from given symbolic instructions. This requires the grounding of information and the utilizing of appropriate sensors and components. In this work, we describe the grounding of symbols based on physical properties, since we describe executable actions in form of physical effects in our system [1].

An approach to designing artificial agents, capable of uttering modal class-membership statements, is outlined. Related research and development problems are determined and briefly described. A possible way of solving the Harnad's classical Symbol Grounding Problem for modal class-membership statements as complex language symbols is formulated. The proposal is based on a related research into technical implementations and application of cognitive semantics. Apart from a classical setting, the resulting agent is to be equipped with an embedded prototype-based ontology, extracted from collected empirical data.

The goal of our paper is to learn the association and the semantic grounding of two sensory input signals that represent the same semantic concept. The input signals can be or cannot be the same modality. This task is inspired by infants learning. We propose a novel framework that has two symbolic Multilayer Perceptron (MLP) in parallel. Furthermore, both networks learn to ground semantic concepts and the same coding scheme for all semantic concepts in both networks. In addition, the training rule follows EM-approach. In contrast, the traditional setup of association task pre-defined the coding scheme before training. We have tested our model in two cases: mono- and multi-modal. Our model achieves similar accuracy association to MLPs with pre-defined coding schemes.

The limited understanding of the surrounding environment still restricts the capabilities of robotic systems in real world applications. Specifically, the acquisition of knowledge about the environment typically relies only on perception, which requires intensive ad hoc training and is not sufficiently reliable in a general setting. In this paper, we aim at integrating new acquisition devices, such as tangible user interfaces, speech technologies and vision-based systems, with established AI methodologies, to present a novel and effective knowledge acquisition approach. A natural interaction paradigm is presented, where humans move within the environment with the robot and easily acquire information by selecting relevant spots, objects, or other relevant landmarks. The synergy between novel interaction technologies and semantic knowledge leverages humans' cognitive skills to support robots in acquiring and grounding knowledge about the environment; such richer representation can be exploited in the realization of robot autonomous skills for task accomplishment.

In human designer usage, symbols have a rich semantics, grounded on experience, which permits flexible usage - e.g. design ideation is improved by meanings triggered by contrastive words. In computational usage however, symbols are syntactic tokens whose semantics is mostly left to the implementation, resulting in brittle failures in many knowledge-based systems. Here we ask if one may define symbols in computational design as {label,meaning} pairs, as opposed to merely the label. We consider three questions that must be answered to bootstrap a symbol learning process: (a) which concepts are most relevant in a given domain, (b) how to define the semantics of such symbols, and (c) how to learn labels for these so as to form a grounded symbol. We propose that relevant symbols may be discovered by learning patterns of functional viability. The stable patterns are information-conserving codes, also called chunks in cognitive science, which relate to the process of acquiring expertise in humans. Regions of a design space that contain functionally superior designs can be mapped to a lower-dimensional manifold; the inter-relations of the design variables discovered thus constitute the chunks. Using these as the initial semantics for symbols, we show how the system can acquire labels for them by communicating with human designers. We demonstrate the first steps in this process in our baby designer approach, by learning two early grounded symbols, tight and loose.

This paper presents the S-Chart framework, an approach for semantic image interpretation of line charts; and the InteliStrata system, an application for semantic interpretation of gamma ray profiles. The S-Chart framework is structured as a set of knowledge models and algorithms that can be instantiated to accomplish chart interpretation in all sorts of domains. The knowledge models are represented in three semantic levels and apply the concept of symbol grounding in order to map the representation primitives between the levels. The interpretation algorithms carry out the interaction between the high-level symbolic reasoning, and the low-level signal processing. In order to demonstrate the applicability of the S-Chart framework, we developed the InteliStrata system, an application in Geology for the semantic interpretation of gamma ray profiles. Using the developed application, we have interpreted the charts of two gamma ray profiles captured in petroleum exploration wells, indicating the position of stratigraphic sequences and maximum flooding surfaces. The results were compared with the interpretation produced by an experienced geologist using the same data input. The system carried out interpretation that were compatible with the geologist interpretation over the data. Our framework has the advantage of allowing the integration of existing domain ontologies with domain independent visual knowledge models and also the ability of grounding domain concepts in low-level data.

TWIG (“Transportable Word Intension Generator”) is a system that allows a robot to learn compositional meanings for new words that are grounded in its sensory capabilities. The system is novel in its use of logical semantics to infer which entities in the environment are the referents (extensions) of unfamiliar words; its ability to learn the meanings of deictic (“I,” “this”) pronouns in a real sensory environment; its use of decision trees to implicitly contrast new word definitions with existing ones, thereby creating more complex definitions than if each word were treated as a separate learning problem; and its ability to use words learned in an unsupervised manner in complete grammatical sentences for production, comprehension, or referent inference. In an experiment with a physically embodied robot, TWIG learns grounded meanings for the words “I” and “you,” learns that “this” and “that” refer to objects of varying proximity, that “he” is someone talked about in the third person, and that “above” and “below” refer to height differences between objects. Follow-up experiments demonstrate the system's ability to learn different conjugations of “to be”; show that removing either the extension inference or implicit contrast components of the system results in worse definitions; and demonstrate how decision trees can be used to model shifts in meaning based on context in the case of color words.

Original article can be found at: http://www.informaworld.com/ Copyright Taylor and Francis [Full text of this article is not available in the UHRA]

This article presents a synthetic modeling approach to the problem of grounded construction of concepts. In many computational models of grounded language acquisition and evolution, meanings are created in the process of discrimination between a chosen object and other objects present on the scene of communication. We argue that categories constructed for the purpose of identification rather than discrimination are more suitable for the detached language use (talking about things not present here and now). We describe a semantics based on so-called identification criteria constructed by extracting cross-situational similarities among instances of a category, and present several computational models. In the model of individual category construction, the instances are grouped to categories by common motor programs (affordances), while in the model of social learning, focused on the influence of naming on category formation, entities are considered members of the same category, if they are labeled with the same word by an external teacher. By these two mechanisms, the learner can construct interactionally grounded representation of objects, properties, relations, changes, complex situations and events. We also report and analyze simulation results of an experiment focused on the dynamics of meanings in iterated intergenerational transmission.

We propose a method for inferring semantic information from textual data in content-based multimedia retrieval. Training examples of images and videos belonging to a specific semantic class are associated with their low-level visual and aural descriptors augmented with textual features such as frequencies of significant words. A fuzzy mapping of a semantic class in the training set to a class of similar objects in the test set is created by using Self-Organizing Maps (SOMs) trained from the low-level descriptors. Experiments with two databases and different textual features show promising results, indicating the usefulness of the approach in bridging the gap from low-level visual features to semantic concepts.

The paper examines Marcello Barbieri’s (2007) Introduction to Biosemiotics. Highlighting debate within the biosemiotic community, it focuses on what the volume offers to those who explain human intellect in relation to what Turing called our ‘physical powers.’ In scrutinising the basis of world-modelling, parallels and contrasts are drawn with other work on embodied-embedded cognition. Models dominate biology. Is this a qualitative fact or does it point to biomechanisms? In evaluating the 18 contributions, it is suggested that the answers will shape the field. First, they will decide if biochemistry and explanatory reduction can be synergised by biosemantics. Second, they will show if our intellectual powers arise from biology. Does thinking use—not a language faculty—but what Markos and colleagues call semiosis by the living? Resolution of such issues, it is suggested, can change how we view cognition. Above all, if the biomechanists win the day, cultural models can be regarded as extending natural meaning. On such a view, biomechanisms prompt us to act and perceive as we model our own natural models. This fits Craik’s vision: intellect gives us the alphanumerical ‘symbols’ that allow thoughts to have objective validity. For the biomechanist, this is explained—not by brains alone—but, rather, by acting under the constraints of historically extended sensoria.

This paper investigates the prospects of Rodney Brooks' proposal for AI without representation. It turns out that the supposedly characteristic features of "new AI" (embodiment, situatedness, absence of reasoning, and absence of representation) are all present in conventional systems: "New AI" is just like old AI. Brooks proposal boils down to the architectural rejection of central control in intelligent agents--Which, however, turns out to be crucial. Some of more recent cognitive science suggests that we might do well to dispose of the image of intelligent agents as central representation processors. If this paradigm shift is achieved, Brooks' proposal for cognition without representation appears promising for full-blown intelligent agents--Though not for conscious agents.

Recently, studies on learning of word meanings by agents have begun. In these studies, a human shows objects to an agent and utters words such as ``red'' or ``box''. The agent finds out object's feature represented by each spoken word. In our method, firstly, the agent learns probability distribution p(x) and conditional probability distribution p(x|w), where x is an object feature and w is a word. If a word w does not represent a feature x, p(x) and p(x|w) will be almost same distribution because x is independent of w. This fact enables the agent to use distance between p(x) and p(x|w) when inferring which feature the word represents. Previous works also employ similar stochastic approaches to detect the feature. However, such approaches need a lot of examples to learn correct distributions.

In this paper, we extend a symbolic association framework for being able to handle missing elements in multimodal sequences. The general scope of the work is the symbolic associations of object-word mappings as it happens in language development in infants. In other words, two different representations of the same abstract concepts can associate in both directions. This scenario has been long interested in Artificial Intelligence, Psychology, and Neuroscience. In this work, we extend a recent approach for multimodal sequences (visual and audio) to also cope with missing elements in one or both modalities. Our method uses two parallel Long Short-Term Memories (LSTMs) with a learning rule based on EM-algorithm. It aligns both LSTM outputs via Dynamic Time Warping (DTW). We propose to include an extra step for the combination with the max operation for exploiting the common elements between both sequences. The motivation behind is that the combination acts as a condition selector for choosing the best representation from both LSTMs. We evaluated the proposed extension in the following scenarios: missing elements in one modality (visual or audio) and missing elements in both modalities (visual and sound). The performance of our extension reaches better results than the original model and similar results to individual LSTM trained in each modality., Under review on Journal of Artificial Intelligence Research (JAIR) -- Special Track on Deep Learning, Knowledge Representation, and Reasoning

Natural language interfaces are powerful tools that enables humans and robots to convey information without the need for extensive training or complex graphical interfaces. Statistical techniques that employ probabilistic graphical models have proven effective at interpreting symbols that represent commands and observations for robot direction-following and object manipulation. A limitation of these approaches is their inefficiency in dealing with larger and more complex symbolic representations. Herein, we present a model for language understanding that uses parse trees and environment models both to learn the structure of probabilistic graphical models and to perform inference over this learned structure for symbol grounding. This model, called the Hierarchical Distributed Correspondence Graph (HDCG), exploits information about symbols that are expressed in the corpus to construct minimalist graphical models that are more efficient to search. In a series of comparative experiments, we demonstrate a significant improvement in efficiency without loss in accuracy over contemporary approaches for human-robot interaction.

Compositionality is a property of natural language which is of prime importance: It enables humans to form and conceptualize potentially novel and complex ideas, by combining words. On the other hand, the symbol grounding problem examines the way meaning is anchored to entities external to language, such as sensory percepts and sensory-motor routines. In this paper we aim towards the exploration of the intersection of compositionality and symbol grounding. We thus propose a methodology for constructing empirically derived models of grounded meaning, which afford composition of grounded semantics. We illustrate our methodology for the case of adjectival modifiers. Grounded models of adjectively modified and unmodified colors are acquired through a specially designed procedure with 134 participants, and then computational models of the modifiers “dark” and “light” are derived. The generalization ability of these learnt models is quantitatively evaluated, and their usage is demonstrated in a real-world physical humanoid robot. We regard this as an important step towards extending empirical approaches for symbol grounding so that they can accommodate compositionality: a necessary step towards the deep understanding of natural language for situated embodied agents, such as sensor-enabled ambient intelligence and interactive robots.

Solving the visual symbol grounding problem has long been a goal of artificial intelligence. The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images. In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure. Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words. By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies. We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation., Comment: NAACL-HLT 2015 camera ready

We address the symbol grounding problem for robot perception through a data-driven approach to deriving categories from robot sensor data. Unlike model-based approaches, where human intuitive correspondences are sought between sensor readings and features of an environment (corners, doors, etc.), our method learns intrinsic categories (or natural kinds) from the raw data itself. We approximate a manifold underlying sensor data using Isomap nonlinear dimension reduction and apply Bayesian clustering (Gaussian mixture models) with model identification techniques to discover categories (or kinds). We demonstrate our method through the learning of sensory kinds from trials in various indoor and outdoor environments with different sensor modalities. Learned kinds are then used to classify new sensor data (out-of-sample readings). We present results indicating greater consistency in classifying sensor data employing mixture models in nonlinear low-dimensional embeddings. © 2007 Wiley Periodicals, Inc.