Search

Large Language Models (LLMs) are often criticized for lacking true "understanding" and the ability to "reason" with their knowledge, being seen merely as autocomplete systems. We believe that this assessment might be missing a nuanced insight. We suggest that LLMs do develop a kind of empirical "understanding" that is "geometry"-like, which seems adequate for a range of applications in NLP, computer vision, coding assistance, etc. However, this "geometric" understanding, built from incomplete and noisy data, makes them unreliable, difficult to generalize, and lacking in inference capabilities and explanations, similar to the challenges faced by heuristics-based expert systems decades ago. To overcome these limitations, we suggest that LLMs should be integrated with an "algebraic" representation of knowledge that includes symbolic AI elements used in expert systems. This integration aims to create large knowledge models (LKMs) that not only possess "deep" knowledge grounded in first principles, but also have the ability to reason and explain, mimicking human expert capabilities. To harness the full potential of generative AI safely and effectively, a paradigm shift is needed from LLM to more comprehensive LKM., Comment: 7 pages, 9 images

The startling success of ChatGPT and other large language models (LLMs) using transformer-based generative neural network architecture in applications such as natural language processing and image synthesis has many researchers excited about potential opportunities in process systems engineering (PSE). The almost human-like performance of LLMs in these areas is indeed very impressive, surprising, and a major breakthrough. Their capabilities are very useful in certain tasks, such as writing first drafts of documents, code writing assistance, text summarization, etc. However, their success is limited in highly scientific domains as they cannot yet reason, plan, or explain due to their lack of in-depth domain knowledge. This is a problem in domains such as chemical engineering as they are governed by fundamental laws of physics and chemistry (and biology), constitutive relations, and highly technical knowledge about materials, processes, and systems. Although purely data-driven machine learning has its immediate uses, the long-term success of AI in scientific and engineering domains would depend on developing hybrid AI systems that use first principles and technical knowledge effectively. We call these hybrid AI systems Large Knowledge Models (LKMs), as they will not be limited to only NLP-based techniques or NLP-like applications. In this paper, we discuss the challenges and opportunities in developing such systems in chemical engineering.

The motility-induced phase separation (MIPS) phenomenon in active matter has been of great interest for the past decade or so. A central conceptual puzzle is that this behavior, which is generally characterized as a nonequilibrium phenomenon, can yet be explained using simple equilibrium models of thermodynamics. Here, we address this problem using a new theory, statistical teleodynamics, which is a conceptual synthesis of game theory and statistical mechanics. In this framework, active agents compete in their pursuit of maximum effective utility, and this self-organizing dynamics results in an arbitrage equilibrium in which all agents have the same effective utility. We show that MIPS is an example of arbitrage equilibrium and that it is mathematically equivalent to other phase-separation phenomena in entirely different domains, such as sociology and economics. As examples, we present the behavior of Janus particles in a potential trap and the effect of chemotaxis on MIPS., Comment: arXiv admin note: substantial text overlap with arXiv:2312.05765

Despite the stunning progress recently in large-scale deep neural network applications, our understanding of their microstructure, 'energy' functions, and optimal design remains incomplete. Here, we present a new game-theoretic framework, called statistical teleodynamics, that reveals important insights into these key properties. The optimally robust design of such networks inherently involves computational benefit-cost trade-offs that are not adequately captured by physics-inspired models. These trade-offs occur as neurons and connections compete to increase their effective utilities under resource constraints during training. In a fully trained network, this results in a state of arbitrage equilibrium, where all neurons in a given layer have the same effective utility, and all connections to a given layer have the same effective utility. The equilibrium is characterized by the emergence of two lognormal distributions of connection weights and neuronal output as the universal microstructure of large deep neural networks. We call such a network the Jaynes Machine. Our theoretical predictions are shown to be supported by empirical data from seven large-scale deep neural networks. We also show that the Hopfield network and the Boltzmann Machine are the same special case of the Jaynes Machine., Comment: arXiv admin note: text overlap with arXiv:2310.06960

Contrary to the widely believed hypothesis that larger, denser cities promote socioeconomic mixing, a recent study (Nilforoshan et al. 2023) reports the opposite behavior, i.e. more segregation. Here, we present a game-theoretic model that predicts such a density-dependent segregation outcome in both one- and two-class systems. The model provides key insights into the analytical conditions that lead to such behavior. Furthermore, the arbitrage equilibrium outcome implies the equality of effective utilities among all agents. This could be interpreted as all agents being equally "happy" in their respective environments in our ideal society. We believe that our model contributes towards a deeper mathematical understanding of social dynamics and behavior, which is important as we strive to develop more harmonious societies., Comment: arXiv admin note: text overlap with arXiv:2312.05765

Patterns in ecological systems such as mussel beds have been of considerable interest for a long time. Several physicochemical mechanisms have been proposed for their formation. Here, we propose a novel framework based on economics and game theory. Since mussels are biological agents instinctively driven by the survival purpose, we mathematically model this purpose explicitly using a new theoretical framework called statistical teleodynamics. We show both analytically and computationally that when every mussel pursues its own self-interest to survive, a stable collective order emerges spontaneously via self-organization. Thus, our mechanism is essentially the same as Adam Smith's invisible hand in a biological context. Our analysis reveals a new insight that the mussel bed patterns could be the result of arbitrage equilibrium in the competition for survival among the mussels.

We present a novel theory of the microstructure of deep neural networks. Using a theoretical framework called statistical teleodynamics, which is a conceptual synthesis of statistical thermodynamics and potential game theory, we predict that all highly connected layers of deep neural networks have a universal microstructure of connection strengths that is distributed lognormally ($LN({\mu}, {\sigma})$). Furthermore, under ideal conditions, the theory predicts that ${\mu}$ and ${\sigma}$ are the same for all layers in all networks. This is shown to be the result of an arbitrage equilibrium where all connections compete and contribute the same effective utility towards the minimization of the overall loss function. These surprising predictions are shown to be supported by empirical data from six large-scale deep neural networks in real life. We also discuss how these results can be exploited to reduce the amount of data, time, and computational resources needed to train large deep neural networks.

Various template-based and template-free approaches have been proposed for single-step retrosynthesis prediction in recent years. While these approaches demonstrate strong performance from a data-driven metrics standpoint, many model architectures do not incorporate underlying chemistry principles. Here, we propose a novel chemistry-aware retrosynthesis prediction framework that combines powerful data-driven models with prior domain knowledge. We present a tree-to-sequence transformer architecture that utilizes hierarchical SMILES grammar-based trees, incorporating crucial chemistry information that is often overlooked by SMILES text-based representations, such as local structures and functional groups. The proposed framework, grammar-based molecular attention tree transformer (G-MATT), achieves significant performance improvements compared to baseline retrosynthesis models. G-MATT achieves a promising top-1 accuracy of 51% (top-10 accuracy of 79.1%), invalid rate of 1.5%, and bioactive similarity rate of 74.8% on the USPTO- 50K dataset. Additional analyses of G-MATT attention maps demonstrate the ability to retain chemistry knowledge without relying on excessively complex model architectures.

Rapid discovery of new reactions and molecules in recent years has been facilitated by the advancements in high throughput screening, accessibility to a much more complex chemical design space, and the development of accurate molecular modeling frameworks. A holistic study of the growing chemistry literature is, therefore, required that focuses on understanding the recent trends and extrapolating them into possible future trajectories. To this end, several network theory-based studies have been reported that use a directed graph representation of chemical reactions. Here, we perform a study based on representing chemical reactions as hypergraphs where the hyperedges represent chemical reactions and nodes represent the participating molecules. We use a standard reactions dataset to construct a hypernetwork and report its statistics such as degree distributions, average path length, assortativity or degree correlations, PageRank centrality, and graph-based clusters (or communities). We also compute each statistic for an equivalent directed graph representation of reactions to draw parallels and highlight differences between the two. To demonstrate the AI applicability of hypergraph reaction representation, we generate dense hypergraph embeddings and use them in the reaction classification problem. We conclude that the hypernetwork representation is flexible, preserves reaction context, and uncovers hidden insights that are otherwise not apparent in a traditional directed graph representation of chemical reactions.

The physics of active biological matter, such as bacterial colonies and bird flocks, exhibiting interesting self-organizing dynamical behavior has gained considerable importance in recent years. Recent theoretical advances use techniques from hydrodynamics, kinetic theory, and non-equilibrium statistical physics. However, for biological agents, these don't seem to recognize explicitly their critical feature, namely, the role of survival-driven purpose and the attendant pursuit of maximum utility. Here, we propose a novel game-theoretic framework and show a surprising result that the bird-like agents, garuds, self-organize dynamically into flocks to approach a stable arbitrage equilibrium of equal effective utilities. While it has been well-known for three centuries that there are constants of motion for passive matter, it comes as a surprise to discover that the dynamics of active matter populations could also have an invariant. This is essentially the invisible hand mechanism of Adam Smith's in a biological context. What we demonstrate is for ideal systems, similar to the ideal gas or Ising model in thermodynamics. The next steps would involve examining and learning how real swarms behave compared to their ideal versions. Our theory is not limited to just birds flocking but can be adapted for the self-organizing dynamics of other active matter systems., Comment: New discussion points have been added

Recent attempts towards a theory of active matter utilize concepts and methods from hydrodynamics, kinetic theory, and non-equilibrium statistical physics. However, such approaches typically do not seem to recognize the critical feature of some kinds of active matter (particularly the biological ones), namely, the role of purpose, and the naturally attendant concept of the pursuit of maximum utility, which we believe is the crucial difference between active and passive matter. Here we introduce a novel game-theoretic framework, statistical teleodynamics, that accounts for this feature explicitly and show how it can be integrated with conventional statistical mechanics to develop a unified theory of arbitrage equilibrium in active and passive matter. We propose a spectrum of self-actualizing capabilities, going from none to completely strategic decision-making, and envision the various examples of active matter systems occupying someplace in this spectrum. We show how statistical teleodynamics reduces to familiar results in statistical mechanics in the limit of zero self-actualization. At the other extreme, in an economic setting, it provides novel insights about the emergence of income distributions and their fairness in an ideal free-market society. As examples of agents in between these limits, we show how our theory predicts the behavior of active Brownian particles, the emergence of ant craters, and phase equilibria in social segregation dynamics. We suggest that our theory offers a novel systems theoretic perspective of emergent phenomena that could serve as the starting point for a more comprehensive theory of design, control, and optimization through self-organization.

Swarm Intelligence-based optimization techniques combine systematic exploration of the search space with information available from neighbors and rely strongly on communication among agents. These algorithms are typically employed to solve problems where the function landscape is not adequately known and there are multiple local optima that could result in premature convergence for other algorithms. Applications of such algorithms can be found in communication systems involving design of networks for efficient information dissemination to a target group, targeted drug-delivery where drug molecules search for the affected site before diffusing, and high-value target localization with a network of drones. In several of such applications, the agents face a hostile environment that can result in loss of agents during the search. Such a loss changes the communication topology of the agents and hence the information available to agents, ultimately influencing the performance of the algorithm. In this paper, we present a study of the impact of loss of agents on the performance of such algorithms as a function of the initial network configuration. We use particle swarm optimization to optimize an objective function with multiple sub-optimal regions in a hostile environment and study its performance for a range of network topologies with loss of agents. The results reveal interesting trade-offs between efficiency, robustness, and performance for different topologies that are subsequently leveraged to discover general properties of networks that maximize performance. Moreover, networks with small-world properties are seen to maximize performance under hostile conditions.

The question about fair income inequality has been an important open question in economics and in political philosophy for over two centuries with only qualitative answers such as the ones suggested by Rawls, Nozick, and Dworkin. We provided a quantitative answer recently, for an ideal free-market society, by developing a game-theoretic framework that proved that the ideal inequality is a lognormal distribution of income at equilibrium. In this paper, we develop another approach, using the Nash Bargaining Solution (NBS) framework, which also leads to the same conclusion. Even though the conclusion is the same, the new approach, however, reveals the true nature of NBS, which has been of considerable interest for several decades. Economists have wondered about the economic meaning or purpose of the NBS. While some have alluded to its fairness property, we show more conclusively that it is all about fairness. Since the essence of entropy is also fairness, we see an interesting connection between the Nash product and entropy for a large population of rational economic agents.

It is very common to observe crowds of individuals solving similar problems with similar information in a largely independent manner. We argue here that crowds can become "smarter," i.e., more efficient and robust, by partially following the average opinion. This observation runs counter to the widely accepted claim that the wisdom of crowds deteriorates with social influence. The key difference is that individuals are self-interested and hence will reject feedbacks that do not improve their performance. We propose a control-theoretic methodology to compute the degree of social influence, i.e., the level to which one accepts the population feedback, that optimizes performance. We conducted an experiment with human subjects ($N = 194$), where the participants were first asked to solve an optimization problem independently, i.e., under no social influence. Our theoretical methodology estimates a $30\%$ degree of social influence to be optimal, resulting in a $29\%$ improvement in the crowd's performance. We then let the same cohort solve a new problem and have access to the average opinion. Surprisingly, we find the average degree of social influence in the cohort to be $32\%$ with a $29\%$ improvement in performance: In other words, the crowd self-organized into a near-optimal setting. We believe this new paradigm for making crowds "smarter" has the potential for making a significant impact on a diverse set of fields including population health to government planning. We include a case study to show how a crowd of states can collectively learn the level of taxation and expenditure that optimizes economic growth., Comment: Venkat Venkatasubramanian is the corresponding author. Email: venkat@columbia.edu

The widening inequality in income distribution in recent years, and the associated excessive pay packages of CEOs in the U.S. and elsewhere, is of growing concern among policy makers as well as the common person. However, there seems to be no satisfactory answer, in conventional economic theories and models, to the fundamental question of what kind of pay distribution we ought to see, at least under ideal conditions, in a free market environment and whether this distribution is fair. We propose a game theoretic framework that addresses these questions and show that the lognormal distribution is the fairest inequality of pay in an organization comprising of homogenous agents, achieved at equilibrium, under ideal free market conditions. We also show that for a population of two different classes of agents, the final distribution is a combination of two different lognormal distributions where one of them, corresponding to the top 3-5% of the population, can be misidentified as a Pareto distribution. Our theory also shows the deep and direct connection between potential game theory and statistical mechanics through entropy, which is a measure of fairness in a distribution. This leads us to propose the fair market hypothesis, that the self-organizing dynamics of the ideal free market, i.e., Adam Smith's "invisible hand", not only promotes efficiency but also maximizes fairness under the given constraints., Comment: Corresponding author: Venkat@columbia.edu

The excessive compensation packages of CEOs of U.S. corporations in recent years have brought to the foreground the issue of fairness in economics. The conventional wisdom is that the free market for labor, which determines the pay packages, cares only about efficiency and not fairness. We present an alternative theory that shows that an ideal free market environment also promotes fairness, as an emergent property resulting from the self-organizing market dynamics. Even though an individual employee may care only about his or her salary and no one else's, the collective actions of all the employees, combined with the profit maximizing actions of all the companies, in a free market environment under budgetary constraints, lead towards a more fair allocation of wages, guided by Adam Smith's invisible hand of self-organization. By exploring deep connections with statistical thermodynamics, we show that entropy is the appropriate measure of fairness in a free market environment which is maximized at equilibrium to yield the lognormal distribution of salaries as the fairest inequality of pay in an organization under ideal conditions.

Topology is a fundamental part of a network that governs connectivity between nodes, the amount of data flow and the efficiency of data flow between nodes. In traditional networks, due to physical limitations, topology remains static for the course of the network operation. Ubiquitous data networks (UDNs), alternatively, are more adaptive and can be configured for changes in their topology. This flexibility in controlling their topology makes them very appealing and an attractive medium for supporting "anywhere, any place" communication. However, it raises the problem of designing a dynamic topology. The dynamic topology design problem is of particular interest to application service providers who need to provide cost-effective data services on a ubiquitous network. In this paper we describe algorithms that decide when and how the topology should be reconfigured in response to a change in the data communication requirements of the network. In particular, we describe and compare a greedy algorithm, which is often used for topology reconfiguration, with a non-greedy algorithm based on metrical task systems. Experiments show the algorithm based on metrical task system has comparable performance to the greedy algorithm at a much lower reconfiguration cost., Comment: 12pages, 4 figures, 2 algorithms, Workshop for Ubiquitous Data Management

The high pay packages of U.S. CEOs have raised serious concerns about what would constitute a fair pay., Comment: 16 pages

We present a general holistic theory for the organization of complex networks, both human-engineered and naturally-evolved. Introducing concepts of value of interactions and satisfaction as generic network performance measures, we show that the underlying organizing principle is to meet an overall performance target for wide-ranging operating or environmental conditions. This design or survival requirement of reliable performance under uncertainty leads, via the maximum entropy principle, to the emergence of a power law vertex degree distribution. The theory also predicts exponential or Poisson degree distributions depending on network redundancy, thus explaining all three regimes as different manifestations of a common underlying phenomenon within a unified theoretical framework., Comment: CIPAC Tech Report 2004-1 delivered as a seminar in Feb 2004 in the LIPS Seminar Series, School of Chemical Engineering, Purdue University

An important feature of many complex systems, both natural and artificial, is the structure and organization of their interaction networks with interesting properties. Here we present a theory of self-organization by evolutionary adaptation in which we show how the structure and organization of a network is related to the survival, or in general the performance, objectives of the system. We propose that a complex system optimizes its network structure in order to maximize its overall survival fitness which is composed of short-term and long-term survival components. These in turn depend on three critical measures of the network, namely, efficiency, robustness and cost, and the environmental selection pressure. Using a graph theoretical case study, we show that when efficiency is paramount the "Star" topology emerges and when robustness is important the "Circle" topology is found. When efficiency and robustness requirements are both important to varying degrees, other classes of networks such as the "Hub" emerge. Our assumptions and results are consistent with observations across a wide variety of applications., Comment: In press, Computers and Chemical Engg., 2004, 21 pages, 12 figures, 1 table