Distributed Adaptive Control: Beyond Single-Instant, Discrete Control Variables

In extensive form noncooperative game theory, at each instant t, each agent i sets its state x i independently of the other agents, by sampling an associated distribution, q i(x i). The coupling between the agents arises in the joint evolution of those distributions. Distributed control problems can be cast the same way. In those problems the system designer sets aspects of the joint evolution of the distributions to try to optimize the goal for the overall system. Now information theory tells us what the separate q i of the agents are most likely to be if the system were to have a particular expected value of the objective function G(x 1, x 2, ...). So one can view the job of the system designer as speeding an iterative process. Each step of that process starts with a specified value of E(G), and the convergence of the q i to the most likely set of distributions consistent with that value. After this the target value for E q(G) is lowered, and then the process repeats. Previous work has elaborated many schemes for implementing this process when the underlying variables x i all have a finite number of possible values and G does not extend to multiple instants in time. That work also is based on a fixed mapping from agents to control devices, so that the the statistical independence of the agents’ moves means independence of the device states. This paper also extends that work to relax all of these restrictions. This extends the applicability of that work to include continuous spaces and Reinforcement Learning. This paper also elaborates how some of that earlier work can be viewed as a first-principles justification of evolution-based search algorithms.