Search

Maintaining the orderliness and efficiency of pedestrian flow through an architectural area is critical for the evacuation process. Especially, clogs and jams are easily triggered in width-changing areas. In this article, we consider pedestrian movement in heterogeneous corridors and design an optimal feedback control to regulate pedestrian flow. Flow characteristics are first studied based on microscopic social-force simulations. A Gaussian process describes the relationship between flow variables with the observation data. The macroscopic model for flow in heterogeneous corridors is developed. To avoid jams, discharges among these corridors are balanced with the narrowest corridor as the primary concern. At the equilibrium, a continuous-time nonlinear control system is formulated, and the adaptive dynamic programming learns the optimal feedback controller. Policy iteration (PI) and neural networks are combined together, and the convergence of neural-network-based PI is demonstrated by analyzing its equivalence to the Gauss–Newton method. Batch normalization is introduced to stabilize the learning process. Simulated experiments demonstrate that the control design can effectively regulate pedestrian flow for both macroscopic and microscopic models. Note to Practitioners—The development of video-processing techniques provides a powerful tool to detect human behavior in real time. In crowd events, the pedestrian movement must be regulated; otherwise, it is easy to fall into the faster-is-slower effect. It is especially important for evacuation routes with different widths. In this article, the optimal feedback control is studied to regulate pedestrian flow in heterogeneous corridors. It takes flow densities as state and produces commands that are composed of entrance influx and free-flow velocities. These commands can be executed with the support of speakers, displays, or the recently developed interactive robots. To avoid congestion, discharges of different corridors are balanced, and the system is optimally stabilized at equilibrium. Based on our work, engineers are able to design pedestrian flow control and achieve optimal evacuation in arbitrary heterogeneous corridors. [ABSTRACT FROM AUTHOR]

For systems that can only be locally stabilized, control laws and their effective regions are both important. In this paper, invariant policy iteration is proposed to solve the optimal control of discrete-time systems. At each iteration, a given policy is evaluated in its invariantly admissible region, and a new policy and a new region are updated for the next iteration. Theoretical analysis shows the method is regionally convergent to the optimal value and the optimal policy. Combined with sum-of-squares polynomials, the method is able to achieve the near-optimal control of a class of discrete-time systems. An invariant adaptive dynamic programming algorithm is developed to extend the method to scenarios where system dynamics is not available. Online data are utilized to learn the near-optimal policy and the invariantly admissible region. Simulated experiments verify the effectiveness of our method. [ABSTRACT FROM AUTHOR]

When an energy storage system comprises multiple batteries, the optimal scheduling of charging/discharging actions must take into account their different characteristics. To prolong battery lifetime, each battery must stay in their safety zones. Since load demand and energy price vary over time, the value function that reflects future power cost is both state- and time-dependent. To address this control-limited optimization problem, a new adaptive dynamic programming algorithm is proposed. The time-varying optimal value function is subdivided into a sequence of time-invariant functions which are solved by a periodic value iteration. These functions are further approximated by fuzzy systems, and the optimal charging/discharging actions are searched by the projected Newton method. Experiment results show the new algorithm is able to coordinate multiple batteries efficiently in their constrained control space. [ABSTRACT FROM AUTHOR]

The adaptive cruise control (ACC) problem can be transformed to an optimal tracking control problem for complex nonlinear systems. In this paper, a novel highly efficient model-free adaptive dynamic programming (ADP) approach with experience replay technology is proposed to design the ACC controller. Experience replay increases the data efficiency by recording the available driving data and repeatedly presenting them to the learning procedure of the acceleration controller in the ACC system. The learning framework that combines ADP with experience replay is described in detail. The distinguishing feature of the algorithm is that when estimating parameters of the critic network and the actor network with gradient rules, the gradients of historical data and current data are used to update parameters concurrently. It is proved with Lyapunov theory that the weight estimation errors of the actor network and the critic network are uniformly ultimately bounded under the novel weight update rules. The learning performance of the ACC controller implemented by this ADP algorithm is clearly demonstrated that experience replay can increase data efficiency significantly, and the approximate optimality and adaptability of the learned control policy are tested with typical driving scenarios. [ABSTRACT FROM AUTHOR]

Online learning is an important property of adaptive dynamic programming (ADP). Online observations contain plentiful dynamics information, and ADP algorithms can utilize them to learn the optimal control policy. This paper reviews the research of online ADP algorithms for the optimal control of continuous-time systems. With the intensive study, ADP has been developed towards model free and data efficient. After separately introducing the algorithms, we compare their performance on the same problem. This paper is desired to provide a comprehensive understanding of continuous-time online ADP algorithms. [ABSTRACT FROM AUTHOR]

In this paper, the fully cooperative game with partially constrained inputs in the continuous-time Markov decision process environment is investigated using a novel data-driven adaptive dynamic programming method. First, the model-based policy iteration algorithm with one iteration loop is proposed, where the knowledge of system dynamics is required. Then, it is proved that the iteration sequences of value functions and control policies can converge to the optimal ones. In order to relax the exact knowledge of the system dynamics, a model-free iterative equation is derived based on the model-based algorithm and the integral reinforcement learning. Furthermore, a data-driven adaptive dynamic programming is developed to solve the model-free equation using generated system data. From the theoretical analysis, we prove that this model-free iterative equation is equivalent to the model-based iterative equations, which means that the data-driven algorithm can approach the optimal value function and control policies. For the implementation purpose, three neural networks are constructed to approximate the solution of the model-free iteration equation using the off-policy learning scheme after the available system data is collected in the online measurement phase. Finally, two examples are provided to demonstrate the effectiveness of the proposed scheme. [ABSTRACT FROM AUTHOR]

Event-triggered control has been an effective tool in dealing with problems with finite communication and computation resources. In this paper, we design an event-triggered control for nonlinear constrained-input continuous-time systems based on the optimal policy. Constraints on controls are handled using a bounded function. To learn the optimal solution with partially unknown dynamics, an online adaptive dynamic programming algorithm is proposed. The identifier network, the critic network, and the actor network are employed to approximate the unknown drift dynamics, the optimal value, and the optimal policy, respectively. The identifier is tuned based on online data, which further trains the critic and actor at triggering instants. A concurrent learning technique repeatedly uses past data to train the critic. Stability of the closed-loop system, and convergence of neural networks to the optimal solutions are proved by Lyapunov analysis. In the end, the algorithm is applied to the overhead crane system to observe the performance. The event-triggered optimal controller with constraints stabilizes the system and consumes much less sampling times. [ABSTRACT FROM AUTHOR]

In this paper, a novel nonlinear learning controller called fuzzy-based goal representation adaptive dynamic programming (Fuzzy-GrADP) is proposed. In the proposed GrADP method, a goal representation network is introduced to generate an adaptive internal reinforcement signal to the critic network to help the controller provide a general mapping between the input and output actions. Moreover, in the proposed architecture, the action network in the GrADP is improved by using the fuzzy hyperbolic model, which combines the merits of the fuzzy model and the neural network model. Based on the back-propagation technique, the parameters in the membership functions and the fuzzy rules are all undergo training and online adapting. The proposed controller is tested on two numerical benchmarks, and the simulation results show that the proposed controller outperforms the original adaptive dynamic fuzzy controller and the pure neural network-based GrADP controller. In addition, the proposed controller is further applied on a large multimachine power system for static var compensator damping control, where simulation results demonstrate the effectiveness of the proposed approach on real applications. Furthermore, in order to demonstrate the theoretical guarantee of the proposed method, Lyapunov stability analysis to support the proposed Fuzzy-GrADP approach has also been carried out. [ABSTRACT FROM PUBLISHER]

The optimal tracking of non‐linear systems without knowing system dynamics is an important and intractable problem. Based on the framework of reinforcement learning (RL) and adaptive dynamic programming, a model‐free adaptive optimal tracking algorithm is proposed in this study. After constructing an augmented system with the tracking errors and the reference states, the tracking problem is converted to a regulation problem with respect to the new system. Several RL techniques are synthesised to form a novel algorithm which learns the optimal solution online in real time without any information of the system dynamics. Continuous adaptation laws are defined by the current observations and the past experience. The convergence is guaranteed by Lyapunov analysis. Two simulations on a linear and a non‐linear systems demonstrate the performance of the proposed approach. [ABSTRACT FROM AUTHOR]

In this paper, a self-learning control scheme is proposed for the infinite horizon optimal control of affine nonlinear systems based on the action dependent heuristic dynamic programming algorithm. The policy iteration technique is introduced to derive the optimal control policy with feasibility and convergence analysis. It shows that the “greedy” control action for each state is uniquely existent, the learned control policy after each policy iteration is admissible, and the optimal control policy is able to be obtained. Two three-layer perceptron neural networks are employed to implement the scheme. The critic network is trained by a novel rule to conform to the Bellman equation, and the action network is trained to yield a better control policy. Both training processes alternate until the optimal control policy is achieved. Two simulation examples are provided to validate the effectiveness of the approach. [ABSTRACT FROM AUTHOR]

Abstract: The paper proposes a novel iterative control scheme based on neural networks for optimally controlling a large class of nonlinear discrete-time systems affected by an unknown time variant delay and system uncertainties. An iterative Dual Heuristic dynamic Programming (DHP) algorithm has been envisaged to design the controller which is proven to converge to the optimal one. The key elements required by the DHP, namely the performance index function, the optimal control policy and the nonlinear delay-affected discrete-time system are modeled with feedforward neural networks. Examples demonstrate the validity of the proposed optimal control approach and its effectiveness in dealing with nonlinear time delay situations. [Copyright &y& Elsevier]

Abstract: The paper proposes a supervised adaptive dynamic programming (SADP) algorithm for a full-range adaptive cruise control (ACC) system, which can be formulated as a dynamic programming problem with stochastic demands. The suggested ACC system has been designed to allow the host vehicle to drive both in highways and in Stop and Go (SG) urban scenarios. The ACC system can autonomously drive the host vehicle to a desired speed and/or a given distance from the target vehicle in both operational cases. Traditional adaptive dynamic programming (ADP) is a suitable tool to address the problem but training usually suffers from low convergence rates and hardly achieves an effective controller. A SADP algorithm which introduces the concept of inducing region is here introduced to overcome such training drawbacks. The SADP algorithm performs very well in all simulation scenarios and always better than more traditional controllers. The conclusion is that the proposed SADP algorithm is an effective control methodology able to effectively address the full-range ACC problem. [Copyright &y& Elsevier]

In this paper, a novel neural-network-based iterative adaptive dynamic programming (ADP) algorithm is proposed. It aims at solving the optimal control problem of a class of nonlinear discrete-time systems with control constraints. By introducing a generalized nonquadratic functional, the iterative ADP algorithm through globalized dual heuristic programming technique is developed to design optimal controller with convergence analysis. Three neural networks are constructed as parametric structures to facilitate the implementation of the iterative algorithm. They are used for approximating at each iteration the cost function, the optimal control law, and the controlled nonlinear discrete-time system, respectively. A simulation example is also provided to verify the effectiveness of the control scheme in solving the constrained optimal control problem. [ABSTRACT FROM AUTHOR]

Abstract: In this paper adaptive dynamic programming (ADP) is applied to learn to play Gomoku. The critic network is used to evaluate board situations. The basic idea is to penalize the last move taken by the loser and reward the last move selected by the winner at the end of a game. The results show that the presented program is able to improve its performance by playing against itself and has approached the candidate level of a commercial Gomoku program called 5-star Gomoku. We also examined the influence of two methods for generating games: self-teaching and learning through watching two experts playing against each other and presented the comparison results and reasons. [Copyright &y& Elsevier]

Abstract: An intelligent-optimal control scheme for unknown nonaffine nonlinear discrete-time systems with discount factor in the cost function is developed in this paper. The iterative adaptive dynamic programming algorithm is introduced to solve the optimal control problem with convergence analysis. Then, the implementation of the iterative algorithm via globalized dual heuristic programming technique is presented by using three neural networks, which will approximate at each iteration the cost function, the control law, and the unknown nonlinear system, respectively. In addition, two simulation examples are provided to verify the effectiveness of the developed optimal control approach. [Copyright &y& Elsevier]