Search

Asynchronous action coordination presents a pervasive challenge in Multi-Agent Systems (MAS), which can be represented as a Stackelberg game (SG). However, the scalability of existing Multi-Agent Reinforcement Learning (MARL) methods based on SG is severely constrained by network structures or environmental limitations. To address this issue, we propose the Stackelberg Decision Transformer (STEER), a heuristic approach that resolves the difficulties of hierarchical coordination among agents. STEER efficiently manages decision-making processes in both spatial and temporal contexts by incorporating the hierarchical decision structure of SG, the modeling capability of autoregressive sequence models, and the exploratory learning methodology of MARL. Our research contributes to the development of an effective and adaptable asynchronous action coordination method that can be widely applied to various task types and environmental configurations in MAS. Experimental results demonstrate that our method can converge to Stackelberg equilibrium solutions and outperforms other existing methods in complex scenarios., 11pages, 7papers

Value decomposition methods have gained popularity in the field of cooperative multi-agent reinforcement learning. However, almost all existing methods follow the principle of Individual Global Max (IGM) or its variants, which limits their problem-solving capabilities. To address this, we propose a dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology, that entirely rejects the IGM premise. Each agent consists of an ego policy for action selection and an alter ego value function to solve the credit assignment problem. The value function factorization can ignore the IGM assumption by utilizing an explicit search procedure. On the basis of the above, we also suggest a novel anti-ego exploration mechanism to avoid the algorithm becoming stuck in a local optimum. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks., 20 pages, 17 figures and 4 tables

In high-dimensional time-series analysis, it is essential to have a set of key factors (namely, the style factors) that explain the change of the observed variable. For example, volatility modeling in finance relies on a set of risk factors, and climate change studies in climatology rely on a set of causal factors. The ideal low-dimensional style factors should balance significance (with high explanatory power) and stability (consistent, no significant fluctuations). However, previous supervised and unsupervised feature extraction methods can hardly address the tradeoff. In this paper, we propose Style Miner, a reinforcement learning method to generate style factors. We first formulate the problem as a Constrained Markov Decision Process with explanatory power as the return and stability as the constraint. Then, we design fine-grained immediate rewards and costs and use a Lagrangian heuristic to balance them adaptively. Experiments on real-world financial data sets show that Style Miner outperforms existing learning-based methods by a large margin and achieves a relatively 10% gain in R-squared explanatory power compared to the industry-renowned factors proposed by human experts., Comment: 9 pages, 6 figures

Centralized training with decentralized execution (CTDE) is a widely-used learning paradigm that has achieved significant success in complex tasks. However, partial observability issues and the absence of effectively shared signals between agents often limit its effectiveness in fostering cooperation. While communication can address this challenge, it simultaneously reduces the algorithm's practicality. Drawing inspiration from human team cooperative learning, we propose a novel paradigm that facilitates a gradual shift from explicit communication to tacit cooperation. In the initial training stage, we promote cooperation by sharing relevant information among agents and concurrently reconstructing this information using each agent's local trajectory. We then combine the explicitly communicated information with the reconstructed information to obtain mixed information. Throughout the training process, we progressively reduce the proportion of explicitly communicated information, facilitating a seamless transition to fully decentralized execution without communication. Experimental results in various scenarios demonstrate that the performance of our method without communication can approaches or even surpasses that of QMIX and communication-based methods., Comment: 16 pages, 10figures

Spatial information is essential in various fields. How to explicitly model according to the spatial location of agents is also very important for the multi-agent problem, especially when the number of agents is changing and the scale is enormous. Inspired by the point cloud task in computer vision, we propose a spatial information extraction structure for multi-agent reinforcement learning in this paper. Agents can effectively share the neighborhood and global information through a spatially encoder-decoder structure. Our method follows the centralized training with decentralized execution (CTDE) paradigm. In addition, our structure can be applied to various existing mainstream reinforcement learning algorithms with minor modifications and can deal with the problem with a variable number of agents. The experiments in several multi-agent scenarios show that the existing methods can get convincing results by adding our spatially explicit architecture., Comment: 8 pages,6 figures, Accepted by IJCNN2023

In multi-agent reinforcement learning (MARL), self-interested agents attempt to establish equilibrium and achieve coordination depending on game structure. However, existing MARL approaches are mostly bound by the simultaneous actions of all agents in the Markov game (MG) framework, and few works consider the formation of equilibrium strategies via asynchronous action coordination. In view of the advantages of Stackelberg equilibrium (SE) over Nash equilibrium, we construct a spatio-temporal sequential decision-making structure derived from the MG and propose an N-level policy model based on a conditional hypernetwork shared by all agents. This approach allows for asymmetric training with symmetric execution, with each agent responding optimally conditioned on the decisions made by superior agents. Agents can learn heterogeneous SE policies while still maintaining parameter sharing, which leads to reduced cost for learning and storage and enhanced scalability as the number of agents increases. Experiments demonstrate that our method effectively converges to the SE policies in repeated matrix game scenarios, and performs admirably in immensely complex settings including cooperative tasks and mixed tasks., Comment: Accepted as a conference paper to the 32nd International Joint Conference on Artificial Intelligence (IJCAI-23)

The application of deep reinforcement learning in multi-agent systems introduces extra challenges. In a scenario with numerous agents, one of the most important concerns currently being addressed is how to develop sufficient collaboration between diverse agents. To address this problem, we consider the form of agent interaction based on neighborhood and propose a multi-agent reinforcement learning (MARL) algorithm based on the actor-critic method, which can adaptively construct the hypergraph structure representing the agent interaction and further implement effective information extraction and representation learning through hypergraph convolution networks, leading to effective cooperation. Based on different hypergraph generation methods, we present two variants: Actor Hypergraph Convolutional Critic Network (HGAC) and Actor Attention Hypergraph Critic Network (ATT-HGAC). Experiments with different settings demonstrate the advantages of our approach over other existing methods., 12 pages, 6 figures

A growing body of research works has focused on the Offline Reinforcement Learning (RL) paradigm. Data providers share large pre-collected datasets on which others can train high-quality agents without interacting with the environments. Such an offline RL paradigm has demonstrated effectiveness in many critical tasks, including robot control, autonomous driving, etc. A well-trained agent can be regarded as a software system. However, less attention is paid to investigating the security threats to the offline RL system. In this paper, we focus on a critical security threat: backdoor attacks. Given normal observations, an agent implanted with backdoors takes actions leading to high rewards. However, the same agent takes actions that lead to low rewards if the observations are injected with triggers that can activate the backdoor. In this paper, we propose Baffle (Backdoor Attack for Offline Reinforcement Learning) and evaluate how different Offline RL algorithms react to this attack. Our experiments conducted on four tasks and four offline RL algorithms expose a disquieting fact: none of the existing offline RL algorithms is immune to such a backdoor attack. More specifically, Baffle modifies $10\%$ of the datasets for four tasks (3 robotic controls and 1 autonomous driving). Agents trained on the poisoned datasets perform well in normal settings. However, when triggers are presented, the agents' performance decreases drastically by $63.6\%$, $57.8\%$, $60.8\%$ and $44.7\%$ in the four tasks on average. The backdoor still persists after fine-tuning poisoned agents on clean datasets. We further show that the inserted backdoor is also hard to be detected by a popular defensive method. This paper calls attention to developing more effective protection for the open-source offline RL dataset., Comment: 13 pages, 6 figures

Almost all multi-agent reinforcement learning algorithms without communication follow the principle of centralized training with decentralized execution. During centralized training, agents can be guided by the same signals, such as the global state. During decentralized execution, however, agents lack the shared signal. Inspired by viewpoint invariance and contrastive learning, we propose consensus learning for cooperative multi-agent reinforcement learning in this paper. Although based on local observations, different agents can infer the same consensus in discrete space. During decentralized execution, we feed the inferred consensus as an explicit input to the network of agents, thereby developing their spirit of cooperation. Our proposed method can be extended to various multi-agent reinforcement learning algorithms with small model changes. Moreover, we carry out them on some fully cooperative tasks and get convincing results., Comment: 14 pages, 13 figures, 2 tables

Recently, model-based agents have achieved better performance than model-free ones using the same computational budget and training time in single-agent environments. However, due to the complexity of multi-agent systems, it is tough to learn the model of the environment. The significant compounding error may hinder the learning process when model-based methods are applied to multi-agent tasks. This paper proposes an implicit model-based multi-agent reinforcement learning method based on value decomposition methods. Under this method, agents can interact with the learned virtual environment and evaluate the current state value according to imagined future states in the latent space, making agents have the foresight. Our approach can be applied to any multi-agent value decomposition method. The experimental results show that our method improves the sample efficiency in different partially observable Markov decision process domains., Comment: 16 pages, 9 figures, 2 tables

The framework of deep reinforcement learning (DRL) provides a powerful and widely applicable mathematical formalization for sequential decision-making. This paper present a novel DRL framework, termed \emph{$f$-Divergence Reinforcement Learning (FRL)}. In FRL, the policy evaluation and policy improvement phases are simultaneously performed by minimizing the $f$-divergence between the learning policy and sampling policy, which is distinct from conventional DRL algorithms that aim to maximize the expected cumulative rewards. We theoretically prove that minimizing such $f$-divergence can make the learning policy converge to the optimal policy. Besides, we convert the process of training agents in FRL framework to a saddle-point optimization problem with a specific $f$ function through Fenchel conjugate, which forms new methods for policy evaluation and policy improvement. Through mathematical proofs and empirical evaluation, we demonstrate that the FRL framework has two advantages: (1) policy evaluation and policy improvement processes are performed simultaneously and (2) the issues of overestimating value function are naturally alleviated. To evaluate the effectiveness of the FRL framework, we conduct experiments on Atari 2600 video games and show that agents trained in the FRL framework match or surpass the baseline DRL algorithms., 17 pages, 4 figures

As one of the solutions to the decentralized partially observable Markov decision process (Dec-POMDP) problems, the value decomposition method has achieved significant results recently. However, most value decomposition methods require the fully observable state of the environment during training, but this is not feasible in some scenarios where only incomplete and noisy observations can be obtained. Therefore, we propose a novel value decomposition framework, named State Inference for value DEcomposition (SIDE), which eliminates the need to know the global state by simultaneously seeking solutions to the two problems of optimal control and state inference. SIDE can be extended to any value decomposition method to tackle partially observable problems. By comparing with the performance of different algorithms in StarCraft II micromanagement tasks, we verified that though without accessible states, SIDE can infer the current state that contributes to the reinforcement learning process based on past local observations and even achieve superior results to many baselines in some complex scenarios., 12 pages, 7 figures, 2 tables. Submitted version

The collaboration between agents has gradually become an important topic in multi-agent systems. The key is how to efficiently solve the credit assignment problems. This paper introduces MGAN for collaborative multi-agent reinforcement learning, a new algorithm that combines graph convolutional networks and value-decomposition methods. MGAN learns the representation of agents from different perspectives through multiple graph networks, and realizes the proper allocation of attention between all agents. We show the amazing ability of the graph network in representation learning by visualizing the output of the graph network, and therefore improve interpretability for the actions of each agent in the multi-agent system., 12 pages, 4 figures, 2 tables

Recently, some challenging tasks in multi-agent systems have been solved by some hierarchical reinforcement learning methods. Inspired by the intra-level and inter-level coordination in the human nervous system, we propose a novel value decomposition framework HAVEN based on hierarchical reinforcement learning for fully cooperative multi-agent problems. To address the instability arising from the concurrent optimization of policies between various levels and agents, we introduce the dual coordination mechanism of inter-level and inter-agent strategies by designing reward functions in a two-level hierarchy. HAVEN does not require domain knowledge and pre-training, and can be applied to any value decomposition variant. Our method achieves desirable results on different decentralized partially observable Markov decision process domains and outperforms other popular multi-agent hierarchical reinforcement learning algorithms., Comment: 13 pages, 11 figures, 2 tables

Configuration not only determines the material flow pattern but also influences the production cost in a manufacturing system. The diversity of the products and growing demands for system flexibility increases the complexity of the configuration. In this paper, an operation-based approach is proposed to measure the configuration complexity of a manufacturing system. The configuration complexity models of the stations are built with single operation, several operations and their parallel types. Subsequently, an operation-based configuration complexity model of the overall system is used to measure the configuration complexity of a manufacturing system using the information entropy. Then, the relationship of the complexity between operations and stations is quantitatively described. An assembly line example is presented to validate the model. The results show that the proposed complexity measurement may evaluate the configuration complexity of a manufacturing system.

MES is an information system for workshop between the planning system and operation control system. It is the basic technical means to realize the automation and agility of production management. The functions of MES include resource allocation, process management, quality control, material management, etc. MES can achieve the function of information tracking, information analysis. It can also predict what may happen in the production process. BOM can describe the material composition of a product and the relationship between its finished materials. MBOM contains the detailed information of various parts, such as the source, quantity and the difference of using time. In essence, it is the description of product materials. Product materials are the core resource of manufacture and management. MBOM provide important information of inventory preparation and material distribution. The traditional material management mode only focuses on the business of purchase, distribution and storage. The relationship between various materials is not considered. Based on the study of the traditional material management mode, this paper puts forward the MES material management mode based on BOM. The information management system is designed and developed according to this management mode. Through the verification by practical application, the material management mode based on BOM realizes the reasonable allocation of materials. Using BOM can improve the efficiency of MES material management.

We present a novel parametric 3D shape representation, Generalized sum of Gaussians (G-SoG), which is particularly suitable for pose estimation of articulated objects. Compared with the original sum-of-Gaussians (SoG), G-SoG can handle both isotropic and anisotropic Gaussians, leading to a more flexible and adaptable shape representation yet with much fewer anisotropic Gaussians involved. An articulated shape template can be developed by embedding G-SoG in a tree-structured skeleton model to represent an articulated object. We further derive a differentiable similarity function between G-SoG (the template) and SoG (observed data) that can be optimized analytically for efficient pose estimation. The experimental results on a standard human pose estimation dataset show the effectiveness and advantages of G-SoG over the original SoG as well as the promise compared with the recent algorithms that use more complicated shape models., 6 pages, 5 figures

Three-dimensionally ordered macroporous (3DOM) SiOC was successfully fabricated on a cordierite monolith inner wall of a polystyrene (PS) sphere template through a colloidal crystal templating (CCT) method (named as 3DOM SiOC/cordierite). And perovskite-type LaCoO3 (LCO) mixed metal oxides, prepared by a facile sol–gel process, were coated onto the wall of the 3DOM SiOC/cordierite by a simple impregnation-sintering approach (named as LCO/3DOM SiOC/cordierite). The morphological and physicochemical properties studies indicate that the 3DOM structure has excellent connectivity, a large specific surface area, and applicable pore size. And the 3DOM structure plays an important role in preventing aggregation of the targeted catalyst particles, improving the contact condition between catalysts and soot as well as enhancing the reducibility of the LCO. The subsequent investigation on the catalytic performance of the 3DOM samples for diesel soot combustion shows that the 3DOM texture offers the catalyst a lower soot combustion temperature than a conventional catalyst, and the LCO/3DOM SiOC/cordierite reduces the combustion temperature even further while avoiding the rise of back-pressure effectively as well.

A blended autopilot algorithm is developed using generalized predictive control and adaptive non-singular terminal sliding mode control, for dual control missile steered by combination of aerodynamic fins and reaction jets. The blended algorithm considers the reaction jets control design prior due to its inherent control error, and then treats the aerodynamic fin control on the basis of the reaction jets control results. The generalized predictive control method is applied to the reaction jets control, which could make the missile achieve desired performance with a small number of consumed reaction jets. Then a novel adaptive non-singular terminal sliding mode control method is proposed for the aerodynamic control system design, to meet the requirements of robustness and fast response for the control of the missile. The unknown bound of the uncertainties and disturbances is estimated adaptively in the control, which makes the control with better robustness. Using the Lyapunov stability theory, the finite time convergence in both reaching and sliding phases is achieved. Finally, the simulations are conducted on the nonlinear longitudinal missile model with uncertainties and disturbances, and the simulation results demonstrate the effectiveness of the blended autopilot algorithm.

Autonomous landing for unmanned seaplanes safely in severe sea states has been a great challenge for decades. A new autonomous landing system of an unmanned seaplane based on active disturbance rejection control (ADRC) is presented in this paper. Firstly, different landing phases of the unmanned seaplane are analyzed. Then according to the characteristics in the different phases, the autonomous landing system consists of a velocity control subsystem and an attitude control subsystem. The velocity control subsystem consists of a velocity ADRC controller and a throttle switch module, while the attitude control subsystem consists of a pitch angle ADRC controller, an altitude PID controller, a pitch angle switch module and a T-S fuzzy reasoning module. Simulations are performed in calm water and irregular wave. The simulations results show that the proposed control system is capable of making the unmanned seaplane land successfully with satisfactory performance.

Flying boat is a fixed-wing airplane capable of landing and taking off on water, and its performance, evaluated by its sea-keeping ability, strongly depends on sea conditions. A Flying boat with good sea-keeping ability can reduce impacts from waves and runs safely and smoothly under a certain sea state. However, sea-keeping ability is still the key problem that hinders its development and application. In this paper, based on the mathematical model of a flying boat, a simulation system predicting the motions of the flying boat on an irregular wave is built, and a controller deduced by active disturbance rejection control (ADRC) and Tagaki-Sugeno (T-S) fuzzy reasoning is applied to improve the sea-keeping ability. The simulation results show that the flying boat has good performances for tracking the surface of waves, reducing impact from water, and making the flying boat take off on three water conditions safely (calm water, water with regular wave and water with irregular wave respectively).

Copper catalysts prepared using two kinds of supports (BaTiO3 and Fe2O3@BaTiO[Formula: see text] have been tested for soot combustion. Perovskite-type oxide BaTiO3 shows excellent properties as support but has small specific surface area. Fe2O3@BaTiO3 core/shell microspheres, as improved support of BaTiO3 support, were fabricated by a hydrothermal-annealing approach. Fe2O3@BaTiO3 support has large specific surface area, which optimizes the contact condition of the gas-soot-catalyst three-phase reaction. Due to the Cu-Fe2O3@BaTiO3 interaction, the redox properties of copper are improved. Soot combustion is tested under O2 gas flow and NOx/O2 gas flow, and Fe2O3@BaTiO3 supported copper catalyst (Cu/Fe2O3@BaTiO[Formula: see text] all shows higher catalytic activity and CO2 selectivity than BaTiO3 supported copper catalyst (Cu/BaTiO[Formula: see text].

As a special airplane, a flying boat can take off from water and land in water, which makes its controller design more complicated than that of other general airplanes. Based on the mathematical model of a flying boat, two controllers applying classical PID (Proportional Integration Differential) method and a compound control method (combining active-disturbance rejection controller (ADRC) and dynamic inversion (DI) methods) are designed respectively. Then the performances of the two controllers are validated by tracking a speed step signal from 15m/s to 16m/s in clam water, holding the speed and the pitch angle of the flying boat in water with regular waves, and taking off from water with and without waves. The results of the simulations show that the controllers can improve the stability and seakeeping quality of the flying boat.

The researches of flying boats have been carried on for about a hundred years, and there are lots of flying boats used all around the world for different tasks. In this paper, the modeling of a flying boat's longitudinal motion was researched. By analyzing the flying boat's geometrical, kinematical and dynamical characters, its hydrodynamic coefficients were approximated by the semi-empirical method which came from Savitsky's researching results of planing boats motion predication. Based on the form of the flying boat, some corrections to the Savitsky's method were made to get a better approximation. The aerodynamic∗ coefficients of the flying boat were obtained by some wind tunnel tests. Based on these results, the longitudinal nonlinear mathematical model of the flying boat in calm water was built. At last, a comparison of the taking-off process between the simulation results and existing experimental data was made. The results showed that the mathematical model was consistent with the actual flying boat.

We propose an improved Tensorfaces algorithm for multi-view face recognition which integrates multi-linear analysis, manifold learning and statistical clustering in one framework. The training face images from different views are first mapped into a 2-D space by the Locality Preserving Projections (LPP) method where statistical clustering is used to capture the view variability. Then a test image of an unknown view can be projected into this 2-D space, and the two closet views can be identified. We develop a modified tensor decomposition method by incorporating two closest views in the calculation of the identity coefficients. The proposed method is evaluated on a large database of multi-view face images that include the CMU PIE and Weizmann databases. Experimental results show that this method outperforms the original TensorFaces method.

Computer vision and pattern recognition technologies make robots' eyes possible, which has rapidly promoted the development of robotic sciences. Barcode, which is simple but contains much information, has been used in navigation, manufacture and other situations by robots. Reading a barcode based on image analysis is an essential technique for such a robot system. Unfortunately, few existing image analysis methods about it are capable of identifying the codes under uncertain conditions, such as varying illumination, different sizes or orientations. This paper presents a method which can locate barcode region at an uncertain 3-D background through hierarchically extracting the distinct features of it with a high precision even for the broken, aberrant, partly blotted out stripes and other poor qualities of images. The method begins with a barcode quality assessment, in which a set of omnidirectional scan lines (active set lines) and a set of description of the whole image (resembling barcode measure), are defined. According to RBM, a series of corresponding methods are chosen to have a pre-processing process of the image and decode the codes respectively. The experiment results prove that our proposed method has two main characteristics. One is fast to locate barcode (get ROI of barcode). The other is to read barcode under unpredicted conditions robustly.