Search

In recommender systems, reinforcement learning solutions have shown promising results in optimizing the interaction sequence between users and the system over the long-term performance. For practical reasons, the policy's actions are typically designed as recommending a list of items to handle users' frequent and continuous browsing requests more efficiently. In this list-wise recommendation scenario, the user state is updated upon every request in the corresponding MDP formulation. However, this request-level formulation is essentially inconsistent with the user's item-level behavior. In this study, we demonstrate that an item-level optimization approach can better utilize item characteristics and optimize the policy's performance even under the request-level MDP. We support this claim by comparing the performance of standard request-level methods with the proposed item-level actor-critic framework in both simulation and online experiments. Furthermore, we show that a reward-based future decomposition strategy can better express the item-wise future impact and improve the recommendation accuracy in the long term. To achieve a more thorough understanding of the decomposition strategy, we propose a model-based re-weighting framework with adversarial learning that further boost the performance and investigate its correlation with the reward-based strategy., Comment: 12 pages, 8 figures

Recently, reputation-based indirect reciprocity has been widely applied to the study on fairness behavior. Previous works mainly investigate indirect reciprocity by considering compulsory participation. While in reality, individuals may choose voluntary participation according to the opponent's reputation. It is still unclear how such reputation-based voluntary participation influences the evolution of fairness. To address this question, we introduce indirect reciprocity with voluntary participation into the dictator game (DG). We respectively consider good dictators or recipients can voluntarily participate in games when the opponents are assessed as bad. We theoretically calculate the fairness level under all social norms of third-order information. Our findings reveal that several social norms induce the high fairness level in both scenarios. However, more social norms lead to a high fairness level for voluntary participation of recipients, compared with the one of good dictators. The results also hold when the probability of voluntary participation is not low. Our results demonstrate that recipients' voluntary participation is more effective in promoting the emergence of fairness behavior.

Let V be a total valuation ring of a division ring K, Q be the additive group of the rational numbers, Aut(K) be the group of automorphisms of K. Let sigma be a group homomorphism from Q to Aut(K). Let K[Q, sigma ] be the skew group ring of Q over K. In this paper, we classify graded extensions of V in K[Q, sigma] into two types and study the structure of them.

Congestion Control (CC), as the core networking task to efficiently utilize network capacity, received great attention and widely used in various Internet communication applications such as 5G, Internet-of-Things, UAN, and more. Various CC algorithms have been proposed both on network and transport layers such as Active Queue Management (AQM) algorithm and Transmission Control Protocol (TCP) congestion control mechanism. But it is hard to model dynamic AQM/TCP system and cooperate two algorithms to obtain excellent performance under different communication scenarios. In this paper, we explore the performance of multi-agent reinforcement learning-based cross-layer congestion control algorithms and present cooperation performance of two agents, known as MACC (Multi-agent Congestion Control). We implement MACC in NS3. The simulation results show that our scheme outperforms other congestion control combination in terms of throughput and delay, etc. Not only does it proves that networking protocols based on multi-agent deep reinforcement learning is efficient for communication managing, but also verifies that networking area can be used as new playground for machine learning algorithms., Comment: 7 pages, 8 figures

We study pursuit-evasion differential games between a faster pursuer moving in 3D space and an evader moving in a plane. We first extend the well-known Apollonius circle to 3D space, by which we construct the isochron for the considered two players. Then both cases with and without a static target are considered and the corresponding optimal strategies are derived using the concept of isochron. In order to guarantee the optimality of the proposed strategies, the value functions are given and are further proved to be the solution of Hamilton-Jacobi-Isaacs equation. Simulations with comparison between the proposed strategies and other classical strategies are carried out and the results show the optimality of the proposed strategies.

In this paper, the circle formation control problem is addressed for a group of cooperative underactuated fish-like robots involving unknown nonlinear dynamics and disturbances. Based on the reinforcement learning and cognitive consistency theory, we propose a decentralized controller without the knowledge of the dynamics of the fish-like robots. The proposed controller can be transferred from simulation to reality. It is only trained in our established simulation environment, and the trained controller can be deployed to real robots without any manual tuning. Simulation results confirm that the proposed model-free robust formation control method is scalable with respect to the group size of the robots and outperforms other representative RL algorithms. Several experiments in the real world verify the effectiveness of our RL-based approach for circle formation control., Comment: to be published in ICRA2021

Underwater communication is extremely challenging for small underwater robots which typically have stringent power and size constraints. In our previous work, we developed an artificial electrocommunication system which could be an alternative for the communication of small underwater robots. This paper further presents a new electrocommunication system that utilizes Binary Frequency Shift Keying (2FSK) modulation and deep-learning-based demodulation for underwater robots. We first derive an underwater electrocommunication model that covers both the near-field area and a large transition area outside of the near-field area. 2FSK modulation is adopted to improve the anti-interference ability of the electric signal. A deep learning algorithm is used to demodulate the electric signal by the receiver. Simulations and experiments show that with the same testing condition, the new communication system outperforms the previous system in both the communication distance and the data transmitting rate. In specific, the newly developed communication system achieves stable communication within the distance of 10 m at a data transfer rate of 5 Kbps with a power consumption of less than 0.1 W. The substantial increase in communication distance further improves the possibility of electrocommunication in underwater robotics., Comment: IROS2020

For heterogeneous unmanned systems composed of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs), using UAVs serve as eyes to assist UGVs in motion planning is a promising research direction due to the UAVs' vast view scope. However, due to UAVs flight altitude limitations, it may be impossible to observe the global map, and motion planning in the local map is a POMDP (Partially Observable Markov Decision Process) problem. This paper proposes a motion planning algorithm for heterogeneous unmanned system under partial observation from UAV without reconstruction of global maps, which consists of two parts designed for perception and decision-making, respectively. For the perception part, we propose the Grid Map Generation Network (GMGN), which is used to perceive scenes from UAV's perspective and classify the pathways and obstacles. For the decision-making part, we propose the Motion Command Generation Network (MCGN). Due to the addition of memory mechanism, MCGN has planning and reasoning abilities under partial observation from UAVs. We evaluate our proposed algorithm by comparing with baseline algorithms. The results show that our method effectively plans the motion of heterogeneous unmanned systems and achieves a relatively high success rate.

Novel robotic grippers have captured increasing interests recently because of their abilities to adapt to varieties of circumstances and their powerful functionalities. Differing from traditional gripper with mechanical components-made fingers, novel robotic grippers are typically made of novel structures and materials, using a novel manufacturing process. In this paper, a novel robotic gripper with external frame and internal thermoplastic elastomer belt-made net is proposed. The gripper grasps objects using the friction between the net and objects. It has the ability of adaptive gripping through flexible contact surface. Stress simulation has been used to explore the regularity between the normal stress on the net and the deformation of the net. Experiments are conducted on a variety of objects to measure the force needed to reliably grip and hold the object. Test results show that the gripper can successfully grip objects with varying shape, dimensions, and textures. It is promising that the gripper can be used for grasping fragile objects in the industry or out in the field, and also grasping the marine organisms without hurting them., Comment: Accepted by 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Dynamic modeling has been capturing attention for its fundamentality in precise locomotion analyses and control of underwater robots. However, the existing researches have mainly focused on investigating two-dimensional motion of underwater robots, and little attention has been paid to three-dimensional dynamic modeling, which is just what we focus on. In this article, a three-dimensional dynamic model of an active-tail-actuated robotic fish with a barycentre regulating mechanism is built by combining Newton's second law for linear motion and Euler's equation for angular motion. The model parameters are determined by three-dimensional computer-aided design (CAD) software SolidWorks, HyperFlow-based computational fluid dynamics (CFD) simulation, and grey-box model estimation method. Both kinematic experiments with a prototype and numerical simulations are applied to validate the accuracy of the dynamic model mutually. Based on the dynamic model, multiple three-dimensional motions, including rectilinear motion, turning motion, gliding motion, and spiral motion, are analyzed. The experimental and simulation results demonstrate the effectiveness of the proposed model in evaluating the trajectory, attitude, and motion parameters, including the velocity, turning radius, angular velocity, etc., of the robotic fish.

The lateral line enables fish to efficiently sense the surrounding environment, thus assisting flow-related fish behaviours. Inspired by this phenomenon, varieties of artificial lateral line systems (ALLSs) have been developed and applied to underwater robots. This article focuses on using the pressure sensor arrays based on ALLS-measured hydrodynamic pressure variations (HPVs) for estimating the relative state between two adjacent robotic fish with leader-follower formation. The relative states include the relative oscillating frequency, amplitude, and offset of the upstream robotic fish to the downstream robotic fish, the relative vertical distance, the relative yaw angle, the relative pitch angle, and the relative roll angle between the two adjacent robotic fish. Regression model between the ALLS-measured and the mentioned relative states is investigated, and regression model-based relative state estimation is conducted. Specifically, two criteria are proposed firstly to investigate not only the sensitivity of each pressure sensor to the variations of relative state but also the insufficiency and redundancy of the pressure sensors. And thus the pressure sensors used for regression analysis are determined. Then four typical regression methods, including random forest algorithm, support vector regression, back propagation neural network, and multiple linear regression method are used for establishing regression models between the ALLS-measured HPVs and the relative states. Then regression effects of the four methods are compared and discussed. Finally, random forest-based method, which has the best regression effect, is used to estimate relative yaw angle and oscillating amplitude using the ALLS-measured HPVs and exhibits excellent estimation performance. This work contributes to local relative estimation for a group of underwater robots, which has always been a challenge., Comment: Accepted by Bioinspiration & Biomimetics

Any phenomenon in nature is potential to be an inspiration for us to propose new ideas. Lateral line is a typical example which has attracted more interest in recent years. With the aid of lateral line, fish is capable of acquiring fluid information around, which is of great significance for them to survive, communicate and hunt underwater. In this paper, we briefly introduce the morphology and mechanism of the lateral line first. Then we focus on the development of artificial lateral line which typically consists of an array of sensors and can be installed on underwater robots. A series of sensors inspired by the lateral line with different sensing principles have been summarized. And then the applications of artificial lateral line system in hydrodynamic environment sensing and vortices detection, dipole oscillation source detection, and autonomous control of underwater robots have been surveyed. In addition, the existing problems and future foci in the field have been further discussed in detail. The current works and future foci have demonstrated that artificial lateral line has great potentials of research and contributes to the applications of underwater robots., Comment: Accepted by Journal of Bionic Engineering

We study the formation control problem for a group of mobile agents in a plane, in which each agent is modeled as a kinematic point and can only use the local measurements in its local frame. The agents are required to maintain a geometric pattern while keeping a desired distance to a static/moving target. The prescribed formation is a general one which can be any geometric pattern, and the neighboring relationship of the N-agent system only has the requirement of containing a directed spanning tree. To solve the formation control problem, a distributed controller is proposed based on the idea of decoupled design. One merit of the controller is that it only uses each agent's local measurements in its local frame, so that a practical issue that the lack of a global coordinate frame or a common reference direction for real multi-robot systems is successfully solved. Considering another practical issue of real robotic applications that sampled data is desirable instead of continuous-time signals, the sampled-data based controller is developed. Theoretical analysis of the convergence to the desired formation is provided for the multi-agent system under both the continuous-time controller with a static/moving target and the sampled-data based one with a static target. Numerical simulations are given to show the effectiveness and performance of the controllers.

We study the general formation problem for a group of mobile agents in a plane, in which the agents are required to maintain a distribution pattern, as well as to rotate around or remain static relative to a static/moving target. The prescribed distribution pattern is a class of general formations that the distances between neighboring agents or the distances from each agent to the target do not need to be equal. Each agent is modeled as a double integrator and can merely perceive the relative information of the target and its neighbors. A distributed control law is designed using the limit-cycle based idea to solve the problem. One merit of the controller is that it can be implemented by each agent in its Frenet-Serret frame so that only local information is utilized without knowing global information. Theoretical analysis is provided of the equilibrium of the N-agent system and of the convergence of its converging part. Numerical simulations are given to show the effectiveness and performance of the proposed controller.