Search

Many CPG-based locomotion models have a problem known as the tracking error problem, where the mismatch between the CPG driving signal and the state of the robot can cause undesirable behaviours for legged robots. Towards alleviating this problem, we introduce a mechanism that modulates the CPG signal using the robot’s interoceptive information. The key concept is to generate a driving signal that is easier for the robot to follow, yet can drive the locomotion of the robot. This can be done by nudging the CPG signal in the direction of lower tracking error, which can be analytically calculated. Unlike other reactive CPG, the proposed method does not rely on any parametric learning ability to adjust the shape of the signal, making it a unique option for a biological adaptive motor control. Our experiment results show that the proposed method successfully reduces the tracking error. We also show that the CPG signal, regulated by the proposed method, is robust to perturbation and can smoothly return back to the default pattern.

Overfitting to learning partners is a known problem, in multi-agent reinforcement learning (MARL), due to the co-evolution of learning agents. Previous works explicitly add diversity to learning partners for mitigating this problem. However, since there are many approaches for introducing diversity, it is not clear which one should be used under what circumstances. In this work, we clarify the situation and reveal that widely used methods such as partner sampling and population-based training are unreliable at introducing diversity under fully cooperative multi-agent Markov decision process. We find that generating pre-trained partners is a simple yet effective procedure to achieve diversity. Finally, we highlight the impact of diversified learning partners on the generalization of learning agents using cross-play and ad-hoc team performance as evaluation metrics.

In most contemporary work in deep reinforcement learning (DRL), agents are trained in simulated environments. Not only are simulated environments fast and inexpensive, they are also ‘safe’. By contrast, training in a real world environment (using robots, for example) is not only slow and costly, but actions can also result in irreversible damage, either to the environment or to the agent (robot) itself. In this paper, we consider taking advantage of the inherent safety in computer simulation by extending the Deep Q-Network (DQN) algorithm with an ability to measure and take risk. In essence, we propose a novel DRL algorithm that encourages risk-seeking behaviour to enhance information acquisition during training. We demonstrate the merit of the exploration heuristic by (i) arguing that our risk estimator implicitly contains both parametric uncertainty and inherent uncertainty of the environment which are propagated back through Temporal Difference error across many time steps and (ii) evaluating our method on three games in the Atari domain and showing that the technique works well on Montezuma’s Revenge, a game that epitomises the challenge of sparse reward.