Search

Robotic systems are crucial in modern manufacturing. Complex assembly tasks require the collaboration of multiple robots. Their orchestration is challenging due to tight tolerances and precision requirements. In this work, we set up two Franka Panda robots performing a peg-in-hole insertion task of 1 mm clearance. We structure the control system hierarchically, planning the robots' feedback-based trajectories with a central policy trained with reinforcement learning. These trajectories are executed by a low-level impedance controller on each robot. To enhance training convergence, we use reverse curriculum learning, novel for such a two-armed control task, iteratively structured with a minimum requirements and fine-tuning phase. We incorporate domain randomization, varying initial joint configurations of the task for generalization of the applicability. After training, we test the system in a simulation, discovering the impact of curriculum parameters on the emerging process time and its variance. Finally, we transfer the trained model to the real-world, resulting in a small decrease in task duration. Comparing our approach to classical path planning and control shows a decrease in process time, but higher robustness towards calibration errors. [ABSTRACT FROM AUTHOR]

Aiming at the problem of online route planning for aircraft, an online autonomous decision-making method for aircraft based on deep reinforcement learning (DRL) is proposed. Firstly, the maneuvering model and detection model of the aircraft are explained, and then the deep deterministic policy gradient (DDPG) algorithm of DRL is employed to construct the frame of the aircraft policy model. On this basis, a curriculum learning (CL)-DDPG algorithm based on CL is proposed, which decomposes the online route planning task, guides the aircraft to learn the strategies of target approach, threat avoidance, and air route optimization. The corresponding Gausdan noises are set to help the aircraft explore and optimize the strategy. And, the adaptive learning and decision-making control of the aircraft in complex scenarios are realized. Simulation experiments show that the CL-DDPG algorithm can effectively improve the training efficiency of the model. The algorithm model has higher task success rate, excellent generalization and robustness, and can be better applied to online route planning tasks in complex dynamic environments. [ABSTRACT FROM AUTHOR]

Reinforcement learning, a cornerstone in the expansive landscape of artificial intelligence, has asserted its dominance as the prevailing methodology in contemporary multiagent system decision-making because of its formidable efficacy. However, the path to the zenith of algorithmic excellence is fraught with challenges intrinsic to traditional multiagent reinforcement learning algorithms, such as dimensionality explosion, scarcity of training samples, and the labyrinthine nature of migration processes. In a concerted effort to surmount these formidable challenges and propel the evolution of algorithmic prowess, this paper unfurls its inquiry from the perspective of learning mechanisms and undertakes an exhaustive exploration of the symbiotic integration of learning mechanisms and reinforcement learning. At the inception of this scholarly expedition, we meticulously delineate the rudimentary principles underpinning multiagent algorithms, present a historical trajectory tracing their developmental evolution, and cast a discerning eye upon the salient challenges that have been formidable impediments in their trajectory. The ensuing narrative charts a course into the avant-garde realm of multiagent reinforcement learning methods anchored in learning mechanisms, a paradigmatic shift that emerges as an innovative frontier in the field. Among these learning mechanisms, meta-learning and transfer learning are empirically validated as useful instruments in hastening the learning trajectory of multiagent systems and simultaneously mitigating the intricate challenges posed by dimensionality explosion. This paper assumes the role of a sagacious guide through the labyrinthine landscape of multiagent reinforcement learning, focusing on the manifold applications of learning mechanisms across diverse domains. A comprehensive review delineates the impact of learning mechanisms in curriculum learning, evolutionary games, meta-learning, hierarchical learning, and transfer learning. The research outcomes within these thematic realms are methodically cataloged, with a discerning eye cast upon the limitations inherent in each methodology and erudite propositions for the trajectory of future improvements. The discourse pivots toward synthesizing advancements and accomplishments wrought by fusion algorithms in practical milieus. This paper meticulously examines the transformative impact of fusion algorithms in real-world applications, with a detailed exposition of their deployment in domains as diverse as traffic control and gaming. Simultaneously, an incisive analysis charting the future trajectory of fusion algorithms is conducted. This prediction encompasses exploring nascent theories, refining algorithmic efficacy, and expanding dissemination and application across a broader spectrum of domains. Through this scholarly odyssey, this paper provides an invaluable compass for navigating the uncharted waters of future research endeavors and the judicious deployment of multiagent reinforcement learning algorithms in pragmatic scenarios.

This paper proposes a method for finding the shortest path of a mobile robot using deep reinforcement learning with utilizing Proximal policy optimization algorithm (PPO) enhanced with curriculum learning. By modelling the environment in 3D space using the Webots simulator, we extend the PPO algorithm's capabilities to handle continuous states from 8 IR sensors and control the velocities of two motors of E-puck robot. Our study uniquely integrates curriculum learning into the PPO framework, aiming to improve adaptability and training efficiency in complex environments. A comparative analysis is conducted between the modified PPO, the original PPO, and the deep deterministic policy gradient algorithm, highlighting the strengths of our approach The results demonstrate that our curriculum-augmented PPO algorithm not only accelerates the training process but also shows superior adaptability and generalization in new environments. This work underscores the significant potential of curriculum learning in enhancing the performance of deep reinforcement learning algorithms for robust and efficient robotic navigation. [ABSTRACT FROM AUTHOR]

Summary: In this paper, we propose a deep reinforcement‐based approach to generate an animation of an agent using a broom‐like tool to transport a target object. The tool is attached to the agent. So when the agent moves, the tool moves as well.The challenge is to control the agent to move and use the tool to push the target while avoiding obstacles. We propose a direction sensor to guide the agent's movement direction in environments with static obstacles. Furthermore, different rewards and a curriculum learning are implemented to make the agent efficiently learn skills for manipulating the tool. Experimental results show that the agent can naturally control the tool with different shapes to transport target objects. The result of ablation tests revealed the impacts of the rewards and some state components. [ABSTRACT FROM AUTHOR]

Artificial Intelligence bots are extensively used in multiplayer First-Person Shooter (FPS) games. By using Machine Learning techniques, we can improve their performance and bring them to human skill levels. In this work, we focused on comparing and combining two Reinforcement Learning training architectures, Curriculum Learning and Behaviour Cloning, applied to an FPS developed in the Unity Engine. We have created four teams of three agents each: one team for Curriculum Learning, one for Behaviour Cloning, and another two for two different methods of combining Curriculum Learning and Behaviour Cloning. After completing the training, each agent was matched to battle against another agent of a different team until each pairing had five wins or ten time-outs. In the end, results showed that the agents trained with Curriculum Learning achieved better performance than the ones trained with Behaviour Cloning by a matter of 23.67% more average victories in one case. In terms of the combination attempts, not only did the agents trained with both devised methods had problems during training, but they also achieved insufficient results in the battle, with an average of 0 wins. [ABSTRACT FROM AUTHOR]

Even though reinforcement-learning-based algorithms achieved superhuman performance in many domains, the field of robotics poses significant challenges as the state and action spaces are continuous, and the reward function is predominantly sparse. Furthermore, on many occasions, the agent is devoid of access to any form of demonstration. Inspired by human learning, in this work, we propose a method named highlight experience replay (HiER) that creates a secondary highlight replay buffer for the most relevant experiences. For the weights update, the transitions are sampled from both the standard and the highlight experience replay buffer. It can be applied with or without the techniques of hindsight experience replay (HER) and prioritized experience replay (PER). Our method significantly improves the performance of the state-of-the-art, validated on 8 tasks of three robotic benchmarks. Furthermore, to exploit the full potential of HiER, we propose HiER+ in which HiER is enhanced with an arbitrary data collection curriculum learning method. Our implementation, the qualitative results, and a video presentation are available on the project site: http://www.danielhorvath.eu/hier/.

Training task-oriented dialog agents based on reinforcement learning is time-consuming and requires a large number of interactions with real users. How to grasp dialog policy within limited dialog experiences remains an obstacle that makes the agent training process less efficient. In addition, most previous frameworks start training by randomly choosing training samples, which differs from the human learning method and hurts the efficiency and stability of training. Therefore, we propose Scheduled Curiosity-Deep Dyna-Q (SC-DDQ), a curiosity-driven curriculum learning framework based on a state-of- the-art model-based reinforcement learning dialog model, Deep Dyna-Q (DDQ). Furthermore, we design ed learning schedules for SC-DDQ and DDQ, respectively, following two opposite training strategies: classic curriculum learning and its reverse version. Our results show that by introducing scheduled learning and curiosity, the new framework leads to a significant improvement over the DDQ and Deep Q-learning (DQN). Surprisingly, we found that traditional curriculum learning was not always effective. Specifically, according to the experimental results, the easy-first and difficult-first strategies are more suitable for SC-DDQ and DDQ. To analyze our results, we adopt ed the entropy of sampled actions to depict action exploration and found that training strategies with high entropy in the first stage and low entropy in the last stage lead to better performance.

Robotic systems are crucial in modern manufacturing. Complex assembly tasks require the collaboration of multiple robots. Their orchestration is challenging due to tight tolerances and precision requirements. In this work, we set up two Franka Panda robots performing a peg-in-hole insertion task of 1 mm clearance. We structure the control system hierarchically, planning the robots’ feedback-based trajectories with a central policy trained with reinforcement learning. These trajectories are executed by a low-level impedance controller on each robot. To enhance training convergence, we use reverse curriculum learning, novel for such a two-armed control task, iteratively structured with a minimum requirements and fine-tuning phase. We incorporate domain randomization, varying initial joint configurations of the task for generalization of the applicability. After training, we test the system in a simulation, discovering the impact of curriculum parameters on the emerging process time and its variance. Finally, we transfer the trained model to the real-world, resulting in a small decrease in task duration. Comparing our approach to classical path planning and control shows a decrease in process time, but higher robustness towards calibration errors.

Achieving omnidirectional walking for bipedal robots is considered one of the most challenging tasks in robotics technology. Reinforcement learning (RL) methods have proved effective in bipedal walking tasks. However, most existing methods use state machines to switch between multiple policies and achieve omnidirectional gait, which results in shaking during the policy switching process for bipedal robots. To achieve a seamless transition between omnidirectional gait and transient motion for full-size bipedal robots, we propose a novel multi-agent RL method. Firstly, a multi-agent RL algorithm based on the actor–critic framework is designed, and policy entropy is introduced to improve exploration efficiency. By learning agents with parallel initial state distributions, we minimize reliance on gait planner effectiveness in the Robot Operating System (ROS). Additionally, we design a novel heterogeneous policy experience replay mechanism based on Euclidean distance. Secondly, considering the periodicity of bipedal robot walking, we develop a new periodic gait function. Including periodic objectives in the policy can accelerate the convergence speed of training periodic gait functions. Finally, to enhance the robustness of the policy, we construct a novel curriculum learning method by discretizing Gaussian distribution and incorporate it into the robot's training task. Our method is validated in a simulation environment, and the results show that our method can achieve multiple gaits through a policy network and achieve smooth transitions between different gaits. [ABSTRACT FROM AUTHOR]

Formulating dialogue policy as a reinforcement learning (RL) task enables a dialogue system to act optimally by interacting with humans. However, typical RL-based methods normally suffer from challenges such as sparse and delayed reward problems. Besides, with user goal unavailable in real scenarios, the reward estimator is unable to generate reward reflecting action validity and task completion. Those issues may slow down and degrade the policy learning significantly. In this paper, we present a novel scheduled knowledge distillation framework for dialogue policy learning, which trains a compact student reward estimator by distilling the prior knowledge of user goals from a large teacher model. To further improve the stability of dialogue policy learning, we propose to leverage self-paced learning to arrange meaningful training order for the student reward estimator. Comprehensive experiments on Microsoft Dialogue Challenge and MultiWOZ datasets indicate that our approach significantly accelerates the learning speed, and the task-completion success rate can be improved from 0.47%∼9.01% compared with several strong baselines. [ABSTRACT FROM AUTHOR]

Using reinforcement learning to control traffic signal systems has been discussed in recent years, but most works focused on simple scenarios such as a single crossroads, and the methods aiming at large-scale traffic scenarios face long-time training and suboptimal results. In this work, we develop a new multi-agent reinforcement model for large-scale traffic signal control tasks, and a curriculum transfer learning method is developed to optimize the joint policy step by step. The policies for different intersections are trained in a partially observable Markov decision process with centralized training and decentralized execution mechanism, and we design transformer modules for both the policy and evaluation networks by attention mechanism. We first train policies in a simple traffic scenario, and then these policies are transferred to the next curriculum by policy reloading, while the experiences of the source task are reused selectively. With the number of agents increasing, our method can achieve satisfactory performances quickly by reusing the knowledge from previous curriculums. We conduct several experiments on the Cityflow testbed. In the case of more than 10 crossroads, our model improve the mean reward from 3.0 to 5.0. [ABSTRACT FROM AUTHOR]

Artificial Intelligence bots are extensively used in multiplayer First-Person Shooter (FPS) games. By using Machine Learning techniques, we can improve their performance and bring them to human skill levels. In this work, we focused on comparing and combining two Reinforcement Learning training architectures, Curriculum Learning and Behaviour Cloning, applied to an FPS developed in the Unity Engine. We have created four teams of three agents each: one team for Curriculum Learning, one for Behaviour Cloning, and another two for two different methods of combining Curriculum Learning and Behaviour Cloning. After completing the training, each agent was matched to battle against another agent of a different team until each pairing had five wins or ten time-outs. In the end, results showed that the agents trained with Curriculum Learning achieved better performance than the ones trained with Behaviour Cloning by a matter of 23.67% more average victories in one case. In terms of the combination attempts, not only did the agents trained with both devised methods had problems during training, but they also achieved insufficient results in the battle, with an average of 0 wins.

Maneuver decision-making is essential for autonomous air combat. However, previous methods usually make decisions to aim at the target instead of hitting the target and use discrete action spaces instead of continuous action spaces. While these simplifications make maneuver decision-making easier, they also make maneuver decision-making more unrealistic. Meanwhile, previous studies usually rely on handcrafted reward functions, which are troublesome to design. Therefore, to solve these problems, we propose an automatic curriculum reinforcement learning method that enables agents to maneuver effectively in air combat from scratch. On the basis of curriculum reinforcement learning, maneuver decision-making is divided into a series of sub-tasks from easy to difficult. Thus, agents can gradually learn how to complete a series of sub-tasks, from easy to difficult without handcrafted reward functions. The ablation studies show that automatic curriculum learning is essential for reinforcement learning; namely, agents cannot make effective decisions without curriculum learning. Simulations show that, after training, agents are able to make effective decisions given different states, including tracking, attacking, and escaping, which are both rational and interpretable. [ABSTRACT FROM AUTHOR]

General mathematical reasoning is computationally undecidable, but humans routinely solve new problems. Moreover, discoveries developed over centuries are taught to subsequent generations quickly. What structure enables this, and how might that inform automated mathematical reasoning? We posit that central to both puzzles is the structure of procedural abstractions underlying mathematics. We explore this idea in a case study on five sections of beginning algebra on the Khan Academy platform. To define a computational foundation, we introduce Peano, a theorem-proving environment where the set of valid actions at any point is finite. We use Peano to formalize introductory algebra problems and axioms, obtaining well-defined search problems. We observe existing reinforcement learning methods for symbolic reasoning to be insufficient to solve harder problems. Adding the ability to induce reusable abstractions ('tactics') from its own solutions allows an agent to make steady progress, solving all problems. Furthermore, these abstractions induce an order to the problems, seen at random during training. The recovered order has significant agreement with the expert-designed Khan Academy curriculum, and second-generation agents trained on the recovered curriculum learn significantly faster. These results illustrate the synergistic role of abstractions and curricula in the cultural transmission of mathematics. This article is part of a discussion meeting issue 'Cognitive artificial intelligence'. [ABSTRACT FROM AUTHOR]

Non-Playable Characters (NPCs) are essential for creating a fully immersive video game experience. However, their behaviours are typically programmed using rigid methods such as Finite State Machines and Behaviour Trees. These methods are hard to maintain, struggle with implementing intricate cooperative behaviours, and patterns in NPC behaviour can easily be exploited by human players. Machine Learning and Reinforcement Learning (RL) methods in particular, offer a solution to these challenges by allowing dynamic and real-time NPC responses to human player actions and other characters in the game. This work describes and extends upon a Curriculum-based RL approach to train a single model for multiple agents representing vehicles to respect traffic regulations. These vehicles learn to navigate a road network which includes a traffic lights crossroad intersection and a roundabout junction. A comparison of our simulated data against datasets obtained from real world data is conducted; results demonstrate that the learned models have similar trends to the real-world data. • Multi-agent simulation for game characters navigating a road network. • Reinforcement Learning in game design can generate compelling agent behaviours. • Curricula can help reinforcement learning process to learn complex behaviours. • Macroscopic behaviour of agents matches real world data. [ABSTRACT FROM AUTHOR]

Reinforcement learning is an effective approach for solving decision-making problems. However, when using reinforcement learning to solve maneuver decision-making with sparse rewards, it costs too much time for training, and the final performance may not be satisfactory. In order to overcome the shortcomings, the method for maneuver decision-making based on curriculum learning and reinforcement learning is proposed. First, three curricula are designed to address the maneuver decision-making problem: angle curriculum, distance curriculum and hybrid curriculum. They are proposed according to the intuition that closer destinations are easier to arrive at. Then, they are used to train agents and compared with the original method without any curriculum. The training results show that angle curriculum can increase the speed and stability of training, and improve the performance of maneuver decision-making; distance curriculum can increase the speed and stability of agent training; hybrid curriculum is not better than the other curricula, because it makes the agent get stuck at the local optimum. The simulation results show that after training, the agent can handle the situations where targets come from different directions, and the maneuver decision-makings are rational, effective, and interpretable, whereas the method without curriculum is invalid.

High-resolution (HR) remote sensing images provide rich information for human activities. However, processing entire HR images is time-consuming, and many computations are meaningless for change detection tasks since objects often cluster in local regions. To alleviate the pressure of downstream detectors, previous studies introduce a regional attention process to roughly sample candidate patches, but most solutions are tailored to particular tasks and datasets. Motivated by these, we develop a novel reinforcement learning sampling framework, and train a human-like agent, named CDZoom, to locate regions of interest by simulating human zooming behaviors. To be specific, the proposed network consists of an encoder block, multiple context blocks and a decision block. It speeds up sequential sampling operations by gradually focusing the scope of observed scene and increasing the resolution. To avoid the sparse reward problem when learning complex sampling tasks, we introduce a novel training paradigm based on curriculum learning and policy distillation. The proposed CDZoom can sample multi-size patches from multi-scale scenes, and thus generalizes well to different requirements. Experiments on public change detection datasets demonstrate the effectiveness of our method. CDZoom can reduce the computational cost by over 50%, while maintaining similar detection accuracy to models which use full HR images. [ABSTRACT FROM AUTHOR]

This paper presents the use of deep Reinforcement Learning (RL) for autonomous navigation of an Unmanned Ground Vehicle (UGV) with an onboard three-dimensional (3D) Light Detection and Ranging (LiDAR) sensor in off-road environments. For training, both the robotic simulator Gazebo and the Curriculum Learning paradigm are applied. Furthermore, an Actor–Critic Neural Network (NN) scheme is chosen with a suitable state and a custom reward function. To employ the 3D LiDAR data as part of the input state of the NNs, a virtual two-dimensional (2D) traversability scanner is developed. The resulting Actor NN has been successfully tested in both real and simulated experiments and favorably compared with a previous reactive navigation approach on the same UGV. [ABSTRACT FROM AUTHOR]

A high degree of freedom (DOF) benefits manipulators by presenting various postures when reaching a target. Using a tendon-driven system with an underactuated structure can provide flexibility and weight reduction to such manipulators. The design and control of such a composite system are challenging owing to its complicated architecture and modeling difficulties. In our previous study, we developed a tendon-driven, high-DOF underactuated manipulator inspired from an ostrich neck referred to as the Robostrich arm. This study particularly focused on the control problems and simulation development of such a tendon-driven high-DOF underactuated manipulator. We proposed a curriculum-based reinforcement-learning approach. Inspired by human learning, progressing from simple to complex tasks, the Robostrich arm can obtain manipulation abilities by step-by-step reinforcement learning ranging from simple position control tasks to practical application tasks. In addition, an approach was developed to simulate tendon-driven manipulation with a complicated structure. The results show that the Robostrich arm can continuously reach various targets and simultaneously maintain its tip at the desired orientation while mounted on a mobile platform in the presence of perturbation. These results show that our system can achieve flexible manipulation ability even if vibrations are presented by locomotion.

Aiming at the attack and defense game problem in the target-missile-defender three-body confrontation scenario, intelligent game strategies based on deep reinforcement learning are proposed, including an attack strategy applicable to attacking missiles and active defense strategy applicable to a target/defender. First, based on the classical three-body adversarial research, the reinforcement learning algorithm is introduced to improve the purposefulness of the algorithm training. The action spaces the reward and punishment conditions of both attack and defense confrontation are considered in the reward function design. Through the analysis of the sign of the action space and design of the reward function in the adversarial form, the combat requirements can be satisfied in both the missile and target/defender training. Then, a curriculum-based deep reinforcement learning algorithm is applied to train the agents and a convergent game strategy is obtained. The simulation results show that the attack strategy of the missile can maneuver according to the battlefield situation and can successfully hit the target after avoiding the defender. The active defense strategy enables the less capable target/defender to achieve the effect similar to a network adversarial attack on the missile agent, shielding targets from attack against missiles with superior maneuverability on the battlefield. [ABSTRACT FROM AUTHOR]

Data inefficiency is one of the major challenges for deploying deep reinforcement learning algorithms widely in industry control fields, especially in regard to long-horizon sparse reward tasks. Even in a simulation-based environment, it is often prohibitive to take weeks to train an algorithm. In this study, a data-efficient training method is proposed in which a DQN is used as a base algorithm, and an elaborate curriculum is designed for the agent in the simulation scenario to accelerate the training process. In the early stage of the training process, the distribution of the initial state is set close to the goal so the agent can obtain an informative reward easily. As the training continues, the initial state distribution is set farther from the goal for the agent to explore more state space. Thus, the agent can obtain a reasonable policy through fewer interactions with the environment. To bridge the sim-to-real gap, the parameters for the output layer of the neural network for the value function are fine-tuned. An experiment on UAV maneuver control is conducted in the proposed training framework to verify the method. We demonstrate that data efficiency is different for the same data in different training stages. [ABSTRACT FROM AUTHOR]

The paper is devoted to scheduling in multiagent systems in the framework of the Flatland 3 competition. The main aim of this competition is to develop an algorithm for the effective control of dense traffic in complex railroad networks according to a given schedule. The proposed solution is based on reinforcement learning. To adapt this method to the particular scheduling problem, a novel approach based on structuring the reward function that stimulates an agent to adhere to its schedule was developed. The architecture of the proposed model is based on a multiagent version of centralized critic with proximal policy optimization (PPO) learning. In addition, a curriculum learning strategy was developed and implemented. This allowed the agent to cope with each level of complexity on time and train the model in more difficult conditions. The proposed solution won first place in the Flatland 3 competition in the reinforcement learning track. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) is currently the cutting-edge artificial intelligence approach in the field of energy management for hybrid electric vehicles. However, inefficient offline training limits the energy-saving efficacy of DRL-based energy management strategies (EMSs). Motivated by this, this article proposes a smart DRL-based EMS in a heuristic learning framework for an urban hybrid electric bus. In order to enhance the sampling efficiency, the prioritized experience replay technique is introduced into soft actor-critic (SAC) for the innovative formulation of an improved SAC algorithm. Additionally, to strengthen the generalizability of the improved SAC agent to real driving scenarios, a stochastic training environment is constructed. Afterward, curriculum learning is employed to develop a heuristic learning framework that expedites convergence. Experimental simulations reveal that the designed EMS expedites convergence by 85.58 % and saves fuel by 6.43 % compared with the cutting-edge baseline EMS. Moreover, the computation complexity test demonstrates that the designed EMS holds significant promise for real-time implementation. These findings highlight the contribution of this article in facilitating fuel conservation for urban hybrid electric buses through the application of emerging artificial intelligence technologies. • An I-SAC algorithm integrated with prioritized experience replay is formulated. • A stochastic training environment is designed based on massive real driving data. • A heuristic learning framework is developed based on curriculum learning. • Both generalizability and real-time performance are verified via online testing. • The proposed EMS expedites convergence by 85.58 % upon the SOTA baseline. [ABSTRACT FROM AUTHOR]

Portfolio management (PM) is a popular financial process that concerns the occasional reallocation of a particular quantity of capital into a portfolio of assets, with the main aim of maximizing profitability conditioned to a certain level of risk. Given the inherent dynamicity of stock exchanges and development for long-term performance, reinforcement learning (RL) has become a dominating solution for solving the problem of portfolio management in an automated and efficient manner. Nevertheless, the present RL-based PM methods just take into account the variations in prices of portfolio assets and the implications of price variations, while overlooking the significant relationships among different assets in the market, which are extremely valuable for managerial decisions. To close this gap, this paper introduces a novel deep model that combines two subnetworks; one to learn a temporal representation of historical prices using a refined temporal learner, while the other learns the relationships between different stocks in the market using a relation graph learner (RGL). Then, the above learners are integrated into the curriculum RL scheme for formulating the PM as a curriculum Markov Decision Process, in which an adaptive curriculum policy is presented to enable the agent to adaptively minimize risk value and maximize cumulative return. Proof-of-concept experiments are performed on data from three public stock indices (namely S&P500, NYSE, and NASDAQ), and the results demonstrate the efficiency of the proposed framework in improving the portfolio management performance over the competing RL solutions. [ABSTRACT FROM AUTHOR]

The poor robustness of the air combat autonomous trajectory planning strategy (ATP) trained through vanilla reinforcement learning (RL) methods is attributed to its dependence on the source environment. When this ATP is transformed from the source to the target environment, it may fail to lead to an effective dogfight victory due to disturbances, which can potentially threaten the flight safety of the unmanned combat aerial vehicle (UCAV). In order to enhance the robustness of ATP, we propose a robust RL-based method for generating an ATP strategy. This method can effectively consider uncertainties in the state and potential model misspecifications. First, a jointly trained adversary is designed to apply disturbances in the environment, forming a two-player zero-sum game with the ATP agent. To solve this game problem and learn a robust ATP, the robust adversarial RL method (RARL) is applied. Additionally, in order to prevent non-convergence of strategies resulting from the introduction of the RARL algorithm, a curriculum learning method is proposed that emulates the way human pilots learn, gradually progressing from simpler to more challenging courses. Finally, a quantitative method for evaluating the robustness of the ATP is proposed, taking into consideration the differences in ATP performance between the source and target combat environments. The results of robustness estimations demonstrate that the RARL algorithm, which is proposed herein, effectively enhances the ATP's robustness. [ABSTRACT FROM AUTHOR]

A novel method, the Pareto Envelope Augmented with Reinforcement Learning (PEARL), has been developed to address the challenges posed by multi-objective problems, particularly in the field of engineering where the evaluation of candidate solutions can be time-consuming. PEARL distinguishes itself from traditional policy-based multi-objective Reinforcement Learning methods by learning a single policy, eliminating the need for multiple neural networks to independently solve simpler sub-problems. Several versions inspired from deep learning and evolutionary techniques have been crafted, catering to both unconstrained and constrained problem domains. Curriculum Learning (CL) is harnessed to effectively manage constraints in these versions. PEARL's performance is first evaluated on classical multi-objective benchmarks. Additionally, it is tested on two practical PWR core Loading Pattern (LP) optimization problems to showcase its real-world applicability. The first problem involves optimizing the Cycle length (L C) and the rod-integrated peaking factor (F Δ h) as the primary objectives, while the second problem incorporates the mean average enrichment as an additional objective. Furthermore, PEARL addresses three types of constraints related to boron concentration (C b), peak pin burnup (B u m a x ), and peak pin power (F q). The results are systematically compared against conventional approaches from stochastic optimization. Notably, PEARL, specifically the PEAL-NdS (▪) variant, efficiently uncovers a Pareto front without necessitating additional efforts from the algorithm designer, as opposed to a single optimization with scaled objectives. It also outperforms the classical approach across multiple performance metrics, including the Hyper-volume. Future works will encompass a sensitivity analysis of hyper-parameters with statistical analysis to optimize the application of PEARL and extend it to more intricate problems. [ABSTRACT FROM AUTHOR]

Cervical spinal cord injuries frequently cause paralysis of all four limbs - a medical condition known as tetraplegia. Functional electrical stimulation (FES), when combined with an appropriate controller, can be used to restore motor function by electrically stimulating the neuromuscular system. Previous works have demonstrated that reinforcement learning can be used to successfully train FES controllers. Here, we demonstrate that transfer learning and curriculum learning can be used to improve the learning rates, accuracies, and workspaces of FES controllers that are trained using reinforcement learning.

Learning from only real-world collected data can be unrealistic and time consuming in many scenario. One alternative is to use synthetic data as learning environments to learn rare situations and replay buffers to speed up the learning. In this work, we examine the hypothesis of how the creation of the environment affects the training of reinforcement learning agent through auto-generated environment mechanisms. We take the autonomous vehicle as an application. We compare the effect of two approaches to generate training data for artificial cognitive agents. We consider the added value of curriculum learning—just as in human learning—as a way to structure novel training data that the agent has not seen before as well as that of using a replay buffer to train further on data the agent has seen before. In other words, the focus of this paper is on characteristics of the training data rather than on learning algorithms. We therefore use two tasks that are commonly trained early on in autonomous vehicle research: lane keeping and pedestrian avoidance. Our main results show that curriculum learning indeed offers an additional benefit over a vanilla reinforcement learning approach (using Deep-Q Learning), but the replay buffer actually has a detrimental effect in most (but not all) combinations of data generation approaches we considered here. The benefit of curriculum learning does depend on the existence of a well-defined difficulty metric with which various training scenarios can be ordered. In the lane-keeping task, we can define it as a function of the curvature of the road, in which the steeper and more occurring curves on the road, the more difficult it gets. Defining such a difficulty metric in other scenarios is not always trivial. In general, the results of this paper emphasize both the importance of considering data characterization, such as curriculum learning, and the importance of defining an appropriate metric for the task.

In this work, we combine Curriculum Learning with Deep Reinforcement Learning to learn without any prior domain knowledge, an end-to-end competitive driving policy for the CARLA autonomous driving simulator. To our knowledge, we are the first to provide consistent results of our driving policy on all towns available in CARLA. Our approach divides the reinforcement learning phase into multiple stages of increasing difficulty, such that our agent is guided towards learning an increasingly better driving policy. The agent architecture comprises various neural networks that complements the main convolutional backbone, represented by a ShuffleNet V2. Further contributions are given by (i) the proposal of a novel value decomposition scheme for learning the value function in a stable way and (ii) an ad-hoc function for normalizing the growth in size of the gradients. We show both quantitative and qualitative results of the learned driving policy. [ABSTRACT FROM AUTHOR]

For years, the fully-automated robotic assembly has been a highly sought-after technology in large-scale manufacturing. Yet it still struggles to find widespread implementation in industrial environments. Traditional programming has so far proven to be insufficient in providing the required flexibility and dexterity to solve complex assembly tasks. Research in robotic control using deep reinforcement learning (DRL) advances quickly, however, the transfer to real-world applications in industrial settings is lagging behind. In this study, we apply DRL for robotic motion control to a multi-body contact automotive assembly task. Our focus lies on optimizing the final performance on the real-world setup. We propose a reward-curriculum learning approach in combination with domain randomization to obtain both force-sensitivity and generalizability of the controller's performance. We train the agent exclusively in simulation and successfully perform the Sim-to-Real transfer. Finally, we evaluate the controller's performance and robustness on an industrial setup and reflect its adherence to the high standards of automotive production. [ABSTRACT FROM AUTHOR]

Maneuver decision-making is essential for autonomous air combat. However, previous methods usually make decisions to aim at the target instead of hitting the target and use discrete action spaces instead of continuous action spaces. While these simplifications make maneuver decision-making easier, they also make maneuver decision-making more unrealistic. Meanwhile, previous studies usually rely on handcrafted reward functions, which are troublesome to design. Therefore, to solve these problems, we propose an automatic curriculum reinforcement learning method that enables agents to maneuver effectively in air combat from scratch. On the basis of curriculum reinforcement learning, maneuver decision-making is divided into a series of sub-tasks from easy to difficult. Thus, agents can gradually learn how to complete a series of sub-tasks, from easy to difficult without handcrafted reward functions. The ablation studies show that automatic curriculum learning is essential for reinforcement learning; namely, agents cannot make effective decisions without curriculum learning. Simulations show that, after training, agents are able to make effective decisions given different states, including tracking, attacking, and escaping, which are both rational and interpretable.

Recently, research on unmanned aerial vehicles (UAVs) has increased significantly. UAVs do not require pilots for operation, and UAVs must possess autonomous flight capabilities to ensure that they can be controlled without a human pilot on the ground. Previous studies have mainly focused on rule-based methods, which require specialized personnel to create rules. Reinforcement learning has been applied to research on UAV autonomous flight; however, it does not include six-degree-of-freedom (6-DOF) environments and lacks realistic application, resulting in difficulties in performing complex tasks. This study proposes a method of efficient learning by connecting two different maneuvering methods using modular learning for autonomous UAV flights. The proposed method divides complex tasks into simpler tasks, learns them individually, and then connects them in order to achieve faster learning by transferring information from one module to another. Additionally, the curriculum learning concept was applied, and the difficulty level of individual tasks was gradually increased, which strengthened the learning stability. In conclusion, modular learning and curriculum learning methods were used to demonstrate that UAVs can effectively perform complex tasks in a realistic, 6-DOF environment.

We propose a muscle‐driven motion generation approach to realize virtual human motion with user interaction and higher fidelity, which can address the problem that the joint‐driven fails to reflect the motion process of the human body. First, a simplified virtual human musculoskeletal model is built based on human biomechanics. Then, a hierarchical policy learning framework is constructed including motion tracking layer, SPD controller and muscle control layer. The motion tracking layer is responsible for mimicking reference motion and completing control command, using proximal policy optimization to train the policy; the muscle control layer is aimed to minimize muscle energy consumption and train the policy based on supervised learning; the SPD controller acts as a link between the two layers. At the same time, we integrate the curriculum learning to improve the efficiency and success rate of policy training. Simulation experiments show that the proposed approach can use motion capture data and pose estimation data as reference motions to generate better and more adaptable motions. Furthermore, the virtual human has the ability to respond to the user control command during the motion, and can complete the target task successfully. [ABSTRACT FROM AUTHOR]

Unmanned Aerial Vehicles (UAVs) are increasingly being used for data harvesting from Wireless Sensor Nodes (SNs). This study aims to minimize the Age of Information (AoI) during data collection, while also considering the energy sustainability of the UAVs. Addressing the challenge of optimizing performance in Multi-Agent Reinforcement Learning (MARL) systems as the number of agents increases, our study introduces a MARL approach combined with curriculum learning and evolutionary strategy. This innovation specifically targets the performance issue in traditional MARL setups when scaling up the number of agents. By applying curriculum learning and evolutionary strategies, our method not only enhances MARL scalability but also integrates energy-efficient charging mechanisms, effectively enhancing system performance in large-scale deployments. Numerical results show that our proposed algorithm outperforms baselines in terms of AoI and charging, proving its effectiveness in managing the complexities of large-scale MARL systems. • MARL with EPC minimizes AoI effectively. • EPC algorithm enhances multi-agent learning. • Energy-efficient trajectories vital for UAV data harvesting. • AoI minimization constant crucial for real-time tasks. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) is effective for autonomous navigation tasks without prior knowledge of the environment. However, traditional mobile robot navigation algorithms, based on off-policy RL, often face challenges such as low sample efficiency during training and lack of adequate safety mechanisms. In this paper, we present an off-policy RL navigation model named Soft Actor-Critic with Curriculum Prioritization and Fuzzy Logic (SCF). The model uses energy as a prioritized evaluation metric for experience replay. And through task-level curriculum, the agent's learning sequence is formulated, thereby enhancing sampling efficiency and safety. We propose a Curriculum-based Energy Prioritization (CEP) approach. It selects a replay trajectory that matches the current agent's capability based on trajectory energy. Our results show that robots using off-policy RL often have limitations in dynamic obstacle avoidance. To rectify this, our model uses a fuzzy logic controller to enhance real-time obstacle avoidance. The SCF approach enables mobile robots to navigate adeptly in unpredictable and dynamic environments, ensuring optimal planning control while being safe and robust. Experiments in Gazebo simulation environment and real world confirm the effectiveness of our proposed method. The comparison results show the superior performance of this method, especially in unknown and dynamic environments. • An end-to-end navigation framework based on off-policy RL is proposed. • Improving agent learning using a task-level curriculum learning framework. • Designing a self-paced function to adaptively compute sample prioritization. • Introducing an energy-based prioritization evaluation metric. • Designing a fuzzy logic controller to help agents avoid obstacles. [ABSTRACT FROM AUTHOR]

This paper presents the use of deep Reinforcement Learning (RL) for autonomous navigation of an Unmanned Ground Vehicle (UGV) with an onboard three-dimensional (3D) Light Detection and Ranging (LiDAR) sensor in off-road environments. For training, both the robotic simulator Gazebo and the Curriculum Learning paradigm are applied. Furthermore, an Actor–Critic Neural Network (NN) scheme is chosen with a suitable state and a custom reward function. To employ the 3D LiDAR data as part of the input state of the NNs, a virtual two-dimensional (2D) traversability scanner is developed. The resulting Actor NN has been successfully tested in both real and simulated experiments and favorably compared with a previous reactive navigation approach on the same UGV.

Many everyday activities are sequential in nature. That is, they can be seen as a sequence of subactions and sometimes subgoals. In the motor execution of sequential action, context effects are observed in which later subactions modulate the execution of earlier subactions (e.g., reaching for an overturned mug, people will optimize their grasp to achieve a comfortable end state). A trajectory (movement) adaptation of an often‐used paradigm in the study of sequential action, the serial response time task, showed several context effects of which centering behavior is of special interest. Centering behavior refers to the tendency (or strategy) of subjects to move their arm or mouse cursor to a position equidistant to all stimuli in the absence of predictive information, thereby reducing movement time to all possible targets. In the current study, we investigated sequential action in a virtual robotic agent trained using proximal policy optimization, a state‐of‐the‐art deep reinforcement learning algorithm. The agent was trained to reach for appearing targets, similar to a serial response time task given to humans. We found that agents were more likely to develop centering behavior similar to human subjects after curricularized learning. In our curriculum, we first rewarded agents for reaching targets before introducing a penalty for energy expenditure. When the penalty was applied with no curriculum, many agents failed to learn the task due to a lack of action space exploration, resulting in high variability of agents' performance. Our findings suggest that in virtual agents, similar to infants, early energetic exploration can promote robust later learning. This may have the same effect as infants' curiosity‐based learning by which they shape their own curriculum. However, introducing new goals cannot wait too long, as there may be critical periods in development after which agents (as humans) cannot flexibly learn to incorporate new objectives. These lessons are making their way into machine learning and offer exciting new avenues for studying both human and machine learning of sequential action. In a sequential reaching task, a robot arm trained using deep reinforcement learning optimizes its behavior in a similar manner to humans. This so‐called centering behavior, however, only tends to develop when learning is performed in a curriculum in which the reward function becomes increasingly complex over time. Furthermore, the onset of the curriculum seems to have a sensitive period, with too early or too late onset leading to unstable or suboptimal performance. [ABSTRACT FROM AUTHOR]

In swarm decision making, hand-crafting agents' rules that use local information to achieve desirable swarm-level behaviours is a non-trivial design problem. Instead of relying entirely on swarm experts for designing these local rules, machine learning (ML) algorithms can be utilised for learning some of the local rules by mapping an agent's perception to an appropriate action. To facilitate this process, we propose the use of Machine Education (ME) as a systematic approach for designing a curriculum for teaching the agents the required skills to autonomously select appropriate behaviours. We study the use of ME in the context of decision-making in best-of-n problems. The proposed approach draws on swarm robotics expertise for identifying agents' capabilities and limitations, the skills required for generating the desirable behaviours, and the corresponding performance measures. In addition, ME utilises ML expertise for the selection and development of the ML algorithms suitable for each skill. The results of the experimental evaluations demonstrate the superior efficacy of the ME-based approach over the state-of-the-art approaches with respect to speed and accuracy. In addition, our approach shows considerable robustness to changes in swarm size and to changes in sensing and communication noise. Our findings promote the use of ME for teaching swarm members the required skills for achieving complex swarm tasks. [ABSTRACT FROM AUTHOR]