Search

Reinforcement learning (RL) has become a highly successful framework for learning in Markov decision processes (MDP). Due to the adoption of RL in realistic and complex environments, solution robustness becomes an increasingly important aspect of RL deployment. Nevertheless, current RL algorithms struggle with robustness to uncertainty, disturbances, or structural changes in the environment. We survey the literature on robust approaches to reinforcement learning and categorize these methods in four different ways: (i) Transition robust designs account for uncertainties in the system dynamics by manipulating the transition probabilities between states; (ii) Disturbance robust designs leverage external forces to model uncertainty in the system behavior; (iii) Action robust designs redirect transitions of the system by corrupting an agent’s output; (iv) Observation robust designs exploit or distort the perceived system state of the policy. Each of these robust designs alters a different aspect of the MDP. Additionally, we address the connection of robustness to the risk-based and entropy-regularized RL formulations. The resulting survey covers all fundamental concepts underlying the approaches to robust reinforcement learning and their recent advances.

Modern industry benefits from the automation capabilities and flexibility of robots. Consequently, the performance depends on the individual task, robot and trajectory, while application periods of several years lead to a significant impact of the use phase on the resource efficiency. In this work, simulation models predicting a robot’s energy consumption are extended by an estimation of the reliability, enabling the consideration of maintenance to enhance the assessment of the application’s life cycle costs. Furthermore, a life cycle assessment yields the greenhouse gas emissions for the individual application. Potential benefits of the combination of motion simulation and cost analysis are highlighted by the application to an exemplary system. For the selected application, the consumed energy has a distinct impact on greenhouse gas emissions, while acquisition costs govern life cycle costs. Low cycle times result in reduced costs per workpiece, however, for short cycle times and higher payloads, the probability of required spare parts distinctly increases for two critical robotic joints. Hence, the analysis of energy consumption and reliability, in combination with maintenance, life cycle costing and life cycle assessment, can provide additional information to improve the resource efficiency.