Search

Excavation autonomy is an area of industrial interest as there is a significant skills shortage for excavator operators, yet the industry is facing higher budget constraints than ever before. In particular, the excavation task of trenching is heavily impacted as it is difficult to perform but is also integral for most construction projects. Traditionally, autonomy in excavation is often rule-based or control theory-based, which is difficult to scale and apply to different scenarios such as different ground conditions or trench design, common in construction. Reinforcement learning has been applied with successes in similar fields, like robotics, demonstrating its capability in handling non-linearity and complex environments. This thesis addresses the deployment reinforcement learning to a trench excavation task, to provide a potential operator-like behaviour. Twin-Delayed Deep Deterministic Policy Gradients (TD3) were identified as the most appropriate for the research, due to their superior performance on robotics tasks. To deploy TD3 to trenching, a three-part strategy was used that focused on developing the environment, state, and reward function that TD3 would be involved with. First, the overall system architecture and environment for a reinforcement learning-based autonomy system was designed, which utilised the Common Data Environment to store and deploy a trained algorithm and the excavation plans. This was designed alongside driver data analysis to provide additional context to the challenges of trenching. Next, the selection of optimal machine sensors was studied, including the comparison of feature selection methods against existing sensor arrays. These were applied to a neural network trained on driver data, and the assumption was this would be like the policy of a trained TD3 algorithm. This was done to determine what impact the state has on the predictive performance of the policy whilst avoiding the inclusion of a reward function, allowing several machine tasks to use the same approach. Finally, TD3 was deployed to an excavator, focused on developing the reward function for trenching. This was done by breaking the trenching task into smaller tasks for developing the distance and mass rewards, before unifying them into one reward function. By focusing on individual elements of a reward function, the impact can be understood more effectively. TD3 was then deployed to perform a trench excavation task, where it was able to dig within the trench region and dump soil into a hopper. The novel findings of this research were 1) Exploratory analysis of driver data during trenching 2) An autonomy roadmap for excavation 3) An architecture of an RL-based autonomy system for excavation that utilises the driver, simulation, and Common Data Environment 4) The use of feature selection methods to determine machine sensor inputs for the policy network of TD3, without using the reward function 5) A new methodology for developing a reward function 6) The deployment of TD3 to perform a trench excavation task.

Multi-Agent Reinforcement Learning (MARL) faces numerous challenges in partially-observable environments due to limited information and the non-stationarity of the learning process. This study aims to determine the optimal rate and type of information sharing in MARL algorithms to improve performance in such environments. Here, we benchmark the Independent Q-Learning (IQL) and Independent Proximal Policy Optimization (IPPO) algorithms, modifying them to facilitate various forms of information sharing in the Level-Based Foraging environment. Our experimental results reaffirm theorized downsides of na¨ıve information sharing from scientific literature and demonstrate that optimal sharing rates may be an additional hyperparameter for optimization: notably, when allowing agents to choose when to share information, models perform competitively while converging to rates of sharing as low as 10% while achieving improved short-term returns. Additionally, we find that pairing information about agents’ actions with their observations yields the highest returns, possibly allowing agent networks to parameterize each others’ policies.

Abstract: This thesis develops foundations for the development of dependable, scalable reinforcement learning algorithms with strong connections to game theory. I present a version of rationality for learning---one grounded in the learner's experience and connected with the rationality concepts of optimality and equilibrium---that demands resiliency to uncertainty, environmental changes, and adversarial pressures. This notion of hindsight rationality is based on regret, a well-known concept for evaluating a sequence of decisions with unilateral deviations. I show that in sequential decision-making tasks, there are many natural deviation sets with critical practical differences beyond those previously studied. I design and implement three extensions to the counterfactual regret minimization (CFR) algorithm, one that is observably sequentially hindsight rational for any given subset of deviations within a broad class; a second that generalizes regression CFR; and a third that applies to continuing Markov decision processes and robust optimization tasks. The first part develops hindsight rationality and the partially observable history process (POHP) formalism for concisely describing multi-agent sequential decision-making from a single agent's perspective.The second part develops the foundations of defining, analyzing, and using deviations in finite-horizon POHPs to develop efficient hindsight rational algorithms, and the practical consequences of designing algorithms around different deviation sets. The third and final part describes experimental applications of these foundations that use function approximation and condensed domain representations to effectively play games and learn cautious behavior in safety challenges.

Data-driven approaches to the design of control policies for robotic systems have the potential to revolutionize our world. By continuously observing the environment and changes therein, learning algorithms can adapt and improve the control policies based on the observed data. One key requirement for these approaches to work well in real-world applications is data-efficiency: how long it takes to learn a successful control policy. This is a difficult problem for machine learning, because standard algorithms often lack the required data-efficiency for real-world applications. Further, treating the problem purely from a machine learning perspective neglects decades of research and experience from the field of control theory. The goal of this thesis is to combine insights from control theory with methodologies from machine learning to develop highly data-efficient learning algorithms for continuous control problems. The first part of this thesis considers automated controller tuning, that is, finding the optimal parameters for policies based on classical approaches from control theory without human intervention. To this end, we employ Bayesian optimization (BO), a data-efficient method that addresses global, stochastic optimization problems. In its standard formulation, BO makes only few assumptions about the underlying objective function. On the one hand, this makes BO a prime candidate to tackle a wide range of applications, but on the other hand, it limits BO's data-efficiency for the specific application controller tuning. This part of the thesis presents three separate methodologies that aim at alleviating some shortcomings of BO: First, we propose to constrain the search space locally around an initial solution to enable the optimization of high-dimensional control policies whilst retaining data-efficiency. Secondly, we propose to encode environmental conditions during experiments as context variables, which allows sharing of experience from previous experiments and thus accelerate subsequent ones. Thirdly, we consider the issue of exogenous perturbations that act on the policy's parameters, and we therefore require the optimal parameters to be robust with respect to these perturbations. The efficacy of these three methodologies is demonstrated on a wide range of simulated and real-world problems. The second part of this thesis considers a more general framework for the data-driven control approach: model-based reinforcement learning (RL). This particularly data-efficient branch of RL employs a learned model---an approximation to the true environment---to simulate artificial data instead of only relying on real-world interactions. However, the learned model always remains imperfect and as such introduces an error source to the learning problem. Hence, a key challenge in model-based RL is model-bias; small errors in the learned model that can compound when simulating new data and impede the learning process. This part of the thesis presents a novel approach to alleviate the issue of model-bias. Specifically, we use the observed data as time-dependent correction terms on top of a learned model, to retain the ability to simulate new data without accumulating errors over long prediction horizons. These correction terms are inspired by a data-driven branch of control theory: iterative learning control, which we thoroughly compare to model-based RL. Further, we motivate the proposed method from a theoretical perspective and demonstrate that it can drastically improve existing model-based approaches in practice without introducing additional tuning parameters.

In recent years, the remarkable achievements of RL have granted it a spot at the forefront of AI research. Crucially, most of these results were obtained in simulated environments, where poor actions have no harmful consequences. However, to unlock RL's full potential, we wish to deploy it in the real world. While this broadens the scope for RL's beneficial impact, it also amplifies the consequences of its harmful actions. Therefore, we must understand and address the causes that may induce RL agents to make potentially damaging decisions in real-world scenarios. This dissertation studies unsafe behaviors that may arise in RL from inaccurate models in small-data regimes. In particular, it focuses on the problem of robustness to distributional shift, i.e., not overfitting to training data and generalizing to previously unseen environmental conditions, and safe exploration, i.e., safely acquiring data during training. We start by introducing a model-free approximation of robustness indicators from linear control theory. We leverage our method to design control policies for a Furuta pendulum and we demonstrate their robustness in sim-to-real and hardware experiments that include a significant distributional shift. Subsequently, we study goal-oriented safe exploration when safety can be expressed as a set of unknown smooth constraints. To address this, we propose an algorithm that enjoys safety and completeness guarantees, and we show in simulated experiments that it significantly improves over existing methods in terms of sample efficiency. We then extend it to adaptive control problems, a class of classical control problems concerned with the distributional shift induced by exogenous variables. We deploy our algorithm to control a rotational axis drive in constantly changing environments. Finally, we present a novel framework to incorporate prior knowledge in safe exploration problems that allows us to lift many assumptions made by previous methods, e.g., smoothness. We provide safety guarantees for this framework and we combine it with deep RL agents to safely train control policies in challenging environments.

A study is presented on visual navigation of wheeled mobile robots (WMR) using deep reinforcement learning in unknown and dynamic environments. Two versions of deep reinforcement learning (DRL) algorithms, namely, value-learning based deep Q-network (DQN) and policy gradient based asynchronous advantage actor critic (A3C) have been considered in this study. Both DRL algorithms have been implemented using RGB and depth images as inputs to generate outputs for the WMR for autonomous navigation in both simulation and real-time. The initial DRL networks were generated and trained progressively in simulation environments using OpenAI Gym Gazebo within robot operating system (ROS) framework for a popular experimental WMR, namely, Kobuki TurtleBot2 with Asus Xtion depth camera. The real-time implementation of the trained DRL networks in ROS framework was achieved using onboard edge computing hardware platform of NVIDIA Jetson TX2 through software framework of TensorFlow. For object detection, classification, and target identification, a pre-trained deep neural network, namely, ResNet50 was used after further training with reduced classification categories for target-driven visual mapless navigation of Turlebot2 through DRL. The simulation based training of DQN and A3C networks was successfully transferred with online learning in real-time navigation of Turlebot2 in physical environments. The performance of A3C was simulated with multiple computation threads (4, 6, and 8) on a desktop. The simulated navigation performance, in terms of the minimum, the average, and the maximum rewards, and the completion tine was compared for DQN and A3C networks for three simulation environments. The performance of A3C with multiple threads (4, 6, and 8) was better than DQN, as expected. The performance of A3C also improved with the number of threads. The real-time implementation results of A3C with 8 threads in unknown and dynamic environments with target objects were promising. Details of the methodology, simulation and real-time implementation results are presented and recommendations for future work are outlined.

Reinforcement learning (RL) has attracted large attention over the past few years. Recently, we developed a data-driven algorithm to solve predictive cruise control (PCC) and games output regulation problems. This work integrates our recent contributions to the application of RL in game theory, output regulation problems, robust control, small-gain theory and PCC. The algorithm was developed for $H_\infty$ adaptive optimal output regulation of uncertain linear systems, and uncertain partially linear systems to reject disturbance and also force the output of the systems to asymptotically track a reference. In the PCC problem, we determined the reference velocity for each autonomous vehicle in the platoon using the traffic information broadcasted from the lights to reduce the vehicles' trip time. Then we employed the algorithm to design an approximate optimal controller for the vehicles. This controller is able to regulate the headway, velocity and acceleration of each vehicle to the desired values. Simulation results validate the effectiveness of the algorithms.