Search

Reinforcement Learning (RL) has emerged as a promising approach to solve complex problems in many different domains, including the Architecture, Engineering, Construction and Operation (AECO) industry. RL is a type of machine learning that focuses on training an agent to interact with an environment in order to maximize a reward signal. In the AECO industry, RL has been used to optimize building design, construction planning and scheduling, and building energy management. This contribution presents an application of RL for the generation of an InfraBIM model of a tunnel built by mechanised excavation using a Tunnel Boring Machine (TBM). In particular, the present application was developed to minimise the distance between the theoretical layout and the operational one to be executed by the TBM, considering the geometry of the ring and the structural joints required for the solidity of the structure. RL has the potential to improve efficiency, sustainability, and safety in the AECO industry by enabling intelligent decision-making and optimization across different phases of the construction process., QC 20231204

Mixed-precision quantization mostly predetermines the model bit-width settings before actual training due to the non-differential bit-width sampling process, obtaining suboptimal performance. Worse still, the conventional static quality-consistent training setting, i.e., all data is assumed to be of the same quality across training and inference, overlooks data quality changes in real-world applications which may lead to poor robustness of the quantized models. In this article, we propose a novel data quality-aware mixed-precision quantization framework, dubbed DQMQ, to dynamically adapt quantization bit-widths to different data qualities. The adaption is based on a bit-width decision policy that can be learned jointly with the quantization training. Concretely, DQMQ is modeled as a hybrid reinforcement learning (RL) task that combines model-based policy optimization with supervised quantization training. By relaxing the discrete bit-width sampling to a continuous probability distribution that is encoded with few learnable parameters, DQMQ is differentiable and can be directly optimized end-to-end with a hybrid optimization target considering both task performance and quantization benefits. Trained on mixed-quality image datasets, DQMQ can implicitly select the most proper bit-width for each layer when facing uneven input qualities. Extensive experiments on various benchmark datasets and networks demonstrate the superiority of DQMQ against existing fixed/mixed-precision quantization methods.

Due to the massive computing demands of the Internet of Things, mobile edge computing (MEC) has been extensively investigated as a means of providing computation-intensive and latency-sensitive services at the network edge. With increasing density of base stations (BSs), users are simultaneously served by multiple BSs, leading to the multicell MEC environment. Intelligent reflecting surface (IRS) provides a promising solution for constructing the virtual Line-of-Sight (LoS) links between cell-edge users (CEUs) and BSs. In this article, we investigate the joint channel estimation and resource allocation in the IRS-aided multicell MEC system. Instead of assuming the perfect channel state information (CSI), we propose a three-phase channel estimation method to obtain the CSI. Our purpose is to minimize the total joint energy and latency cost (JELC) in terms of both task-execution latency and energy consumption in the IRS-aided multicell MEC problem by jointly optimizing the task offloading volume, precoding matrix, and IRS phase shifts. We propose a quadratically constrained program (QCP)-assisted proximal policy optimization (PPO) reinforcement learning algorithm with two modules (i.e., QCP optimizer and PPO agent) execute iteratively. The QCP optimizer is utilized to compute the offloading decision variables, and the PPO agent is adapted to determine the optimal channel precoding matrix and the phase shifts of IRS. Numerical results validate that our QCP-assisted PPO algorithm executes more rapidly than benchmarks. Moreover, the proposed QCP-assisted PPO algorithm delivers the best performance compared to benchmarks. Furthermore, the multicell IRS-aided MEC framework yields additional performance gains compared to those without IRS. © 2014 IEEE.

In the rapidly evolving landscape of industrial automation, Automated Guided Vehicles (AGVs) have become indispensable in enhancing efficiency and productivity in manufacturing and warehouse operations. This thesis project explores the optimization of routing and scheduling for AGVs in industrial environments using reinforcement learning (RL). Given the crucial role of AGVs in material handling and logistics within factories and warehouses, enhancing their efficiency is pivotal. The project is informed by the advancements in AI and the increasing integration of such technologies in industrial operations, aligning with initiatives like Sweden’s Production2030. A simulated environment where created and RL models where tested against traditional dispatched rules, underscoring the potential RL to enhance the productivity and efficiency of AGVs in warehouse settings.

Many real-world problems fall in the category of sequential decision-making under uncertainty; Markov Decision Processes (MDPs) are a common method for modeling such problems. To solve an MDP, one could start from scratch or one could already have an idea of what good policies look like. Furthermore, there could be uncertainty in this idea. In existing literature, a policy search procedure is accelerated by encoding this prior knowledge in an action distribution which is used for policy sampling. Moreover, this is then extended by inferring these action distributions while inferring the policy through Gibbs sampling. Implicitly, this approach assumes a generalization of good and bad actions over the entire state space. This thesis extends the existing method by leveraging a division of the state space into regions and inferring action distributions over these regions, rather than over the entire state space. We show that this can accelerate the policy search. We also show that the algorithm manages to recover if the division is unjustified. The division into regions can hence also be considered a form of prior knowledge of the policy with uncertainty. Finally, inference of the regions themselves is also explored and yields promising results., Computer Science

In this paper, the Proximal Policy Optimization (PPO) algorithm is used to perform a constrained wing shape optimization. The PPO algorithm is a Machine Learning (ML) algorithm that improves itself by repeatedly performing the same optimization and learning from its results. The complete adaptation of the PPO framework to the design problem is detailed and evaluated. Not only was the PPO framework able to consistently optimize the wing 4% further than the Particle Swarm Optimization (PSO) algorithm, it was able to do so 35 times faster once the model is fully trained. The PPO framework was able to find more efficient wing shapes than the PSO framework. The trained PPO model was able to optimize the wing of other similar aircraft, even without direct retraining. These results illustrate that PPO could be a promising technique for automated aerospace design problems. Due to the significant training time of the ML approach, the PPO algorithm is not an effective replacement of traditional optimization algorithms for design problems where only a single optimization is required., Aerospace Engineering

This paper presents a hierarchical reinforcement learning framework for efficient robotic manipulation in sequential contact tasks. We leverage this hierarchical structure to sequentially execute behavior primitives with variable stiffness control capabilities for contact tasks. Our proposed approach relies on three key components: an action space enabling variable stiffness control, an adaptive stiffness controller for dynamic stiffness adjustments during primitive execution, and affordance coupling for efficient exploration while encouraging compliance. Through comprehensive training and evaluation, our framework learns efficient stiffness control capabilities and demonstrates improvements in learning efficiency, compositionality in primitive selection, and success rates compared to the state-of-the-art. The training environments include block lifting, door opening, object pushing, and surface cleaning. Real world evaluations further confirm the framework’s sim2real capability. This work lays the foundation for more adaptive and versatile robotic manipulation systems, with potential applications in more complex contact-based tasks., Mechanical Engineering | Vehicle Engineering | Cognitive Robotics

Mask optimization is a vital step in the VLSI manufacturing process in advanced technology nodes. As one of the most representative techniques, optical proximity correction (OPC) is widely applied to enhance printability. Since conventional OPC methods consume prohibitive computational overhead, recent research has applied machine learning techniques for efficient mask optimization. However, existing discriminative learning models rely on a given dataset for supervised training, and generative learning models usually leverage a proxy optimization objective for end-to-end learning, which may limit the feasibility. In this article, we pioneer introducing the reinforcement learning (RL) model for mask optimization, which directly optimizes the preferred objective without leveraging a differentiable proxy. Intensive experiments show that our method outperforms state-of-the-art solutions, including academic approaches and commercial toolkits.

The modular multilevel converter (MMC) is a prominent solution for medium- to high-voltage and high-power conversion applications. Recently, distributed control strategies have been proposed to make this converter modular in terms of software and control hardware. In this control architecture, high-level control tasks are performed by a central controller (CC), whereas low-level control tasks are achieved by local controllers (LCs) placed on the MMC submodules. The CC and LCs use a cyber-physical system (CPS) to share all the necessary information to execute their respective control schemes. In this context, the CPS is vulnerable to cyberattacks, such as the false data injection attack (FDIA), where the data seen by the controllers are corrupted through illegitimate data intrusion. This cyberattack may hinder the MMC performance, producing suboptimal, or even unstable operations. Even more, diverse FDIAs can be generated using artificial intelligence methods to deceive FDIA detectors. This article proposes an FDIA detector based on the reinforcement learning (RL) technique to detect sophisticated FDIAs targeting the MMC control system. The performance of the proposed RL-based FDIA detector is verified via hardware-in-the-loop studies, showing its effectiveness in detecting sophisticated attack sequences affecting the MMC control system.

More recently, the advancement of deep learning techniques enables reinforcement learning (RL) to handle high-dimensional decision-making problems. It brings increasing attention in transport areas. Some work has applied them to solve the intractable traffic signal control (TSC) problem and achieved promising results. However, very few research comprehensively investigates the impacts of key design elements, including state definitions, reward functions and deep reinforcement learning (DRL) methods, on TSC performance. To fill this research gap, this paper first selects commonly used design elements from existing literature. Then, we compare their learning stability and control performance at an isolated intersection under different scenarios via simulation experiments. The experimental results show that the quantitative state (e.g., the number of vehicles on lanes) and image-like state (e.g., vehicle position and speed) have no significant impact on the performance under different traffic demands. However, the impact of various reward functions on performance is vivid, especially under high traffic demand. Also, high-resolution vehicular network data may not be possible to improve control performance versus ordinary camera data. In addition, the value-based DRL algorithms outperform the policy-based algorithm and traditional TSC control methods. The findings of this research would provide insights and guidance for transport engineers to design an efficient DRL-based TSC system in a real traffic environment., QC 20231106

(English) The expeditious growth and development of Beyond-5G and 6G networks inaugurated a new sector in the research of cellular networking. Automation and scaling are now considered a necessity to cope with the anticipated demand of the new high-performing communication networks. The current networks expand at a rate that creates an urgent pressure for Infrastructure Providers to adopt intelligent, AI-based technologies for automated network management and orchestration. This Ph.D. Thesis is dedicated to the development of an intelligent, automated and scalable service management and orchestration framework, that does not require human intervention to operate. Several experiments were conducted to modeland validate the use of the RL-based zero-touch management and orchestration algorithms under different conditions and settings. This Ph.D Thesis is divided into three main parts. The first part is dedicated to the new paradigm that modern beyond 5G and 6G telecommunications brought to society. Speed, reliability, and scale are now at the epicenter of the industry. The enabling technologies that allow reaching the promised Key Performance Indicators are discussed. Slicing, the main focus of Beyond-5G networks is introduced and analyzed from both technical and business standpoints. The second part of this dissertation is devoted to the automated management and orchestration of computing and network resources in 5G and beyond networks. A dynamic, resource-aware Virtual Network Functions placement AI-based algorithm is proposed and explained in detail. Later, an end-to-end data-driven service orchestration framework is proposed and discussed. A special vehicular Ultra-Reliable Low Latency Communications (URLLC) scenario is further explored to evaluate and demonstrate the performance of the proposed algorithms. The main focus of this part is the End-to-End average latency reduction for such services. Finally, in the third part is focused on the developed SCHEMA framewo, (Català) El ràpid creixement i desenvolupament de les xarxes Beyond-5G i 6G va inaugurar un nou sector en la recerca de xarxes cel·lulars. L’automatització i l’escalat es consideren ara una necessitat per fer front a la demanda prevista de les noves xarxes de comunicació d’alt rendiment. Les xarxes actuals s’expandeixen a un ritme que crea una pressió urgent perquè els prove¨ıdors d’infraestructures adoptin tecnologies intel·ligents basades en IA per a la gestió i l’orquestració de xarxes automatitzades. Aquesta tesi doctoral està dedicada al desenvolupament d’un marc d’orquestració i gestió de serveis intel·ligent, automatitzat i escalable, que no requereix intervenció humana per funcionar. Es van realitzar diversos experiments per modelar i validar l’ús dels algorismes d’orquestració i gestió sense toc basats en RL en diferents condicions i configuracions. Aquesta tesi doctoral es divideix en tres parts principals. La primera part està dedicada al nou paradigma que les telecomunicacions modernes més enllà del 5G i 6G van aportar a la societat. La velocitat, la fiabilitat i l’escala es troben ara a l’epicentre de la indústria. Es discuteixen les tecnologies habilitadores que permeten assolir els indicadors clau de rendiment promesos. Slicing, el focus principal de les xarxes Beyond-5G s’introdueix i s’analitza tant des del punt de vista tècnic com empresarial. La segona part d’aquesta tesi està dedicada a la gestió i orquestració automatitzada de recursos informàtics i de xarxa en xarxes 5G i més enllà. Es proposa i s’explica amb detall un algorisme de col·locació de funcions de xarxa virtual dinàmic i conscient dels recursos basat en IA. Més tard, es proposa i es discuteix un marc d’orquestració de serveis basat en dades d’extrem a extrem. S’explora un escenari especial de comunicacions de baixa latència ultra fiable (URLLC) per a vehicles per avaluar i demostrar el rendiment dels algorismes proposats. L’objectiu principal d’aquesta part és la reducció de la latèn, Postprint (published version)

Digital transformation within smart manufacturing presents new challenges for wireless communication, demanding stringent reliability and latency. One prominent approach to meet these requirements in 5G technology is to leverage spatial diversity techniques, such as the transmission of duplicated packets via independent user plane paths. While spatial diversity and hardware redundancy ensure high availability and reduced latency, they increase wireless resource utilization significantly. In this paper, we investigate a scenario where large industrial devices can access multiple user plane paths via multiple user equipment. To manage this effectively, we propose a deep Q-network-based reinforcement learning control framework that optimizes spatial diversity use to maximize communication service availability with minimized wireless resource usage. We implement our solution on a 3GPP-compliant simulator for a factory automation scenario. Our results show that our framework can adapt to varying delay bounds and greatly enhance communication service availability compared to the baselines. Remarkably, our method achieves these results more resource-efficiently, evading the baseline's need for double the bandwidth for comparable availability levels., Part of ISBN 9783800762262QC 20240704

Aerospace Engineering

This research paper aims to investigate the effect of entropy while training the agent on the robustness of the agent. This is important because robustness is defined as the agent's adaptability to different environments. A self-driving car should adapt to every environment that it is being used in since a mistake could cost someone's life. Therefore, robustness is of great importance in self-driving cars. An increase in entropy would promote the exploration of different strategies and prevent convergence on local maximum results. In order to test entropy values in training, the Soft-Actor Critic algorithm is used. The algorithm is run on a simulated city environment called Carla. In the end, collected data shows that the agent which is trained with a higher entropy value adapts to environments it did not train on better than the low entropy agent. However, a low entropy agent performs better in the environment it is trained in. Therefore, increasing the entropy increases robustness but it lowers the performance in the training environment itself., CSE3000 Research Project, Computer Science and Engineering

We demonstrate a management system that lets a service provider achieve end-to-end management objectives under varying load for applications on a service mesh based on the Istio and Kubernetes platforms. The management objectives for the demonstration include end-to-end delay bounds on service requests, throughput objectives, and service differentiation. Our method for finding effective control policies includes a simulator and a control module. The simulator is instantiated with traces from a testbed, and the control module trains a reinforcement learning (RL) agent to efficiently learn effective control policies on the simulator. The learned policies are then transfered to the testbed to perform dynamic control actions based on monitored system metrics. We show that the learned policies dynamically meet management objectives on the testbed and can be changed on the fly., Part of ISBN 9781665477161QC 20230821

Softmax selection is one of the most popular methods for action selection in reinforcement learning. Although various recently proposed methods may be more effective with full parameter tuning, implementing a complicated method that requires the tuning of many parameters can be difficult. Thus, softmax selection is still worth revisiting, considering the cost savings of its implementation and tuning. In fact, this method works adequately in practice with only one parameter appropriately set for the environment. The aim of this paper is to improve the variable setting of this method to extend the bandwidth of good parameters, thereby reducing the cost of implementation and parameter tuning. To achieve this, we take advantage of the asymptotic equipartition property in a Markov decision process to extend the peak bandwidth of softmax selection. Using a variety of episodic tasks, we show that our setting is effective in extending the bandwidth and that it yields a better policy in terms of stability. The bandwidth is quantitatively assessed in a series of statistical tests., source:http://ieeexplore.ieee.org/document/7468547

We exhibit an important property called the asymptotic equipartition property (AEP) on empirical sequences in an ergodic multiagent Markov decision process (MDP). Using the AEP which facilitates the analysis of multiagent learning, we give a statistical property of multiagent learning, such as reinforcement learning (RL), near the end of the learning process. We examine the effect of the conditions among the agents on the achievement of a cooperative policy in three different cases: blind, visible, and communicable. Also, we derive a bound on the speed with which the empirical sequence converges to the best sequence in probability, so that the multiagent learning yields the best cooperative result.

Information freshness is crucial for time-critical IoT applications, e.g., environment monitoring and control systems. We consider an Internet of things (IoT) network with multiple energy harvesting sensors, users, and an edge node. The users are interested in time-sensitive information about physical quantities, each measured by a sensor. Users make on-demand requests to a cache-enabled edge node where the cache contains the most recently received measurements from each sensor. The edge node serves the users' requests by deciding whether to command the corresponding sensors to send fresh status updates or use the aged data in the cache. We aim to find the best actions of the edge node at each time slot for each sensor, i.e., optimal policies, that minimizes the average age of information (AoI) of the served measurements, i.e., average on-demand AoI. In the present work, we first study status updating for decoupled sensors, i.e., the sensors have independent communication channels to the edge node. We model this problem as a Markov decision process (MDP) and developed two classes of reinforcement learning (RL) based algorithms: a model-based relative value iteration algorithm (RVIA) relying on dynamic programming, and a model-free Q-learning method. Then, we study status updating in IoT networks under a transmission constraint where the edge node can command only a limited number of sensors at each time slot, i.e., leading to per-slot transmission constraint. We model the problem as an MDP for which an iterative algorithm is proposed to obtain an optimal policy. Note, however, that the computational complexity of finding an optimal policy increases exponentially in the number of sensors. Thus, we develop an {asymptotically optimal low-complexity} algorithm -- termed {relax-then-truncate} -- and prove that it is optimal as the number of sensors goes to infinity. Finally, we study status updating under inexact knowledge about the battery levels of the sensors, Tiivistelmä Tiedon tuoreus on ratkaisevan tärkeää aikakriittisissä IoT-sovelluksissa, kuten ympäristön seuranta- ja valvontajärjestelmissä. Tutkimme esineiden internetin (IoT) verkkoa, jossa on useita energiaa kerääviä antureita, käyttäjiä ja reunasolmu. Käyttäjät ovat kiinnostuneita aikaherkistä tiedoista fyysisistä suureista, joista kutakin mitataan anturilla. Käyttäjät lähettävät aina tarvitessaan pyyntöjä välimuistia käyttävälle reunasolmulle, jonka välimuistissa ovat viimeisimmät vastaanotetut mittaukset kustakin anturista. Reunasolmu vastaa käyttäjien pyyntöihin päättämällä, ohjataanko vastaavat anturit lähettämään tuoreita tilapäivityksiä vai käytetäänkö välimuistissa olevia jo ikääntyneitä tietoja. Pyrimme löytämään parhaat reunasolmun valitsemat toiminnot kullakin hetkellä kutakin anturia varten eli optimaaliset käytännöt, jotka minimoivat tarjottujen mittausten keskimääräisen tiedon iän (AoI) ja siis pyydettäessä tarjottavien tietojen keskimääräisen iän. Tässä tutkielmassa tutkimme ensin irti kytkettyjen antureiden tilan päivittämistä, jolloin siis antureilla on itsenäiset viestintäkanavat reunasolmuun. Mallinnamme tämän ongelman Markovin päätöksentekoprosessina (MDP). Olemme kehittäneet kaksi vahvistusoppimiseen (RL) perustuvien algoritmien luokkaa: mallipohjaisen suhteellisen arvon iterointialgoritmin (RVIA), jossa käytetään dynaamista ohjelmointia, sekä mallittoman Q-oppimismenetelmän. Seuraavaksi tutkimme tilan päivittämistä siirtorajoitetuissa IoT-verkoissa, joissa reunasolmu voi ohjata vain rajallista määrää antureita kullakin aikavälillä. Voimassa on siis aikavälikohtainen siirtorajoitus. Mallinnamme ongelman MDP:nä, jossa ehdotetaan käytettäväksi iteratiivista algoritmia optimaalisen toimintatavan saavuttamiseksi. On kuitenkin syytä huomata, että optimaalisen toimintatavan löytämisen laskennallinen monimutkaisuus kasvaa eksponentiaalisesti antureiden lukumäärän mukaan. Siksi kehitämme asymptoottisesti optimaalisen matalan monimutkaisuuden algoritmi

Physical activity is one of the main factors that contribute to reducing the chance of chronic diseases such as cardiovascular disease, obesity, and depression, all while improving an individual’s health in general. While this is the case, the fact still remains that many adults across the world do not reach the minimum recommendations for physical activity. Setting physical activity goals is one of the most common approaches found in both self-help fitness apps, or medical interventions for increasing physical activity. The issue is that goal setting on its own cannot help people become more physically active, since, if they are not committed to reaching a goal, a person will just abandon it instead. Recent literature shows that creating plans for when to perform physical activity has the potential to help people commit to reaching their goal. Being committed to following the plan is therefore important for ultimately reaching the goal, but only creating the plan is not enough, since people might abandon it. Furthermore, dyadic planning, in which a helper aids the person in creating the plan, has produced even better results than individual planning. Thus, the aim of this work is to develop a virtual coach, which plays the role of the helper in dyadic planning and motivates people to commit themselves to following the plan, so that they can reach their physical activity goals. To facilitate this process, the virtual coach, named Jamie, operates based on reinforcement learning, giving it the ability to select the best persuasive strategy to use. It does so by taking into account the person's situation (opinions of the plan and of planning in general), as well as how these opinions might change based on what persuasive strategy the agent chooses to employ. The persuasive strategies considered were: proposing to make changes to the plan, explaining why planning is useful, identifying and dealing with barriers, and showing testimonials from other people who created pla, https://doi.org/10.17605/OSF.IO/8ADP9 OSF pre-registration of the study https://doi.org/10.5281/zenodo.10126244 Code for the virtual coach used in the study https://doi.org/10.4121/2796f502-0610-4a7d-a8ee-ebc36639e0b1 Dataset and analysis code, Computer Science

This thesis addresses the Learning-Based Control (LBC) of unknown partially observable systems in the Linear Quadratic (LQ) paradigm. In this setting of learning-based LQ control, the control action influences not only the control performance but also the rate at which the system is being learnt, causing a conflict between learning and control (exploration and exploitation), which is particularly challenging to address. This thesis aims to develop a novel LBC algorithm for unknown partially observable systems in the LQG setting that is computationally efficient and can guarantee an optimal exploration-exploitation trade-off, quantified by a metric called regret. The regret quantifies the cumulative performance gap between the LBC policy and the ideal controller having full knowledge of the true system dynamics. The contributions in this thesis involve a novel LBC algorithm deployed in a two-phase structure. The first phase involves injecting Gaussian input signals to obtain an initial system model. The subsequent second phase deploys the proposed LBC strategy in an episodic setting, where the model is updated for each episode, and the resulting updated LQG controller is applied with additive Gaussian signals for exploration. In addition, the thesis establishes strong theoretical guarantees on optimal regret growth., Mechanical Engineering | Systems and Control

In this article, a novel reinforcement learning (RL) approach, continuous dynamic policy programming (CDPP), is proposed to tackle the issues of both learning stability and sample efficiency in the current RL methods with continuous actions. The proposed method naturally extends the relative entropy regularization from the value function-based framework to the actor-critic (AC) framework of deep deterministic policy gradient (DDPG) to stabilize the learning process in continuous action space. It tackles the intractable softmax operation over continuous actions in the critic by Monte Carlo estimation and explores the practical advantages of the Mellowmax operator. A Boltzmann sampling policy is proposed to guide the exploration of actor following the relative entropy regularized critic for superior learning capability, exploration efficiency, and robustness. Evaluated by several benchmark and real-robot-based simulation tasks, the proposed method illustrates the positive impact of the relative entropy regularization including efficient exploration behavior and stable policy update in RL with continuous action space and successfully outperforms the related baseline approaches in both sample efficiency and learning stability.

The critical challenge for employing autonomous control systems in aircraft is ensuring robustness and safety. This study introduces an intelligent and fault-tolerant controller that merges two Reinforcement Learning (RL) algorithms in a hybrid approach: the Distributional Soft Actor-Critic (DSAC) and the Incremental Dual Heuristic Programming (IDHP). The integration combines the strengths of DSAC in learning a robust control strategy and IDHP in allowing real-time control adaption. Compared to earlier controllers, such as a hybrid using the Soft Actor-Critic (SAC) algorithm and strictly offline DSAC and SAC, our hybrid demonstrates enhanced robustness against changing flight conditions and in the face of sensor noise and bias. During fault tolerance tests, it maintains superior control even when the effectiveness of the aircraft’s ailerons and elevators is compromised. By demonstrating the potential of RL-based controllers to provide robustness and fault tolerance, this research advances the feasibility of safe and autonomous flight control operations., Hybrid RL Flight Control Code Repository Link - https://github.com/iamlucasvieira/HybridRL-FlightControl, Aerospace Engineering

Wind energy, generated by windfarms, is playing an increasingly critical role in meeting current and future energy demands. windfarms, however, face a challenge due to the inherent flaw of wake-induced power losses when turbines are located in close proximity. Wakes, characterized by regions of turbulence and lower wind speed, are created as air passes through the rotors, reducing the efficiency of downstream turbines. Wake losses can be reduced by yawing upstream turbines to steer the wake away from downstream turbines. While there are power losses associated with turning turbines off-wind, the gains in the subsequent turbines can outweigh these losses. Yawing turbines to increase overall power output is known as Active Wake Control, and the literature shows that single-agent reinforcement learning algorithms can be used to learn such control policies. However, these approaches are limited to a small number of turbines and do not scale well to larger windfarms. Multi agent reinforcement learning algorithms do scale to larger windfarms and this paper investigates the application of the DGN algorithm for windfarm active wake control. DGN is a fully cooperative algorithm that utilizes a graph representation of agents to encourage collaboration among neighboring turbines. The use of DGN is particularly interesting because windfarms naturally have a topological structure, and depending on the modeling choices, graphs can capture a significant amount of information. This paper demonstrates that DGN can learn useful policies for wake control. Although it does not outperform the DQN single-agent algorithm on small windfarms, its advantage becomes apparent in larger windfarms, where its performance remains consistent while DQN’s performance deteriorates., CSE3000 Research Project, Computer Science and Engineering

Reinforcement Learning (RL) has gained atten-tion as a way of creating autonomous agents for self-driving cars. This paper explores the adap- tation of the Deep Q Network (DQN), a popular deep RL algorithm, in the Carla traffic simulator for autonomous driving. It investigates the influ- ence of action space discretization and DQN ex- tensions on training performance and robustness. Results show that action space discretization en- hances behaviour consistency but negatively af- fects Q-values, training performance, and robust- ness. Double Q-Learning decreases training per- formance and leads to suboptimal convergence, re- ducing robustness. Prioritized Experience Replay also performs worse during training, but consis-tently outperforms in robustness testing, reward es-timation and generalization., CSE3000 Research Project, Computer Science and Engineering

Greenhouses allow production of crops that would otherwise be impossible. Permitting more local, fresher and nutrient richer crop production. Eorts are taken to minimize societal harm due to energy and resource consumption by greenhouse production systems. One way to control such systems is by using model predictive control. Optimal crop yield and resource eciency can, in theory, be achieved by model predictive control. Unfortunately, one major drawback of model predictive control is that it is not well equipped to deal with parametric uncertainty. Significant prediction errors can occur when a mismatch between the model and the real system exists, resulting in deteriorated performance of the system. Strategies exist, such as robust MPC, that are designed to handle uncertainty, but those often result in conservative control policies. This thesis proposes to use model predictive control as a function approximator for RL in order to learn values for model and MPC parameters that can deliver optimal performance in the case of model mismatch. In this thesis, data-driven economic nonlinear model predictive control using Q-learning is proposed as a method to alter the model parameters. The performance of the system af- ter learning is compared to approaches using robust and nominal model predictive control. Three dierent goals are determined: maximizing economic profit, minimizing the constraint violations and maximizing the economic performance while minimizing constraint violations. In this work, an ENMPC scheme is used as a function approximator in a Q-learning envi- ronment. The optimization solution from the ENMPC scheme is used as the input to the system, while the Q-learning agent optimizes the parameter values of the ENMPC scheme and model for the environment. The performance of the system after learning is compared to approaches using robust and nominal model predictive control. The simulation results show that the data-driven ENMPC using reinforcement le, Mechanical Engineering | Systems and Control

Photovoltaic (PV) energy, representing a renewable source of energy, plays a key role in the reduction of greenhouse gas emissions and the achievement of a sustainable mix of energy generation. To achieve the maximum solar energy harvest, PV power systems require the implementation of Maximum Power Point Tracking (MPPT). Traditional MPPT controllers, such as P&O, are easy to implement, but they are by nature slow and oscillate around the MPP losing efficiency. This work presents a Reinforcement learning (RL)-based control to increase the speed and the efficiency of the controller. Deep Deterministic Policy Gradient (DDPG), the selected RL algorithm, works with continuous actions and space state to achieve a stable output at MPP. A Digital Twin (DT) enables simulation training, which accelerates the process and allows it to operate independent of weather conditions. In addition, we use the maximum power achieved in the DT to adjust the reward function, making the training more efficient. The RL control is compared with a traditional P&O controller to validate the speed and efficiency increase both in simulations and real implementations. The results show an improvement of 10.45% in total power output and a settling time 24.54 times faster in simulations. Moreover, in real-time tests, an improvement of 51.45% in total power output and a 0.25 s settling time of the DDPG compared with 4.26 s of the P&O is obtained.

With the rapid growth of logistics transportation in the framework of Industry 4.0, automated guided vehicle (AGV) technologies have developed speedily. These systems present two coupled control problems: the control of the longitudinal velocity, essential to ensure the application requirements such as throughput and tag time, and the trajectory tracking control, necessary to ensure the proper accuracy in loading and unloading manoeuvres. When the paths are very short or have abrupt changes, the kinematic constraints play a restrictive role, and the tracking control becomes more challenging. In this case, advanced control strategies such as those based on intelligent techniques, including machine learning (ML) can be useful. Hence, in this work, we present an intelligent hybrid control scheme that combines reinforcement learning-based control (RLC) with conventional PI regulators to face both control problems simultaneously. On the one hand, PIs are used to control the speed of each wheel. On the other hand, the input reference of these regulators is calculated by the RLC in order to reduce the guiding error of the path tracking and to maintain the longitudinal speed. The latter is compared with a PID path following controller. The PID regulators have been tuned by genetic algorithms. The RLC allows the vehicle to learn how to improve the trajectory tracking in an adaptive way and thus, the AGV can face disturbances or unknown physical system parameters that may change due to friction and degradation of AGV mechanical components. Extensive simulation experiments of the proposed intelligent control strategy on a hybrid tricycle and differential AGV model, that considers the kinematics and the dynamics of the vehicle, prove the efficiency of the approach when following different demanding trajectories. The performance of the RL tracking controller in comparison with the optimized PID gives errors around 70% smaller, and the average maximum error is also 48% lower., Open access funding enabled and organized by Projekt DEAL.

The ability to model other agents can be of great value in multi-agent sequential decision making problems and has become more accessible due to the introduction of deep learning into reinforcement learning. In this study, the aim is to investigate the usefulness of modelling other agents using variational autoencoder based models in partially observable settings. Previous studies that model other agents using (variational) autoencoders have shown promising results. In these studies, a single protagonist agent learns representations of other agents to then use them as additional components of its observation space which is, as such, augmented with those representations. It is, however, not always entirely clear what is being modelled and what would be the best feature of the other agent to represent. Moreover, in these works, a comparison between the used variational autoencoder based models and a baseline classifier trained to solve the same classification task is missing. This study investigates which features can best be used for the augmentation of the observations of deep reinforcement learning agents and if these features can be represented by variational autoencoder based models. Subsequently, it compares these models with a baseline classifier that solves the same classification problem to find out which model yields the best results when used for augmenting observations. Overall, the results suggest that it is beneficial to augment the observations of deep reinforcement learning agents with features related to other agents learned in a pre-training phase. Another interesting result is that the baseline classifier achieves similar or better performance compared to the variational autoencoder based model. Further research needs to be conducted to confirm the soundness of these findings., Computer Science | Artificial Intelligence

In mobile robot applications, the trajectory tracking task hides several difficulties, including the choice of the setpoint and the search for an acceptable trade-off between performance and computational constraints. In this work, we discuss practical issues of a Reinforcement Learning (RL) based Model Predictive Control (MPC) tuning approach by focusing on a specific mobile robot application, where the objective is to maximize the velocity, while keeping the robot within the track bounds. Among others, we show that softening the latter constraints allows us to obtain a RL-tuned tracking controller with the same performance of an economic nonlinear MPC formulation, but requiring significantly less computational resources.

Non-terrestrial networks (NTNs) traditionally have certain limited applications. However, the recent technological advancements and manufacturing cost reduction opened up myriad applications of NTNs for 5G and beyond networks, especially when integrated into terrestrial networks (TNs). This article comprehensively surveys the evolution of NTNs highlighting their relevance to 5G networks and essentially, how it will play a pivotal role in the development of 6G ecosystem. We discuss important features of NTNs integration into TNs and the synergies by delving into the new range of services and use cases, various architectures, technological enablers, and higher layer aspects pertinent to NTNs integration. Moreover, we review the corresponding challenges arising from the technical peculiarities and the new approaches being adopted to develop efficient integrated ground-air-space (GAS) networks. Our survey further includes the major progress and outcomes from academic research as well as industrial efforts representing the main industrial trends, field trials, and prototyping towards the 6G networks., QC 20231009

In vehicular networks, the benefits of jointly-optimizing data prefetching and caching with broadcast transmission scheduling and rate adaptation strategies at road-side units (RSUs) have already been demonstrated in the literature. Nevertheless, the effectiveness of the solution depends greatly on the accurate knowledge of vehicular trajectories. In practice, as specified in the standards, this is at odds with the important issue of privacy. One of the main contributions of this work is to address this issue by providing a scheme that jointly optimizes the throughput of broadcast-transmission scheduling from RSUs to vehicles, and the privacy of vehicles' locations, by enabling them to disseminate obfuscated locations to the server from time to time. We formulate this problem as a reinforcement learning (RL) problem, where the vehicles learn when to report obfuscated locations, in order to maximize a utility that encompasses both the network capacity (related to vehicles' throughput) and the level of privacy achieved. The proposed scheme is shown to consistently outperform alternative randomized schemes considered in past work, and in particular it proves to be robust against prediction errors of the future locations of the vehicles.

Deep Reinforcement Learning (DRL) shows great potential for flight control, due to its adaptability, fault-tolerance, and as it does not require an accurate system model. However, these techniques, like many machine learning applications, are considered black-box as their inner workings are hidden. This paper aims to break open the black box of RL for adaptive flight control by applying Shapley Additive Explanations (SHAP). The generated explanations are aimed at control experts, but can be useful for anyone interested in RL for adaptive flight control. This research proposes a novel Constant Weight Segment Detection (CWSD) algorithm, facilitating the use of eXplainable Artificial Intelligence techniques to adaptive RL. The algorithm and its usefulness are tested on an Adaptive Critic Design controlling a high-fidelity model of a Cessna Citation aircraft. It is demonstrated that SHAP in combination with CWSD provides detailed and useful insights into the relation between input and output of the RL algorithm. Using SHAP, linear relations between input and output are discovered, simplifying the understanding of the learned strategy., Aerospace Engineering

Machine learning models are increasingly being used in fields that have a direct impact on the lives of humans. Often these machine learning models are black-box models and they lack transparency and trust which is holding back the implementation. To increase transparency and trust this research investigates whether imitation learning, specifically Generative Adversarial ImitationLearning (GAIL), can be used to give insights into the black-box models by extracting decision trees. To achieve this, an extension of GAIL was made allowing it to extract decision trees. The decision trees were then measured in terms of performance, fidelity, behavior, and interpretability in three different environments. We find that GAIL is able to extract decision trees with high fidelity and can give insightful information into the expert models. Moreover, further research can be done on more complex environments and black-box models, other surrogate models, and possibilities for more specific local insights., CSE3000 Research Project, Computer Science and Engineering

The domains of the negotiation can vary significantly. It is possible that a domain is very cooperative, where both agents can receive a high utility; the opposite is also possible, where the domain is very competitive and the agents cannot both get a high utility. In the same manner, the agents can have different strategies leading to a complicated problem with no obvious solution. This research seeks to represent the differences in negotiation domains to improve a machine learning based agent to help the agent generalize these domains. To achieve this several ways of representing the domain have been explored. First is the shared domain information. With this representation, the agent uses information about the amount of issues, values and possible bids there are. Second is the private domain information, in this representation, the agent uses different calculations to get a view of how favorable the domain is in terms of utility. Last is the derived information, this is the representation where the agent learns about the domain by interaction with the environment or the opposing agent. From the experiments, a conclusion could be made that a part of these representations had a positive impact on the final utility of the agent. The shared domain information had a considerable improvement over the base agent with the features having a non-negligible impact on the negotiation. The derived information also had a considerable impact on the final outcome., https://github.com/brenting/negotiation_PPO Implementation discussed in paper can be found here, CSE3000 Research Project, Computer Science and Engineering

This thesis has provided insight into how machine learning can be beneficial to path planning in container terminals. Path planning algorithms can be used in environments with automated vehicles. A well known algorithm is the A* path planning algorithm, which is the fastest optimal path planning algorithm under satisfied conditions. However, the behaviour of a container terminal is unknown beforehand, costs can change over iterations. Therefore, Liu et al. [Liu et al., 2019] and Keselman et al. [Keselman et al., 2018] show the advantage of combining A* with Machine Learning. This way, the exploring part of the ML algorithm is combined with the fast andmore precise properties of the A* PP algorithm. This thesis has proposed the machine learning algorithm Vehicle Aware Reinforcement Learning Path Planning Algorithm VARLPPA. This algorithm uses Monte Carlo Control method. This is a model free approach, which has been shown in both experiments to find more efficient solutions in exceptional situations., Marine Technology | Transport Engineering and Logistics

Background. Game developers are continuously searching for new ways of populating their vast game worlds with competent and engaging Non-Player Characters (NPCs), and researchers believe Deep Reinforcement Learning (DRL) might be the solution for emergent behavior. Consequently, fusing NPCs with DRL practices has surged in recent years, however, proposed solutions rarely outperform traditional script-based NPCs. Objectives. This thesis explores a novel method of developing an adversarial DRL NPC by combining Reinforcement Learning (RL) algorithms. Our goal is to produce an agent that surpasses its script-based opponents by first mimicking their actions. Methods. The experiment commences with Imitation Learning (IL) before proceeding with supplementary DRL training where the agent is expected to improve its strategies. Lastly, we make all agents participate in 100-deathmatch tournaments to statistically evaluate and differentiate their deathmatch performances. Results. Statistical tests reveal that the agents reliably differ from one another and that our learning agent performed poorly in comparison to its script-based opponents. Conclusions. Based on our computed statistics, we can conclude that our solution was unsuccessful in developing a talented hostile DRL agent as it was unable to convey any form of proficiency in deathmatches. No further improvements could be applied to our ML agent due to the time constraints. However, we believe our outcome can be used as a stepping-stone for future experiments within this branch of research.

Edge computing provides a promising paradigm to support the implementation of Internet of Things (IoT) by offloading tasks to nearby edge nodes. Meanwhile, the increasing network size makes it impractical for centralized data processing due to limited bandwidth, and consequently a decentralized learning scheme is preferable. Reinforcement learning (RL) has been widely investigated and shown to be a promising solution for decision-making and optimal control processes. For RL in a decentralized setup, edge nodes (agents) connected through a communication network aim to work collaboratively to find a policy to optimize the global reward as the sum of local rewards. However, communication costs, scalability, and adaptation in complex environments with heterogeneous agents may significantly limit the performance of decentralized RL. Alternating direction method of multipliers (ADMM) has a structure that allows for decentralized implementation and has shown faster convergence than gradient descent-based methods. Therefore, we propose an adaptive stochastic incremental ADMM (asI-ADMM) algorithm and apply the asI-ADMM to decentralized RL with edge-computing-empowered IoT networks. We provide convergence properties for the proposed algorithms by designing a Lyapunov function and prove that the asI-ADMM has O(1/k) + O(1/M) convergence rate, where k and M are the number of iterations and batch samples, respectively. Then, we test our algorithm with two supervised learning problems. For performance evaluation, we simulate two applications in decentralized RL settings with homogeneous and heterogeneous agents. The experimental results show that our proposed algorithms outperform the state of the art in terms of communication costs and scalability and can well adapt to complex IoT environments., QC 20230412

This article presents a novel spatiotemporal optimization approach for vertical path planning (i.e., waypoint optimization) to maximize the net output power of an ocean current turbine (OCT) under uncertain ocean velocities. To determine the net power, OCT power generation from hydrokinetic energy and the power consumption for controlling the depth are modeled. The stochastic behavior of ocean velocities is a function of spatial and temporal parameters, which is modeled through a Gaussian process (GP) approach. Two different algorithms, including model predictive control (MPC) as a model-based method and reinforcement learning (RL) as a learning-based method, are applied to solve the formulated spatiotemporal optimization problem with constraints. Comparative studies show that the MPC- and RL-based methods are computationally feasible to address vertical path planning, which are evaluated with a baseline A∗ approach. Analysis of the robustness is further carried out under the inaccurate ocean velocity predictions. Results verify the efficiency of the presented methods in finding the optimal path to maximize the total power of an OCT system, where the total harnessed energy after 200 h shows over an 18% increase compared to the case without optimization.

This article investigates the multirobot cooperative navigation problem based on raw visual observations. A fully end-to-end learning framework is presented, which leverages graph neural networks to learn local motion coordination and utilizes deep reinforcement learning to generate visuomotor policy that enables each robot to move to its goal without the need of environment map and global positioning information. Experimental results show that, with a few tens of robots, our approach achieves comparable performance with the state-of-the-art imitation learning-based approaches with bird-view state inputs. We also illustrate our generalizability to crowded and large environments and our scalability to ten times number of the training robots. In addition, we demonstrate that our model trained for multirobot case can also improve the success rate in the single-robot navigation task in unseen environments. IEEE

One of the most crucial challenges faced by the Li-ion battery community concerns the search for the minimum time charging without damaging the cells. This goal can be achieved by solving a large-scale constrained optimal control problem, which relies on accurate electrochemical models. However, these models are limited by their high computational cost, as well as identifiability and observability issues. As an alternative, simple output-feedback algorithms can be employed, but their performance strictly depends on trial and error tuning. Moreover, particular techniques have to be adopted to handle safety constraints. With the aim of overcoming these limitations, we propose an optimal-charging procedure based on deep reinforcement learning. In particular, we focus on a policy gradient method to cope with continuous sets of states and actions. First, we assume full state measurements from the Doyle-Fuller-Newman (DFN) model, which is projected to a lower dimensional feature space via the principal component analysis. Subsequently, this assumption is removed, and only output measurements are considered as the agent observations. Finally, we show the adaptability of the proposed policy to changes in the environment's parameters. The results are compared with other methodologies presented in the literature, such as the reference governor and the proportional-integral-derivative approach.

Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future., Interactive Intelligence, Algorithmics

The implementation of a model-free, off-policy, actor-critic deep reinforcement learning algorithm consistent of two separate agents to a six-degree-of freedom spacecraft docking maneuver to develop a control policy is carried out in the research presented in this article. Reinforcement learning has the ability to learn without instruction, this aspect provides a potential framework for autonomous docking maneuvers in uncertain environments with low on-board computational cost. A Twin-Delayed Deep Deterministic Policy Gradient algorithm consistent of two agents is used to synthesise the docking control policy valid for the six degree-of-freedom continuous state-space. Testing of the resultant policy exhibits its behaviour and capability to achieve successful docking within the established position and attitude ranges., Aerospace Engineering

With the recent increase in the complexity of aerospace systems and autonomous operations, there is a need for an increased level of adaptability and model-free controller synthesis. Such operations require the controller to maintain safety and performance without human intervention in non-static environments with partial observability and uncertainty. Deep Reinforcement Learning (DRL) algorithms have the potential to increase the safety and autonomy of aerospace control systems. It has been shown that the soft actor-critic (SAC) algorithm can achieve robust control of a CS-25 certified aircraft and has the generalization power to react to failure scenarios. Traditional DRL approaches, such as the state-of-the-art SAC algorithm struggle with inconsistent learning in high-dimensional tasks and fall short of modelling uncertainty and risk in the environment. In contrast, distributional RL algorithms estimate the entire probability distribution of rewards, improve the learning characteristics and enable the synthesis of risk- sensitive policies. This paper demonstrates the improved learning characteristics of distributional soft actor-critic (DSAC) compared to traditional SAC and discusses the benefits of risk-sensitive learning applied to flight control. We show that the addition of distributional critics significantly improves learning consistency, and successfully approximates the uncertainty when applied to a fully-coupled attitude control task of a jet aircraft., Public code repository https://github.com/peter-seres/dsac-flight, Aerospace Engineering

The ever-increasing demand for sustainable transportation necessitates advancements in automotive engineering to achieve significant reductions in fuel consumption and emissions. Artificial Intelligence (AI) has emerged as a powerful tool in this pursuit, offering innovative approaches to optimize fuel efficiency within complex vehicle powertrain systems. This paper comprehensively examines the application of AI in automotive engineering, focusing on advanced models, techniques, and real-world case studies that demonstrate their effectiveness in improving fuel economy and minimizing environmental impact. The paper begins with a critical overview of the challenges in fuel efficiency optimization. Traditional control strategies based on rule-based systems struggle to adapt to dynamic driving conditions and complex engine behavior. Additionally, the intricate interactions between various powertrain components further complicate the optimization process. AI, with its capability to learn and adapt from vast datasets, offers a paradigm shift in addressing these challenges. The paper delves into various AI models employed for fuel efficiency optimization. Machine Learning (ML) techniques, particularly supervised learning algorithms like Regression models and Support Vector Machines (SVM) are explored. These algorithms utilize historical vehicle data encompassing engine parameters, driving conditions, and fuel consumption to establish predictive models that optimize fuel economy by anticipating future driving scenarios. Further, the paper explores the application of Deep Learning (DL) architectures, specifically Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) for fuel efficiency optimization. CNNs excel at extracting features from sensor data related to engine operation and driving patterns. RNNs, with their ability to capture temporal dependencies, are particularly valuable in predicting future fuel consumption based on sequential driving data. Th

This article investigates the multirobot cooperative navigation problem based on raw visual observations. A fully end-to-end learning framework is presented, which leverages graph neural networks to learn local motion coordination and utilizes deep reinforcement learning to generate visuomotor policy that enables each robot to move to its goal without the need of environment map and global positioning information. Experimental results show that, with a few tens of robots, our approach achieves comparable performance with the state-of-the-art imitation learning-based approaches with bird-view state inputs. We also illustrate our generalizability to crowded and large environments and our scalability to ten times number of the training robots. In addition, we demonstrate that our model trained for multirobot case can also improve the success rate in the single-robot navigation task in unseen environments. IEEE

Distribution System Operators (DSOs) are responsible preventing grid congestion, while accounting for growing demand and the intermittent nature of renewable energy resources. Incentive-based demand response programs promise real-time flexibility to relieve grid congestion. To include residential consumers in these programs, aggregators can financially incentivize participants to reduce their energy demand and make aggregated energy reduction available to DSOs. A key challenge for aggregators is to coordinate heterogeneous preferences from multiple participants while preserving their privacy. This thesis proposes MARL-iDR: a decentralized Multi-Agent Reinforcement Learning approach to an incentive-based demand response program. The approach respects participants' privacy and preferences and makes decisions in real-time when deployed. The aggregator and each participant are controlled by Deep Reinforcement Learning agents that learn to maximize their reward. The aggregator agent learns a policy that dispatches suitable incentives to participants based on total energy demand and a target reduction, while minimizing financial costs. The participant agent learns to respond to these incentives by reducing consumption to a fraction of the original demand. The participant agents curtail or shift requested household appliances based on the selected consumption reduction using a novel Disjunctively Constrained Knapsack Problem optimization, while minimizing residents' dissatisfaction. A case study with real-world electricity data from 25 households demonstrates the capability to induce demand-side flexibility. The approach is compared to the case without demand response and to a centralized myopic baseline approach. A 9% reduction of the Peak-to-Average ratio (PAR) was achieved compared to the original PAR (no demand response).

Brace sleeve (BS) fasteners, i.e., nut and bolt, are small components but play essential roles in fixing BS and cantilever in railway catenary system. They are commonly inspected by onboard cameras using computer vision to ensure the safety of railway operation. However, most BS fasteners cannot be directly localized because they are too small in the inspection images. Instead, the BS is first localized for detecting the BS fastener. This leads to a new problem that the localized BS boxes may not contain the complete BS fasteners due to low localization accuracy, making it infeasible to further diagnose the fastener conditions. To tackle this problem, this article proposes a novel pipeline for BS fastener looseness diagnosis. First, the competitive deep learning model Faster RCNN ResNet101 is used to coarsely localize BSs. Second, an action-driven reinforcement learning agent is adopted to refine the coarse-localized boxes through a dynamic position searching process. Then, BS fasteners are extracted from the refined localized BS image by the deep segmentation model YOLACT++, which is fast and interpretable. Finally, a looseness diagnosis criterion based on segmented information are proposed. We evaluate the performance of submodels independently and the overall performance of the whole model on a real-life catenary image dataset collected from a high-speed line in China. The test results show that the proposed method is effective for BS looseness detection in railway catenary., Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public., Railway Engineering

Seasonal thermal energy storage systems (STESSs) can shift the delivery of renewable energy sources and mitigate their uncertainty problems. However, to maximize the operational profit of STESSs and ensure their long-term profitability, control strategies that allow them to trade on wholesale electricity markets are required. While control strategies for STESSs have been proposed before, none of them addressed electricity market interaction and trading. In particular, due to the seasonal nature of STESSs, accounting for the long-term uncertainty in electricity prices has been very challenging. In this article, we develop the first control algorithms to control STESSs when interacting with different wholesale electricity markets. As different control solutions have different merits, we propose solutions based on model predictive control and solutions based on reinforcement learning. We show that this is critical since different markets require different control strategies: MPC strategies are better for day-ahead markets due to the flexibility of MPC, whereas reinforcement learning (RL) strategies are better for real-time markets because of fast computation times and better risk modeling. To study the proposed algorithms in a real-life setup, we consider a real STESS interacting with the day-ahead and imbalance markets in The Netherlands and Belgium. Based on the obtained results, we show that: 1) the developed controllers successfully maximize the profits of STESSs due to market trading and 2) the developed control strategies make STESSs important players in the energy transition: by optimally controlling STESSs and reacting to imbalances, STESSs help to reduce grid imbalances., Team DeSchutter