Search

This project’s goal was to assess both the challenges of implementing the Deep Q-Learning algorithm to create an autonomous car in the CARLA simulator, and the driving performance of the resulting model. An agent was trained to follow waypoints based on two main approaches. First, a camera-based approach, which allowed the agent to gather information about the environment from a camera sensor. The image along with other driving features were fed to a convolutional neural network. Second, an approach focused purely on following the waypoints without the camera sensor. The camera sensor was substituted for an array containing the agent’s angle with respect to the upcoming waypoints along with other driving features. Even though the camera-based approach was the best during evaluation, no approach was successful in consistently following the waypoints of a straight route. To increase the performance of the camera-based approach more training episodes need to be provided. Furthermore, both approaches would greatly benefit from experimentation and optimization of the model’s neural network configuration and its hyperparameters.

Many automatic Web testing techniques generate test cases by analyzing the GUI of the Web applications under test, aiming to exercise sequences of actions that are similar to the ones that testers could manually execute. However, the efficiency of the test generation process is severely limited by the cost of analyzing the content of the GUI screens after executing each action. In this paper, we introduce an inference component, S, which accumulates knowledge about the behavior of the GUI after each action. S enables the test generators to reuse the results computed for GUI screens that recur multiple times during the test generation process, thus improving the efficiency of Web testing techniques. We experimented S with Web testing techniques based on three different GUI exploration strategies (Random, Depth-first, and Q-learning) and nine target systems, observing reductions from 22% to 96% of the test generation time

With recent Industry 4.0 developments, companies tend to automate their industries. Warehousing companies also take part in this trend. A shuttle-based storage and retrieval system (SBS/RS) is an automated storage and retrieval system technology experiencing recent drastic market growth. This technology is mostly utilized in large distribution centers processing mini-loads. With the recent increase in e-commerce practices, fast delivery requirements with low volume orders have increased. SBS/RS provides ultrahigh-speed load handling due to having an excess amount of shuttles in the system. However, not only the physical design of an automated warehousing technology but also the design of operational system policies would help with fast handling targets. In this work, in an effort to increase the performance of an SBS/RS, we apply a machine learning (ML) (i.e., Q-learning) approach on a newly proposed tier-to-tier SBS/RS design, redesigned from a traditional tier-captive SBS/RS. The novelty of this paper is twofold: First, we propose a novel SBS/RS design where shuttles can travel between tiers in the system; second, due to the complexity of operation of shuttles in that newly proposed design, we implement an ML-based algorithm for transaction selection in that system. The ML-based solution is compared with traditional scheduling approaches: first-in-first-out and shortest process time (i.e., travel) scheduling rules. The results indicate that in most cases, the Q-learning approach performs better than the two static scheduling approaches.

This paper discusses how Q-Learning and Deep Q-Networks (DQN) canbe applied to state-action problems described by a Markov decision process(MDP). These are machine learning methods for finding the optimal choiceof action at each time step, resulting in the optimal policy. The limitationsand advantages for the two methods are discussed, with the main limitationbeing the fact that Q-learning is unable to be used on problems with infinitestate spaces. Q-learning, however, has an advantage in the simplicity of thealgorithm, leading to a better understanding of what the algorithm is actuallydoing. Q-Learning did manage to find the optimal policy for the simpleproblem studied in this paper, but was unable to do so for the advancedproblem. The Deep Q-Network (DQN) approach was able to solve bothproblems, with a drawback in it being harder to understand what the algorithmactually is doing.

Mask optimization is a vital step in the VLSI manufacturing process in advanced technology nodes. As one of the most representative techniques, optical proximity correction (OPC) is widely applied to enhance printability. Since conventional OPC methods consume prohibitive computational overhead, recent research has applied machine learning techniques for efficient mask optimization. However, existing discriminative learning models rely on a given dataset for supervised training, and generative learning models usually leverage a proxy optimization objective for end-to-end learning, which may limit the feasibility. In this article, we pioneer introducing the reinforcement learning (RL) model for mask optimization, which directly optimizes the preferred objective without leveraging a differentiable proxy. Intensive experiments show that our method outperforms state-of-the-art solutions, including academic approaches and commercial toolkits.

Platforms play an essential role in the modern economy. At the same time, due to advances in artificial intelligence (AI), algorithms are becoming more widely used for pricing and other business functions. Previous literature examined algorithmic pricing, but not in the context of network effects and platforms. Moreover, platform competition literature has not considered how algorithms may affect competition. We study the performance of AI pricing algorithms (Q-learning and Particle Swarm Optimization) and naïve algorithms (price-matching) under platform competition. We find that algorithms set an optimal price structure that internalizes network effects. However, no algorithm is always the best because profitability depends on the type of competing algorithms and market characteristics, such as differentiation and network effects. Additionally, algorithms learn autonomously when an equilibrium is unstable and avoid it. When algorithm adoption is an endogenous strategic decision, several algorithms can be adopted in equilibrium; we characterize the conditions for the various outcomes and show that the equilibrium and platform profits are sensitive to algorithm design changes. Overall, our research suggests that AI algorithms can be effective in the presence of network effects, and platforms are likely to adopt a variety of algorithms. Lastly, we reflect on the business value of AI and identify opportunities for future research at the intersection of AI algorithms and platforms., CRUE-CSIC, Depto. de Análisis Económico y Economía Cuantitativa, Fac. de Ciencias Económicas y Empresariales, TRUE, pub

The use of multiple mobile robots has grown significantly over the past few years in logistics, manufacturing and public services. Conflict-free route planning is one of the major research challenges for such mobile robots. Optimization methods such as graph search algorithms are used extensively to solve route planning problems. Those methods can assure the quality of solutions, however, they are not flexible to deal with unexpected situations. In this article, we propose a flexible route planning method that combines the reinforcement learning algorithm and a graph search algorithm for conflict-free route planning problems for multiple robots. In the proposed method, Q-learning, a reinforcement algorithm, is applied to avoid collisions using off-line learning with a limited state space to reduce the total learning time. Each vehicle independently finds the shortest route using the A* algorithm, and Q-learning is used to avoid collisions. The effectiveness of the proposed method is examined by comparing it with conventional methods in terms of computation time and the quality of solutions. Computational results show that for dynamic transportation problems, the proposed method can generate the solutions with approximately 10% of the computation time compared to the conventional Q-learning approach. We found that the required computation time is linearly increased with respect to the number of vehicles and nodes in the problems.

In recent years, reinforcement learning (RL) has emerged as a popular approach for solving sequence-based tasks in machine learning. However, finding suitable alternatives to RL remains an exciting and innovative research area. One such alternative that has garnered attention is the Non-Axiomatic Reasoning System (NARS), which is a general-purpose cognitive reasoning framework. In this paper, we delve into the potential of NARS as a substitute for RL in solving sequence-based tasks. To investigate this, we conduct a comparative analysis of the performance of ONA as an implementation of NARS and Q-Learning in various environments that were created using the Open AI gym. The environments have different difficulty levels, ranging from simple to complex. Our results demonstrate that NARS is a promising alternative to RL, with competitive performance in diverse environments, particularly in non-deterministic ones.

Network virtualization is a technology proven to be a key enabling a family of strategies in different targets, such as energy efficiency, economic revenue, network usage, adaptability or failure protection. Network virtualization allows us to adapt the needs of a network to new circumstances, resulting in greater flexibility. The allocation decisions of the demands onto the physical network resources impact the costs and the benefits. Therefore it is one of the major current problems, called virtual network embedding (VNE). Many algorithms have been proposed recently in the literature to solve the VNE problem for different targets. Due to the current successful rise of artificial intelligence, it has been widely used recently to solve technological problems. In this context, this paper investigates the requirements and analyses the use of the Q-learning algorithm for energy-efficient VNE. The results achieved validate the strategy and show clear improvements in terms of cost/revenue and energy savings, compared to traditional algorithms., This work has been supported by the Agencia Estatal de Investigación of Ministerio de Ciencia e Innovación of Spain under project PID2019-108713RB-C51 MCIN/AEI/10.13039/501100011033., Peer Reviewed, Postprint (published version)

Nowadays, as the need for capacity continues to grow, entirely novel services are emerging. A solid cloud-network integrated infrastructure is necessary to supply these services in a real-time responsive, and scalable way. Due to their diverse characteristics and limited capacity, communication and computing resources must be collaboratively managed to unleash their full potential. Although several innovative methods have been proposed to orchestrate the resources, most ignored network resources or relaxed the network as a simple graph, focusing only on cloud resources. This paper fills the gap by studying the joint problem of communication and computing resource allocation, dubbed CCRA, including function placement and assignment, traffic prioritization, and path selection considering capacity constraints and quality requirements, to minimize total cost. We formulate the problem as a non-linear programming model and propose two approaches, dubbed B&B-CCRA and WF-CCRA, based on the Branch & Bound and Water-Filling algorithms to solve it when the system is fully known. Then, for partially known systems, a Double Deep Q-Learning (DDQL) architecture is designed. Numerical simulations show that B&CCRA optimally solves the problem, whereas WF-CCRA delivers near-optimal solutions in a substantially shorter time. Furthermore, it is demonstrated that DDQL-CCRA obtains near-optimal solutions in the absence of request-specific information.

Autonomous systems, or agents as they sometimes are called can be anything from drones, self-driving cars, or autonomous construction equipment. The systems are often given tasks of accomplishing missions in a group or more. This may require that they can work within the same area without colliding or disturbing other agents' tasks. There are several tools for planning and designing such systems, one of them being UPPAAL STRATEGO. Multi-agent planning (MAP) is about planning actions in optimal ways such that the agents can accomplish their mission efficiently. A method of doing this named MCRL, utilizes Q learning as the algorithm for finding an optimal plan. These plans then need to be verified to ensure that they can accomplish what a user intended within the allowed time, something that UPPAAL STRATEGO can do. This is because a Q-learning algorithm does not have a correctness guarantee. Using this method alleviates the state-explosion problem that exists with an increasing number of agents. Using UPPAAL STRATEGO it is also possible to acquire the best and worst-case execution time (BCET and WCET) and their corresponding traces. This thesis aims to obtain the BCET and WCET and their corresponding traces in the model.

The Internet of Things (IoT) is widely adopted across various fields due to its flexibility and low cost. Energy-harvesting Wireless Sensor Networks (WSNs) are becoming a building block of many IoT applications and provide a perpetual source of energy to power energy-constrained IoT devices. However, the dynamic and stochastic nature of the available harvested energy drives the need for adaptive energy management solutions. Duty cycling is among the most prominent adaptive approaches that help consolidate the effort of energy management solutions at the routing and application layers to ensure energy sustainability and, hence, continuous network operation. The IEEE 802.15.4 standard defines the physical layer and the Medium Access Control (MAC) sub-layer of low-data-rate wireless devices with limited energy consumption requirements. The MAC sub-layer’s functionalities include the scheduling of the duty cycle of individual devices. However, the scheduling of the duty cycle is left open to the industry. Various computational mechanisms are used to compute the duty cycle of IoT nodes to ensure optimal performance in energy sustainability and Quality of Service (QoS). Reinforcement Learning (RL) is the most employed mechanism in this context. The literature depicts various RL-based solutions to adjust the duty cycle of IoT devices to adapt to changes in the IoT environment. However, these solutions are usually tailored to specific scenarios or focus mainly on one aspect of the problem, namely QoS performance or energy limitation. This work proposes a generic adaptive duty cycling solution and evaluates its performance under different energy generation and traffic conditions. Moreover, it emphasizes the energy sustainability aspect while taking the QoS performance into account. While different approaches exist to achieve energy sustainability, Energy Neutral Operation (ENO)-based solutions provide the most prominent approach to ensure energy-sustainable performance. N

An effective transportation system must include intelligent traffic light regulation. An intelligent traffic light control system should dynamically respond to real-time traffic, unlike traditional traffic lights, which are typically operated using manual instructions. Q-reinforcement learning is a technology that is increasingly being applied to traffic light regulation, and recent experiments have yielded promising results. In this study, an adaptive traffic signal scheduling strategy is designed utilizing Q-learning (QL) to minimize the number of vehicles blocking an intersection.

This paper introduces an energy management system (EMS) aiming to minimize electricity operating costs using reinforcement learning (RL) with a linear function approximation. The proposed EMS uses a Q-learning with tile coding (QLTC) algorithm and is compared to a deterministic mixed-integer linear programming (MILP) with perfect forecast information. The comparison is performed using a case study on an industrial manufacturing company in the Netherlands, considering measured electricity consumption, PV generation, and wholesale electricity prices during one week of operation. The results show that the proposed EMS can adjust the prosumer's power consumption considering favorable prices. The electricity costs obtained using the QLTC algorithm are 99% close to those obtained with the MILP model. Furthermore, the results demonstrate that the QLTC model can generalize on previously learned control policies even in the case of missing data and can deploy actions 80% near to the MILP's optimal solution., Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public., Intelligent Electrical Power Grids, Electrical Sustainable Energy

Q-learning is an attractive option for GUI testing, allowing for sophisticated test generation strategies that learn and exploit effective GUI interactions. However, learning comprehensive models requires long test sessions. This issue is exacerbated by the needs of both testers, who might want to run multiple testing sessions to fine-tune the test strategy to their applications under test, and researchers, who might want to experiment with multiple alternative approaches. To address these concerns, this paper presents $GT^{PQL}$, a testing tool that supports GUI testing with a parallel deployment Q-learning, and that can be flexibly configured and extended with multiple state-space abstractions and Q-leaning variants.

Actor-critic (AC) cooperative multiagent reinforcement learning (MARL) over directed graphs is studied in this article. The goal of the agents in MARL is to maximize the globally averaged return in a distributed way, i.e., each agent can only exchange information with its neighboring agents. AC methods proposed in the literature require the communication graphs to be undirected and the weight matrices to be doubly stochastic (more precisely, the weight matrices are row stochastic and their expectation are column stochastic). Differently from these methods, we propose a distributed AC algorithm for MARL over directed graph with fixed topology that only requires the weight matrix to be row stochastic. Then, we also study the MARL over directed graphs (possibly not connected) with changing topologies, proposing a different distributed AC algorithm based on the push-sum protocol that only requires the weight matrices to be column stochastic. Convergence of the proposed algorithms is proven for linear function approximation of the action value function. Simulations are presented to demonstrate the effectiveness of the proposed algorithms., Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public., Team Bart De Schutter

Current building operations can be improved through smart predictive operation based on weather and use patterns in order to save energy with minimal impact on the building fabric and daily use. The existing literature has investigated implementations, and potential savings through combining with variable tariffs, however, this thesis addresses the issue of how different buildings differ in their suitability for such smart control. In this thesis, a digital twin of a school is created and adjusted to test differences in building fabric factors. These are combined with multiple Deep Reinforcement Learning (DRL) agents, which are trained to operate the schools more efficiently by controlling the heating set-point as well as natural ventilation of the buildings in order to save energy while maintaining comfort. The DRL agents vary in their ability to observe future weather as well as their internal network model architecture. The results show a high energy saving compared to a simple baseline, despite the few building controls available to the agents. In addition, some algorithms out-compete even rule-based controllers, which were tested as a stricter baseline. The results also confirm a theory revealed through the literature review, that buildings with higher energy input, storage and control have a larger potential for energy savings. Additionally, the types of DRL models used also greatly influences the agents’ ability to perform well, and generally more advanced models performed better. The findings can be used to access a building’s suitability for and potential benefits from such predictive smart control., Architecture, Urbanism and Building Sciences | Building Technology

This paper studies a sequential task offloading problem for a multiuser mobile edge computing (MEC) system. While most of the existing works consider static one-shot offloading optimization with fixed wireless channel conditions and fixed computational tasks, we consider a dynamic optimization approach, which embraces wireless channel fluctuations and random deep neural network (DNN) task arrivals over an infinite horizon. Specifically, we introduce a local CPU workload queue (WD-QSI) and an MEC server workload queue (MEC-QSI) to model the dynamic workload of DNN tasks at each WD and the MEC server, respectively. The transmit power and the partitioning of the local DNN task at each WD are dynamically determined based on the instantaneous channel conditions (to capture the transmission opportunities) and the instantaneous WD-QSI and MEC-QSI (to capture the dynamic urgency of the tasks) to minimize the average latency of the DNN tasks. The joint optimization can be formulated as an ergodic Markov decision process (MDP), in which the optimality condition is characterized by a centralized Bellman equation. However, the brute force solution of the MDP is not viable due to the curse of dimensionality as well as the requirement for knowledge of the global state information. To overcome these issues, we first decompose the MDP into multiple lower dimensional sub-MDPs, each of which can be associated with a WD or the MEC server. Next, we further develop a parametric online Q-learning algorithm, so that each sub-MDP is solved locally at its associated WD or the MEC server. The proposed solution is completely decentralized in the sense that the transmit power for sequential offloading and the DNN task partitioning can be determined based on the local channel state information (CSI) and the local WD-QSI at the WD only. Additionally, no prior knowledge of the distribution of the DNN task arrivals or the channel statistics will be needed for the MEC server. The proposed solution

As Moore's law slows down, computing systems must pivot towards higher energy efficiency to continue scaling performance. Reinforcement learning (RL) performs more adaptively than conventional methods in runtime power management under varied hardware configurations and varying software workloads. However, prior works on either model-free or model-based RL approaches face a non-negligible challenge: relearning the policies to adapt to the new environment is unacceptably time-consuming, especially when encountering significant variances in workloads or hardware configurations. Moreover, existing research on accelerating learning has focused on the speedup while largely ignoring the efficiency degradation of the results. In this paper, we present a smart transfer-enabled Q-learning (STQL) approach to boost the learning process and guarantee the learning efficiency through a contradiction checking mechanism, which wisely evicts inappropriate transferred knowledge. Experiments on realistic applications show that the proposed method can speed up the learning process to up to 2.3x and achieve a 6.2% energy-delay product (EDP) reduction compared to the state-of-the-art design. © 2023 EDAA.

The constrained shortest path ( CSP ) problem has wide applications in travel path planning, mobile video broadcasting and network routing. Existing works do not work well on large dynamic graphs and suffer from either ineffectiveness or low scalability issues. To overcome these issues, in this paper, we propose an efficient and effective solution framework, namely CSP_GS . The solution framework includes two key components: (1) the techniques to decompose a large CSP instance into multiple small sub-instances and (2) the developed learning model CSP_DQN to solve small CSP instances. The evaluation result on real road network graphs indicates that our approach CSP_GS performs well on large dynamic graphs by rather high quality and reasonable running time, and particularly adapt to significant graph changes even with broken edges. To the best of our knowledge, this is the first learning-based model to well solve the CSP problem on large dynamic graphs.

The IEEE 802.15.4 Time-Slotted Channel Hopping (TSCH) is widely used as a reliable, low-power, and low-cost communication technology for many industrial Internet-of-Things (IoT) networks. In many applications, Quality-of-Service (QoS) requirements are different for heterogeneous nodes, necessitating non-equal parameter settings per node. This results in a very large configuration space making space exploration complex and time-consuming. Moreover, network state and QoS requirements may change over time. Thus, run-time configuration mechanisms are needed for making decisions about proper node settings to consistently satisfy diverse and dynamic QoS requirements. In this paper, we propose a run-time decentralized self-optimization framework based on Deep Reinforcement Learning (DRL) for parameter configuration of a multi-hop TSCH network. DRL adopts neural networks as approximate functions to speed up the process of converging to QoS-satisfying configurations. Simulation results show that our proposed framework enables the network to use the right configuration settings according to the diverse QoS demands of different nodes. Moreover, it is shown that the convergence time of the learning framework is in the order of a few minutes which is acceptable for many IoT applications.

One of the significant aspects for enabling the intelligent behavior to the Unmanned Aerial Vehicles (UAVs) is by providing an algorithm for navigation through the dynamic and unseen environment. Therefore, to be autonomous, they need sensors to perceive their surroundings and utilize gathered information to decide which action to take. Having that in mind, in this paper, the authors designed the system for obstacle avoidance and also investigate the elements of the Markov decision process and their influence on each other. The flying mobile robot used within the considered problem is quadrotor type and has an integrated Lidar sensor which is utilized to detect obstacles. The sequential decision-making model based on Q-learning is trained within the MATLAB Simulink environment. The simulation results demonstrate that the UAV can navigate through the environment in most algorithm runs without colliding with surrounding obstacles.

The quest for identifying feasible routes holds immense significance in the realm of transportation, spanning a diverse range of applications, from logistics and emergency systems to taxis and public transport services. This research area offers multifaceted benefits, including optimising traffic management, maximising traffic flow, and reducing carbon emissions and fuel consumption. Extensive studies have been conducted to address this critical issue, with a primary focus on finding the shortest paths, while some of them incorporate various traffic conditions such as waiting times at traffic lights and traffic speeds on road segments. In this study, we direct our attention towards historical data sets that encapsulate individuals' route preferences, assuming they encompass all traffic conditions, real-time decisions and topological features. We acknowledge that the prevailing preferences during the recorded period serve as a guide for feasible routes. The study’s noteworthy contribution lies in our departure from analysing individual preferences and trajectory information, instead focusing solely on macro-level measurements of each road segment, such as traffic flow or traffic speed. These types of macro-level measurements are easier to collect compared to individual data sets. We propose an algorithm based on Q-learning, employing traffic measurements within a road network as positive attractive rewards for an agent. In short, observations from macro-level decisions will help us to determine optimal routes between any two points. Preliminary results demonstrate the agent’s ability to accurately identify the most feasible routes within a short training period.

Cyber-physical system security for electric distribution systems is critical. In direct switching attacks, often coordinated, attackers seek to toggle remote-controlled switches in the distribution network. Due to the typically radial operation, certain configurations may lead to outages and/or voltage violations. Existing optimization methods that model the interactions between the attacker and the power system operator (defender) assume knowledge of the attacker's parameters. This reduces their usability. Furthermore, the trend with coordinated cyberattack detection has been the use of centralized mechanisms, correlating data from dispersed security systems. This can be prone to single point failures. In this paper, novel mathematical models are presented for the attacker and the defender. The models do not assume any knowledge of the attacker's parameters by the defender. Instead, a machine learning (ML) technique implemented by a multi-agent system correlates detected attacks in a decentralized manner, predicting the targets of the attacker. Furthermore, agents learn optimal mitigation of the communication level through Q-learning. The learned attacker motive is also used by the defender to determine a new configuration of the distribution network. Simulations of the technique have been performed using the IEEE 123-Node Test Feeder. The simulation results validate the capability and performance of the algorithm.

This thesis introduces a new method, called Mixed Iteration, for controlling Markov Decision Processes when partial information is known about the dynamics of the Markov Decision Process. The algorithm uses sampling to calculate the expectation of partially known dynamics in stochastic environments. Its goal is to lower the number of iterations and computational steps required for convergence compared to traditional model-free algorithms. By lowering the number of samples required to achieve convergence Markov Decision Processes can be controlled and trained more efficiently. Additionally, the thesis discusses how this algorithm can enhance the sample efficiency and convergence rate of Reinforcement Learning algorithms like Q-Learning. The effectiveness of the proposed method will be evaluated in standard Reinforcement Learning problems and compared with the performance of Q-learning. The results show that under certain conditions that will be discussed in the thesis, the new proposed algorithm outperforms classical algorithms in terms of sample efficiency. The study will provide insight into the field of previous partial information in Reinforcement Learning alternatives, as well as the challenges that researchers in this field continue to face., Mechanical Engineering | Systems and Control

Recent success stories in reinforcement learning have demonstrated that leveraging structural properties of the underlying environment is key in devising viable methods capable of solving complex tasks. We study off-policy learning in discounted reinforcement learning, where some equivalence relation in the environment exists. We introduce a new model-free algorithm, called QL-ES (Q-learning with equivalence structure), which is a variant of (asynchronous) Q-learning tailored to exploit the equivalence structure in the MDP. We report a non-asymptotic PAC-type sample complexity bound for QL-ES, thereby establishing its sample efficiency. This bound also allows us to quantify the superiority of QL-ES over Q-learning analytically, which shows that the theoretical gain in some domains can be massive. We report extensive numerical experiments demonstrating that QL-ES converges significantly faster than (structure-oblivious) Q-learning empirically. They imply that the empirical performance gain obtained by exploiting the equivalence structure could be massive, even in simple domains. To the best of our knowledge, QL-ES is the first provably efficient model-free algorithm to exploit the equivalence structure in finite MDPs.

With the emerging monitoring technologies, condition-based maintenance is nowadays a reality for the wind energy industry. This is important to avoid unnecessary maintenance actions, which increase the operation and maintenance costs, along with the costs associated with downtime. However, condition-based maintenance requires a policy to transform system conditions into decision-making while considering monetary restrictions and energy productivity objectives. To address this challenge, an intelligent Petri net algorithm has been created and applied to model and optimize offshore wind turbines’ operation and maintenance. The proposed method combines advanced Petri net modelling with Reinforcement Learning and is formulated in a general manner so it can be applied to optimize any Petri net model. The resulting methodology is applied to a case study considering the operation and maintenance of a wind turbine using operation and degradation data. The results show that the proposed method is capable to reach optimal condition-based maintenance policy considering maximum availability (equal to 99.4%) and minimal operational costs.

This paper considers optimal traffic signal control in smart cities, which has been taken as a complex networked system control problem. Given the interacting dynamics among traffic lights and road networks, attaining controller adaptivity and scalability stands out as a primary challenge. Capturing the spatial-temporal correlation among traffic lights under the framework of Multi-Agent Reinforcement Learning (MARL) is a promising solution. Nevertheless, existing MARL algorithms ignore effective information aggregation which is fundamental for improving the learning capacity of decentralized agents. In this paper, we design a new decentralized control architecture with improved environmental observability to capture the spatial-temporal correlation. Specifically, we first develop a topology-aware information aggregation strategy to extract correlation-related information from unstructured data gathered in the road network. Particularly, we transfer the road network topology into a graph shift operator by forming a diffusion process on the topology, which subsequently facilitates the construction of graph signals. A diffusion convolution module is developed, forming a new MARL algorithm, which endows agents with the capabilities of graph learning. Extensive experiments based on both synthetic and real-world datasets verify that our proposal outperforms existing decentralized algorithms. IEEE

This master's thesis is focused on applying offline reinforcement learning techniques to the ambulance dispatch problem, which involves selecting the most appropriate ambulance to dispatch when an incident occurs. The research is part of the SIA-REMU project, conducted both in France and Switzerland. An incident has an associated priority level. Incidents of priority 0 are vital emergencies, incidents of priority 1 are non-vital emergencies, and incidents of priority 2 are non-emergencies. The primary objective of the presented work is to train reinforcement learning agents capable of prioritizing incidents appropriately when dispatching ambulances. Initially, a dataset of experiences was constructed using data provided by the Centre de régulation du Centre Hospitalier Universitaire Vaudois (CHUV), which contains valuable information about incidents, interventions, and resources. This dataset of experiences served as a static dataset for training reinforcement learning agents in an offline setting, without interacting with an environment. State-of-the-art offline reinforcement learning algorithms were employed to train the agents, and their hyperparameters were tuned performing a Random Search. To evaluate and test the trained agents, a virtual environment was implemented. Finally, the policies learned by the agents were analyzed to draw meaningful conclusions from the obtained results.

This paper presents a novel Q-Learning method in forwarding the proximity sensing information to the remote server through low-power mesh network overlays on Bluetooth Low Energy (BLE) technology. Even though proximity sensing information can be easily monitored with pervasive smartphones, it is almost impossible to remotely monitor this information in a harsh location where it is not easy to access the Internet. With our overlay mesh network, each node should decide to either forward the packet or continue with their own activity when receiving the packet forwarding request, so as to minimize the end-to-end packet delivery latency but maximize the utilization of underlying infrastructures. Reinforcement learning (RL) is employed to train each node to make the above decision. Despite extensive upfront training, there is a high possibility that each node might still encounter an unseen state owing to the network dynamics. However, our novel Q-learning is able to deal with above challenges by constructing a Q-table during online learning, and then use the Q-table as input data for offline training. The experimental results indicate the substantial performance gain of our proposed approach in comparison to the existing Q-learning methods. IEEE

A novel fuzzy energy management strategy (EMS) based on improved Q-learning controller and genetic algorithm (GA) is proposed for the real-time power split between fuel cell and supercapacitor of hybrid electric vehicle (HEV). Different from driving pattern recognition–based method, Q-Learning controller takes actions by observing the driving states and compensates to fuzzy controller, that is, no need to know the driving pattern in advance. Aimed to prolong the fuel cell lifetime and decrease its energy consumption, the initial values of Q-table are optimized by GA. Moreover, to enhance the environment adaptation capability, the learning strategy of Q-learning controller is improved. Two adaptive energy management strategies have been compared, and simulation results show that current fluctuation can be reduced by 6.9% and 41.5%, and H2 consumption can be saved by 0.35% and 6.08%, respectively. Meanwhile, state of charge (SOC) of supercapacitor is sustained within the desired safe range. © The Author(s) 2022.

The integration of pipeline energy storage in the control of a district heating system can lead to profit gain, for example by adjusting the electricity production of a combined heat and power (CHP) unit to the fluctuating electricity price. The uncertainty from the environment, the computational complexity of an accurate model, and the scarcity of placed sensors in a district heating system make the operational use of pipeline energy storage challenging. A vast majority of previous works determined a control strategy by a decomposition of a mixed-integer nonlinear model and significant simplifications. To mitigate consequential stability, feasibility, and computational complexity challenges, we model CHP economic dispatch as a Markov decision process. We use a reinforcement learning (RL) algorithm to estimate the system’s dynamics through interactions with the simulation environment. The RL approach is compared with a detailed nonlinear mathematical optimizer on day-ahead and real-time electricity markets and two district heating grid models. The proposed method achieves moderate profit impacted by environment stochasticity. The advantages of the RL approach are reflected in three aspects: stability, feasibility, and time scale flexibility. From this, it can be concluded that RL is a promising alternative for real-time control of complex, nonlinear industrial systems., Algorithmics

In the process of drilling wells to produce hydrocarbons, an exploration strategy is used to determine which wells should be drilled and in which order. This strategy is vital, as a suboptimal drilling sequence will lead to more expenses and fewer gains. Furthermore, the wells considered in most exploration strategies are geologically dependent. Thus, a realistic model of these dependencies will be beneficial and contribute to a more reliable optimal drilling strategy. Previous research has shown that modelling similarities between the geological properties of prospect wells in the same region and updating the drilling strategy dynamically after more information is available can add much value. However, the currently developed models are not realistic enough to use in practice. Previous models separated the process into two parts, first modelling the geological success with deterministic rewards and then applying dynamic programming to obtain the sequential drilling strategy. This thesis proposes a new model with the addition of flexible uncertainty and dependency models between the geological characteristics in the hydrocarbon volume calculation of a reservoir. For this purpose, non-parametric Bayesian networks in combination with copulae are used. This model and the input format of the required data align with how data is gathered in the oil and gas industry. The dependencies are modelled based on simple expert assessments. After the new model for the geological properties of prospect wells and their dependencies is constructed, a decision policy that hinges on the observations of previously drilled wells is constructed. Previously used dynamic programming becomes infeasible with continuous variables and with a large number of wells. Therefore, the reinforcement learning algorithm Q-learning is used, which is better equipped to handle these more complex models. The analysis shows that Q-learning achieves similar results to dynam, Applied Mathematics

Robust model fitting is a core algorithm in several computer vision applications. Despite being studied for decades, solving this problem efficiently for datasets that are heavily contaminated by outliers is still challenging: due to the underlying computational complexity. A recent focus has been on learning-based algorithms. However, most of these approaches are supervised (which require a large amount of labelled training data). In this paper, we introduce a novel unsupervised learning framework: that learns to directly (without labelled data) solve robust model fitting. Moreover, unlike other learning-based methods, our work is agnostic to the underlying input features, and can be easily generalized to a wide variety of LP-type problems with quasi-convex residuals. We empirically show that our method outperforms existing (un)supervised learning approaches, and also achieves competitive results compared to traditional (non-learning-based) methods. Our approach is designed to try to maximise consensus (MaxCon), similar to the popular RANSAC. The basis of our approach, is to adopt a Reinforcement Learning framework. This requires designing appropriate reward functions, and state encodings. We provide a family of reward functions, tunable by choice of a parameter. We also investigate the application of different basic and enhanced Q-learning components.

Many emergency departments are today suffering from a overcrowding of people seeking care. The first stage in seeking care is being prioritised in different orders depending on symptoms by a doctor or nurse called medical triage. This is a cumbersome process that could be subject of automatisation. This master thesis investigates the possibility of using reinforcement learning for performing medical triage of patients. A deep Q-learning approach is taken for designing the agent for the environment together with the two extensions of using double Q-learning and a duelling network architecture. The agent is deployed to train in two different environments. The goal for the agent in the first environment is to ask questions to a patient and then decide, when enough information has been collected, how the patient should be prioritised. The second environment makes the agent decide which questions should be asked to the patient and then a separate classifier is used with the information gained to perform the actual triage decision of the patient. The training and testing process of the agent in the two environments reveal difficulties in exploring the environment efficiently and thoroughly. It was also shown that defining a reward function for the environments that guides the agent into asking valuable questions and learninga stopping condition for asking questions is a complicated task. Suitable future work is discussed that would, in combination with the work performed in this paper, create a better reinforcement learning model that could potentially show more promising results in the task of performing medical triage of patients.

Monitoring waterbirds have several benefits, including analyzing the number of endangered species, giving a reliable indication of public health, etc. Monitoring waterbirds in their habitat is a challenging task since the location is distant, and the collection of monitoring data requires large bandwidth. A promising technology to tackle these challenges is thought to be Wireless Multimedia Sensor Networks (WMSN). These networks are composed of small energy-constrained IoT devices that communicate together to collect data or monitor a given location. Performances in such networks are impacted by not only upper-layer protocols (transmission, routing, application layer) but also Medium Access Control (MAC) Layer. Therefore, improvement in this layer can increase the performance considerably. Traditional contention-based MAC modes like CSMA have large energy expenditure even though they have a good network performance profile. Energy-constrained devices cannot have a long lifespan with this type of MAC layer technology. Therefore, the IEEE 802.15.4e amendment proposed TSCH MAC mode which takes advantage of time-slotted access and channel hopping techniques. IETF integrated TSCH protocol into IPv6-based wireless sensor networks and standardized it as 6TiSCH which is a unique protocol stack for Low-Power and Lossy Networks (LLN). WMSN applications (e.g. Waterbirds Monitoring Application) generates heterogeneous traffic. Heterogeneous traffic can be defined as a mixture of different traffic types (light: temperature, humidity, etc. and heavy: audio, picture, video, etc.). TSCH-based WMSNs are considered a fit for this kind of traffic since they provide better performance and low power usage. Yet, the 6TiSCH Working Group left open the scheduling of TSCH communication for industries to make TSCH more easily adaptable to any kind of application. Until now, there have been a huge number of scheduling algorithms from industries and academia. Each scheduling algorithm has a d

Market making – the process of simultaneously and continuously providing buy and sell prices in a financial asset – is rather complicated to optimize. Applying reinforcement learning (RL) to infer optimal market making strategies is a relatively uncharted and novel research area. Most published articles in the field are notably opaque concerning most aspects, including precise methods, parameters, and results. This thesis attempts to explore and shed some light on the techniques, problem formulations, algorithms, and hyperparameters used to construct RL-derived strategies for market making. First, a simple probabilistic model of a limit order book is used to compare analytical and RL-derived strategies. Second, a market making agent is trained on a more complex Markov chain model of a limit order book using tabular Q-learning and deep reinforcement learning with double deep Q-learning. Results and strategies are analyzed, compared, and discussed. Finally, we propose some exciting extensions and directions for future work in this research field., Likviditetsgarantering (eng. ”market making”) – processen att simultant och kontinuerligt kvotera köp- och säljpriser i en finansiell tillgång – är förhållandevis komplicerat att optimera. Att använda förstärkningsinlärning (eng. ”reinforcement learning”) för att härleda optimala strategier för likviditetsgarantering är ett relativt outrett och nytt forskningsområde. De flesta publicerade artiklarna inom området är anmärkningsvärt återhållsamma gällande detaljer om de tekniker, problemformuleringar, algoritmer och hyperparametrar som används för att framställa förstärkningsinlärningsbaserade strategier. I detta examensarbete så gör vi ett försök på att utforska och bringa klarhet över dessa punkter. Först används en rudimentär probabilistisk modell av en limitorderbok som underlag för att jämföra analytiska och förstärkningsinlärda strategier. Därefter brukas en mer sofistikerad Markovkedjemodell av en limitorderbok för att jämföra tabulära och djupa inlärningsmetoder. Till sist presenteras även spännande utökningar och direktiv för framtida arbeten inom området.

In the context of Industry 4.0, the optimization of manufacturing processes is a challenge. Although in recent years many of the efforts have been in this direction, there is still improvement opportunities in these processes. The optimisation of the power consumed by the processes can be improved by means of the parameters of control. To date, this challenge has been addressed by Multi-Objective optimization techniques, however, Reinforcement Learning based approaches are raising with promising results in many industrial fields.In this paper, we propose a Reinforcement Learning (RL) based approach to optimize the active power consumption of a machining process by the cutting conditions selection. Through the application of Q-Learning algorithm, the agent self-learns the optimal solution through interacting with the environment. The approach was validated in three different scenarios demonstrating the feasibility of RL application to determine the cutting conditions values in order to optimize the active power consumption.

Reducing waiting time due to scheduling process and exploiting multi-access transmission, grant-free non-orthogonal multiple access (GF-NOMA) has been considered as a promising access technology for URLLC-enabled 5G system with strict requirements on reliability and latency. However, GF-NOMAbased systems can suffer from severe interference caused by the grant-free (GF) access manner which may degrade the system performance and violate the URLLC-related requirements. To overcome this issue, the paper proposes a novel reinforcementlearning (RL)-based random access (RA) protocol based on which each device can learn from the previous decision and its corresponding performance to select the best subchannels and transmit power level for data transmission to avoid strong cross-interference. The learning-based framework is developed to maximize the system access efficiency which is defined as the ratio between the number of successful transmissions and the number of subchannels. Simulation results show that our proposed framework can improve the system access efficiency significantly in overloaded scenarios.

Aquest Treball de Fi de Grau deu el seu nom al seu objectiu principal: abordar l’estudi de la incorporació del Reinforcement Learning als trencaclosques que integren el contingut del treball. En aquest paradigma, els programes aprenen a resoldre problemes matemàticament complexos a partir d'estímuls senzills relacionats amb els seus èxits. Per fer possible el desenvolupament de l’estudi, s'ha implementat la tècnica del Reinforcement Learning com a base i s’ha adaptat a la naturalesa de cada un dels jocs que es tracten. Es fa servir el Q-Learning per a aquells trencaclosques que són possibles d'abordar per la seva capacitat mínimament limitada. En canvi, es contempla el cas d’un joc que no segueix aquesta tendència per marcar una dinàmica massa àmplia. En aquest, s'adaptarà la tècnica de Deep Q-Learning per abordar-ho amb una xarxa neuronal que faciliti la manejabilitat dels estats que formen el trencaclosques. Els resultats que s’han obtingut en aquest estudi reflecteixen que l’aprenentatge d’un model mitjançant estímuls i recompenses en els jocs basats en procediments i regles és totalment assumible. Això s’evidencia ja que l'agent no només aprèn els requisits del joc en qüestió sinó que proporciona la solució òptima respecte a l'elecció de la solució més curta i, per tant, de la minimització d'eleccions errònies. Finalment cal dir que els objectius d’aquest estudi s’han aconseguit satisfactòriament i per comprovar l’aprenentatge, es realitza una comparació del comportament humà amb el del model. Així doncs, s'evidencia la bona integració del camp dels jocs amb la intel·ligència artificial a causa de les regles que regeixen el model de comportament de l’agent., This Final Degree Project owes its name to its main o bjective: einforcement L earning to address the study of the incorporation of Reinforcement Learning in the puzzles that make up the content of the project. In this paradigm, programs learn to solve mathematically complex problems from simple stimuli related to their achievements.To make the development of the project possible, the Reinforcement Learning technique has been implemented as a basis and has been adapted to the nature of each of the games in question. Q Learning is used for those puzzles that are possibl e to address due to their minimally limited capacity. In contrast, the case of a game that does not follow this trend due to have a dynamic that is too broad. In this one, the Deep Qadapted to address it with a neural network th Learning technique will be at facilitates the manageability of the states that make up the game. The results obtained in this project show that model learning through stimuli and rewards in games based on procedures and rules is fully achievable. This is evident since the agent not only learns the requirements of the game in question but also provides the optimal solution regarding the choice of the shortest solution and, therefore, the minimization of erroneous choices. Finally, the objectives of this study have been satisfactoril y achieved and to verify learning, a comparison of human behavior with that of the model is made. Thus, the good integration of the field of games with artificial intelligence is evident due to the rules that govern the behavior model of the agent. II

Machine-type communications (MTC) should account for half the connections to the internet by 2030. The use case massive MTC (mMTC) allows for applications to connect a massive number of low-power and low-complexity devices, leading to challenges in resource allocation. Not only that, mMTC networks suffer under rigid random access schemes due to mMTC ultra-dense nature resulting in poor performance. In this sense, this paper proposes a Q -Learning-based random access method for massive machine-type communications, with device clustering and non-orthogonal multiple access (NOMA). The traditional NOMA implementation increases spectral efficiency, but at the same time, demands a larger Q -Table, thus slowing down convergence, which is known to be a highly detrimental effect on massive networks. We use pre-clustering through short-range device-to-device technology to mitigate this drawback, allowing devices to operate with a smaller Q -Table. Furthermore, the previous selection of partner devices allows us to implement a full-feedback-based reward mechanism so that clusters avoid time slots already successfully allocated. Additionally, to cope with the negative impact of system overload, we propose an adaptive frame size algorithm to run in the base station (BS). It allows adjusting the frame size to the network load, preventing idle slots in an underloaded scenario, and providing extra slots when the network is overloaded. The results show the great benefits in terms of throughput of the proposed method. In addition, the impact of the use of clustering and the size of the clusters, as well as the frame size adaptation, are analyzed.

2022 International Balkan Conference on Communications and Networking, BalkanCom 2022 -- 22 August 2022 through 24 August 2022 -- 183326, In this study, three-dimensional optimal trajectory planning is performed for the Unmanned Aerial Vehicle (UAV) with the base station (BS) attached to it to increase the service provided to the users. The case that heterogeneous quality of service (QoS) requirements for different users are considered. While planning the trajectory, the coverage area of the UAV-BS and backhaul capacity between the UAV-BS and Ground Base Station (GBS) are limited. Under these constraints, the aim is to find a trajectory for the UAV-BS that maximizes the total data rate provided to the users during the flight using reinforcement learning. With the application of Q-learning in our problem, the UAV-BS learns to take action to achieve the desired goal. As a result of trial and error processes with different learning parameters, appropriate parameters are determined and a reinforcement learning model is trained. Different communication scenarios are compared for analyzing the effects of the constraints. According to the effects of the mentioned constraints and heterogeneous QoS demands, UAV-BS's trajectory preferences and total transmission rate changes are examined. Three prominent results shows the effects of coverage, backhaul, and heterogeneous QoS. The UAV-BS tends to increase its altitude as the coverage constraint increases. Moreover, the backhaul constraint forces the UAV-BS's trajectory closer to the GBS. Lastly, UAV-BS takes into account different QoS requirements of users as much as possible. UAV-BS maximizes the total transmission rate by determining the most suitable trajectory to meet these constraints. © 2022 IEEE.

NSF, 2021 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2021 -- 9 December 2021 through 12 December 2021 -- -- 176400, The concept of generating molecular structures with specific desirable characteristics underlies some of the crucial problems in drug discovery. In this paper, we present a model which constructs new molecules for specific desired properties. This model uses graph neural networks to generate molecular representations, and combining these representations with Reinforcement Learning architecture (Deep Q-Learning), builds new molecules. We used two different graph neural network architectures: Graph Convolutional Network and Graph Attention Network. We compared the molecular representations obtained from these two models with the Morgan Fingerprint representation in three separate experiments using the same Reinforcement Learning design. These experiments are single-objective optimization (drug-likeness), optimizing the penalized LogP with similarity constraint, and multi-objective optimization (drug-likeness with similarity constraint). The results show that the Reinforcement Learning models trained with molecular representations obtained from graph neural networks are more successful than the model trained with Morgan Fingerprint representation. © 2021 IEEE., Türkiye Bilimsel ve Teknolojik Araştirma Kurumu, TÜBITAK: 118E759, This study is partially funded by The Scientific and Technological Research Council of Turkey (Grant No : 118E759)

We present an approach to reduce the communication of information needed on a Distributed Q-Learning system inspired by Event Triggered Control (ETC) techniques. We consider a baseline scenario of a Distributed Q-Learning problem on a Markov Decision Process (MDP). Following an event-based approach, N agents sharing a value function explore the MDP and compute a trajectory-dependent triggering signal which they use distributedly to decide when to communicate information to a central learner in charge of computing updates on the action-value function. These decision functions form an Event Based distributed Q learning system (EBd-Q), and we derive convergence guarantees resulting from the reduction of communication. We then apply the proposed algorithm to a cooperative path planning problem, and show how the agents are able to learn optimal trajectories communicating a fraction of the information. Additionally, we discuss what effects (desired and undesired) these event-based approaches have on the learning processes studied, and how they can be applied to more complex multi-agent systems., Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public., Team Manuel Mazo Jr

In this work, we consider the design of a radio resource management (RRM) solution for traffic steering (TS) use-case in the open radio access network (O-RAN). The O-RAN TS deals with the quality-of-service (QoS)-aware steering of the traffic by connectivity management (e.g., device-to-cell association, radio spectrum, and power allocation) for emerging heterogeneous networks (HetNets) in 5G-and-beyond systems. However, TS in HetNets is a complex problem in terms of efficiently assigning/utilizing the radio resources while satisfying the diverse QoS requirements of especially the cell-edge users due to their poor signal-to-interference-plus-noise ratio (SINR). In this respect, we propose an intelligent non-orthogonal multiple access (NOMA)-based RRM technique for a small cell base station (SBS) within a macro gNB. A Q-learning-assisted algorithm is designed to allocate the transmit power and frequency sub-bands at the O-RAN control layer such that interference from macro gNB to SBS devices is minimized while ensuring the QoS of the maximum number of devices. The numerical results show that the proposed method enhances the overall spectral efficiency of the NOMA-based TS use case without adding to the system's complexity or cost compared to traditional HetNet topologies such as co-channel deployments and dedicated channel deployments.

As important services of the future sixth-generation (6G) wireless networks, vehicular communication and mobile edge computing (MEC) have received considerable interest in recent years for their significant potential applications in intelligent transportation systems. However, MEC-enabled vehicular networks depend heavily on network access and communication infrastructure, often unavailable in remote areas, making computation offloading susceptible to breaking down. To address this issue, we propose an MEC-enabled vehicular network assisted through aerial-terrestrial connectivity to provide network access and high data-rate entertainment services to a vehicular network. We present a time-varying, dynamic system model where high altitude platforms (HAPs) equipped with MEC servers, connected to a backhaul system of low-earth orbit (LEO) satellites, are used to provide computation offloading capability to the vehicles, as well as to provide network access for vehicle-to-vehicle (V2V) communications. Our main objective is to minimize the total computation and communication overhead of the joint computation offloading and resource allocation strategies for the system of vehicles. Since our formulated optimization problem is a mixed-integer non-linear programming (MINLP) problem, which is NP-hard, we propose a decentralized value-iteration-based reinforcement learning (RL) approach as a solution. In our Q-learning-assisted analysis, each vehicle acts as an intelligent agent to form optimal strategies for offloading and resource allocation. We further extend our solution to deep Q-learning (DQL) and double deep Q-learning to overcome the issues of dimensionality and the over-estimation of the value functions, as in Q-learning. Simulation results prove the effectiveness of our solution in successfully reducing system costs compared to baseline schemes.

In recent years, the development of deep learning models that can generate more accurate predictions and operate in real-time has brought both opportunities and challenges across the various domains of robotic vision. This breakthrough enables researchers to design and deploy more challenging tasks on intelligent mobile robots, which require emphasized abilities of learning and reasoning. In this paper, a new method for intelligent robot control, based on deep learning and reinforcement learning is proposed. The fundamental idea of this work is how the UAV equipped with a monocular camera can learn significant information about the object of interest in the context of its localization and navigation. For such purpose, the object detection system based on Tiny YOLOv2 architecture is employed. Furthermore, bounding box data generated by a convolution neural network is utilized for depth estimation and determining object boundaries. This information has shown how the state-space dimensions can be significantly reduced, which was essential for further implementation of the Q-learning algorithm. In order to test the proposed framework, a model is developed in MATLAB Simulink. The simulation, which covered different scenarios, was carried out on the UAV within the 3D scene rendered by Unreal Engine. The obtained results have demonstrated the applicability of the proposed methodology for depth estimation, gathering information about the object, object-driven navigation, and autonomous localization and navigation.