Search

In inverse reinforcement learning (RL), there are two agents. An expert target agent has a performance cost function and exhibits control and state behaviors to a learner. The learner agent does not know the expert's performance cost function but seeks to reconstruct it by observing the expert's behaviors and tries to imitate these behaviors optimally by its own response. In this article, we formulate an imitation problem where the optimal performance intent of a discrete-time (DT) expert target agent is unknown to a DT Learner agent. Using only the observed expert's behavior trajectory, the learner seeks to determine a cost function that yields the same optimal feedback gain as the expert's, and thus, imitates the optimal response of the expert. We develop an inverse RL approach with a new scheme to solve the behavior imitation problem. The approach consists of a cost function update based on an extension of RL policy iteration and inverse optimal control, and a control policy update based on optimal control. Then, under this scheme, we develop an inverse reinforcement Q-learning algorithm, which is an extension of RL Q-learning. This algorithm does not require any knowledge of agent dynamics. Proofs of stability, convergence, and optimality are given. A key property about the nonunique solution is also shown. Finally, simulation experiments are presented to show the effectiveness of the new approach.

Cyber-physical system security for electric distribution systems is critical. In direct switching attacks, often coordinated, attackers seek to toggle remote-controlled switches in the distribution network. Due to the typically radial operation, certain configurations may lead to outages and/or voltage violations. Existing optimization methods that model the interactions between the attacker and the power system operator (defender) assume knowledge of the attacker's parameters. This reduces their usability. Furthermore, the trend with coordinated cyberattack detection has been the use of centralized mechanisms, correlating data from dispersed security systems. This can be prone to single point failures. In this paper, novel mathematical models are presented for the attacker and the defender. The models do not assume any knowledge of the attacker's parameters by the defender. Instead, a machine learning (ML) technique implemented by a multi-agent system correlates detected attacks in a decentralized manner, predicting the targets of the attacker. Furthermore, agents learn optimal mitigation of the communication level through Q-learning. The learned attacker motive is also used by the defender to determine a new configuration of the distribution network. Simulations of the technique have been performed using the IEEE 123-Node Test Feeder. The simulation results validate the capability and performance of the algorithm. U.S. National Science Foundation [1837359]; Commonwealth Cyber Initiative (CCI), State of Virginia [PR6ZBSTA, 5GPG] Published version This work was supported in part by the U.S. National Science Foundation under Grant 1837359; and in part by the projects under Grant PR6ZBSTA and Grant 5GPG sponsored by Commonwealth Cyber Initiative (CCI), State of Virginia.

Given the rapid growth of user-generated videos, internet traffic has been heavily dominated by online video streaming. Caching videos on edge servers in close proximity to users has been an effective approach to reduce the backbone traffic and the request response time, as well as to improve the video quality on the user side. Video popularity, however, can be highly dynamic over time. The cost of cache replacement at edge servers, particularly that related to service interruption during replacement, is not yet well understood. This paper presents a novel lightweight video caching algorithm for edge servers, seeking to optimize the hit rate with real-time decisions and minimized cost. Inspired by recent advances in deep Q-learning, our DQN-based online video caching (DQN-OVC) makes effective use of the rich and readily available information from users and networks. We decompose the Q-value function as a product of the video value function and the action function, which significantly reduces the state space. We instantiate the action function for cost-aware caching decisions with low complexity so that the cached videos can be updated continuously and instantly with dynamic video popularity. We used video traces from Tencent, one of the largest online video providers in China, to evaluate the performance of our DQN-OVC and to compare it with state-of-the-art solutions. The results demonstrate that DQN-OVC significantly outperforms the baseline algorithms in the edge caching context.

In B5G heterogeneous cellular networks, a rapid increase in the number of small cell base stations (SBSs) to support a massive number of devices tends to waste a considerable amount of energy. Therefore, intelligent management of SBSs’ power consumption is one of the most important research issues. We herein propose quasi-distributed Q-learning-based cell breathing (QD-QCB) considering full and partial SBS collaborations for maximizing network energy efficiency. Also, the concept of an aggregated active SBS set based on regional user distributions is proposed for computing- and energy-efficient operation. Through intensive simulations, we show that the proposed QD-QCB algorithm can achieve optimal energy efficiency, and improve the network energy efficiency significantly compared with conventional algorithms such as no transmit power control, random cell breathing, and greedy cell breathing algorithms.

The incorporation of advanced networking technologies makes modern systems vulnerable to cyber-attacks that can result in a number of harmful outcomes. Due to the increase of security incidents and massive activities on networks, existing works have mainly focused on designing Intrusion Detection Systems (IDSs) based on traditional machine learning and deep learning models. In recent times, state of the art performance has been achieved in various fields through Deep Reinforcement Learning (DRL), which combines deep learning with reinforcement learning. In this paper, we propose a new DRL-based IDS for network traffics using Markov decision process (MDP) to improve the IDS decision-making performance. In addition, an extensive analysis of the IDS behavior is provided through modeling the interaction between the well-behaving IDS and attacker players using Stochastic Game Theory. Specifically, we used a non-zero-sum stochastic game, where the transitions between states depend on both the IDS and the attacker's actions at each stage of the game. We show that our game reaches a Nash Equilibrium upon convergence to seek the optimal solution, which corresponds to the optimal decision policy where both players maximize their profits. We compared the performance of our proposed DRL-IDS to the baseline benchmark of standard reinforcement learning (RL) and several machine learning algorithms using NSL-KDD dataset. As a result, our proposed DRL-IDS outperforms the existing models by improving both the detection rate and the accuracy while reducing false alarms. Results were provided to demonstrate the convergence of the game theory-based IDS under various settings toward equilibrium. This equilibrium corresponds to the safe state where both players are playing their respective best strategies. 1967-2012 IEEE. Scopus

Artificial intelligence (AI) technology leads to new insights into the manipulation of quantum systems in the Noisy Intermediate-Scale Quantum (NISQ) era. Classical agent-based artificial intelligence algorithms provide a framework for the design or control of quantum systems. Traditional reinforcement learning methods are designed for the Markov Decision Process (MDP) and, hence, have difficulty in dealing with partially observable or quantum observable decision processes. Due to the difficulty of building or inferring a model of a specified quantum system, a model-free-based control approach is more practical and feasible than its counterpart of a model-based approach. In this work, we apply a model-free deep recurrent Q-network (DRQN) reinforcement learning method for qubit-based quantum circuit architecture design problems. This paper is the first attempt to solve the quantum circuit design problem from the recurrent reinforcement learning algorithm, while using discrete policy. Simulation results suggest that our long short-term memory (LSTM)-based DRQN method is able to learn quantum circuits for entangled Bell–Greenberger–Horne–Zeilinger (Bell–GHZ) states. However, since we also observe unstable learning curves in experiments, suggesting that the DRQN could be a promising method for AI-based quantum circuit design application, more investigation on the stability issue would be required.

International audience; We propose a solution method for online vehicle routing, which integrates a machine learning routine to improve tours’ quality. Our optimization model is based on the Bertsimas et al. (2019) re-optimization approach. Two separate routines are developed. The first one uses a neural network to produce realistic pick-up times for the customers to serve. The second one relies on Q-learning in addition to random walks for the construction of the backbone graph corresponding to the instance problem of each time step. The second routine gives improved results compared to the original approach.

As one of adaptive optimal controls, the Q-learning based supervisory control for hybrid electric vehicle (HEV) energy management is rarely studied for its adaptability. In real-world driving scenarios, conditions such as vehicle loads, road conditions and traffic conditions may vary. If these changes occur and the vehicle supervisory control does not adapt to it, the resulting fuel economy may not be optimal. To our best knowledge, for the first time, the study investigates the adaptability of Q-learning based supervisory control for HEVs. A comprehensive analysis is presented for the adaptability interpretation with three varying factors: driving cycle, vehicle load condition, and road grade. A parallel HEV architecture is considered and Q-learning is used as the reinforcement learning algorithm to control the torque split between the engine and the electric motor. Model Predictive Control, Equivalent consumption minimization strategy and thermostatic control strategy are implemented for comparison. The Q-learning based supervisory control shows strong adaptability under different conditions, and it leads the fuel economy among four supervisory controls in all three varying conditions.

With the rapid development of non-geostationary orbit (NGSO) constellation networks, the co-frequency interference between geostationary orbit (GSO) and NGSO systems is becoming increasingly inevitable. Due to the characteristics of complex multi-mission planning and the large spatial and temporal scale of NGSO constellations, it is difficult to maximize the system performance under the constraints of GSO networks' interference protections. In this paper, based on an iterative greedy algorithm, a modified Q-learning approach in reinforcement learning (RL) is proposed to upgrade the system performance of NGSO network. Compared with the traditional RL algorithms, this modified approach has the advantages of improving the system performance and meeting the aggregate interference protections of GSO networks, by redefining the actions, rewards, and Q-tables. Simulation results show that by adopting the proposed algorithm, all NGSO links timely select and flexibly adapt their actions, including access objects, transmission powers, communication rates, as well as modulation and coding schemes, to achieve more global rewards with co-frequency interference constraints.

In air-to-ground communications, the network lifetime depends on the operation time of unmanned aerial vehicle-base stations (UAV-BSs) owing to the restricted battery capacity. Therefore, the maximization of energy efficiency and the minimization of outage ground users are important metrics of network performance. To achieve these two objectives, the location and transmit power of the UAV-BSs in the network must be optimized. This optimization problem may not be tractable in the conventional optimization framework because multiple UAV-BSs interact in a complicated manner. Hence, we formulate the problem as a Markov decision process and develop an algorithm to obtain a solution in a reinforcement learning framework. To avoid a central controller and high computational complexity, we employ a multi-agent distributed Q-learning algorithm to obtain a solution. Specifically, we propose a multi-agent Q-learning-based UAV-BS deployment and power control strategy to maximize energy efficiency and minimize the number of outage users in multi-UAV wireless networks. Through intensive simulations, it is demonstrated that the proposed algorithm can outperform benchmark algorithms in terms of average energy efficiency and number of average outage users in multi-UAV wireless networks.

Online matching between idle drivers and waiting passengers is one of the most key components in a ride-sourcing system. It is naturally expected that a more effective bipartite matching can be implemented if the platform accumulates more idle drivers and waiting passengers in the matching pool. A specific passenger request can also benefit from a delayed matching since he/she may be matched with closer idle drivers after waiting for a few seconds. Motivated by the potential benefits of delayed matching, this paper establishes a two-stage framework which incorporates a combinatorial optimization and multi-agent deep reinforcement learning methods. The multi-agent reinforcement learning methods are used to dynamically determine the delayed time for each passenger request, while the combinatorial optimization conducts an optimal bipartite matching between idle drivers and waiting passengers in the matching pool. Four tailored reinforcement learning methods, delayed multi-agent deep Q learning (Delayed-M-DQN), delayed multi-agent actor-critic (Delayed-M-A2C), delayed multi-agent Proximal Policy Optimization (Delayed-M-PPO), and delayed multi-agent actor-critic with experience replay (Delayed-M-ACER), are developed. Through extensive empirical experiments with a well-designed simulator, we show that the proposed framework is able to remarkably improve system performances, by well balancing the trade-off among pick-up time, matching time, successful matching rate.

Deep Q-Learning is an important reinforcement learning algorithm, which involves training a deep neural network, called Deep Q-Network (DQN), to approximate the well-known Q-function. Although wildly successful under laboratory conditions, serious gaps between theory and practice as well as a lack of formal guarantees prevent its use in the real world. Adopting a dynamical systems perspective, we provide a theoretical analysis of a popular version of Deep Q-Learning under realistic and verifiable assumptions. More specifically, we prove an important result on the convergence of the algorithm, characterizing the asymptotic behavior of the learning process. Our result sheds light on hitherto unexplained properties of the algorithm and helps understand empirical observations, such as performance inconsistencies even after training. Unlike previous theories, our analysis accommodates state Markov processes with multiple stationary distributions. In spite of the focus on Deep Q-Learning, we believe that our theory may be applied to understand other deep learning algorithms, 4 figures

Owing to energy liberalization and increasing penetration of renewables, renewable energy trading among suppliers and users has gained much attention and created a new market. This article investigates a double-auction scheme operated by an aggregator with limited supervision for energy trading. To ensure beneficial bidding for renewable generators and end users (EUs) considered as agents, a multiagent Q-learning (MAQL) based bidding strategy is developed to maximize their cumulative reward. Each agent first provides their information about renewable supply or demand to an aggregator who will then return the information about aggregate supply and demand. Without knowing the business model of the aggregator, the agents use Q-tables to estimate the expected cumulative reward and determine their bidding prices accordingly. Finally, the aggregator coordinates energy trading between agents who will then update their Q-tables on the basis of the amount of power bought or sold at the prices they bid. The proposed approach can avoid some unnecessary or unrealistic assumptions generally made by model-based approaches, such as the assumption on the knowledge of others’ bidding profiles or the assumption of an oligopoly; it can consider the influence of bidding strategies on the market, which cannot be properly addressed by a conventional proportional allocation mechanism. A numerical analysis using real-world data and considering a profit maximization model for the aggregator shows that the proposed approach outperformed comparable methods in terms of profits of renewable generators and energy satisfaction level of EUs: iterative double auction by approximately 29%, heuristic bidding by 39.9%, random bidding by 38.1%, NSGA-II-based multiobjective approach by 62%, and MOEA/D-based multiobjective approach by 83.1% on average.

We prove that a single-layer neural network trained with the Q-learning algorithm converges in distribution to a random ordinary differential equation as the size of the model and the number of training steps become large. Analysis of the limit differential equation shows that it has a unique stationary solution that is the solution of the Bellman equation, thus giving the optimal control for the problem. In addition, we study the convergence of the limit differential equation to the stationary solution. As a by-product of our analysis, we obtain the limiting behavior of single-layer neural networks when trained on independent and identically distributed data with stochastic gradient descent under the widely used Xavier initialization.

Virtualized network function (VNF) orchestration dynamically deploys network slices, which provides an effective means of customized service provision. To achieve a realistic and comprehensive perspective of the decision process for customized service provision, we propose a virtualized resource orchestration strategy in radio access network (RAN) of Internet of things (Iots) based on chemical reaction optimization (CRO). Specifically, we apply particle swarm optimization (PSO), a Gaussian process, random walk model, and Q-learning to enhance the CRO algorithm to quickly obtain the approximate optimal solution for the proposed CRO-based resource orchestration strategy (CROROS). Simulation results show that, compared with existing access methods, CROROS can reduce the service rejection rate of a virtualized radio access network and improve the utilization rate of network system resources. Compared with other heuristic algorithms (e.g., PSO, genetic algorithm (GA), and CRO), CROROS can accelerate the global approximate optimal solution and improve the approximate fitness of the approximate optimal solution within a specified time.

Crowdsourcing is a hotspot research field which can facilitate machine learning by collecting labels to train models. Consequently, the state-of-the-art research efforts in crowdsourcing focus on truth inference or label integration, to remove inconsistent labels or to alleviate biased labeling. In turn, the integrated labels will be used to fine-tune machine learning models. Particularly, in this paper, we change the target of truth inference in crowdsourcing from discrete labels to multiple comments given by online participants, that is, the integration of the crowdsourced comments. For such a goal, we propose a Self-play and Sentiment-Emphasized Comment Integration Framework (SSECIF), based on deep Q-learning, with three unique features. First, our framework SSECIF can generate the comment integration in a totally self-play way, without relying on the ground truth generated by human effort. Second, the integrated comment generated by SSECIF can include salient content with low redundancy. Third, the proposed framework SSECIF has emphasized, with a higher intensity, the sentiment in the integrated comment, in order to reflect the attitude or opinion more obviously. Extensive evaluation on real-world datasets demonstrates that SSECIF has achieved the best overall performance in terms of both effectiveness and efficiency, compared with the state-of-the-art methods. Index Terms: Crowdsourcing; Comment Integration; Reinforcement Learning; Deep Q-Learning; Sentiment Analysis.

The performance of autonomous robots in varying environments needs to be improved. For such incremental improvement, here we propose an incremental learning framework based on Q-learning and the adaptive kernel linear (AKL) model. The AKL model is used for storing behavioral policies that are learned by Q-learning. Both the structure and parameters of the AKL model can be trained using a novel L2-norm kernel recursive least squares (L2-KRLS) algorithm. AKL model initially without nodes and gradually accumulates content. The proposed framework allows to learn new behaviors without forgetting the previous ones. A novel local -greedy policy is proposed to speed the convergence rate of Q-learning. It calculates the exploration probability of each state for generating and selecting more important training samples. The performance of our incremental learning framework was validated in two experiments. A curve fitting example shows that the L2-KRLS based AKL model is suitable for incremental learning. The second experiment is based on robot learning tasks. The results show that our framework can incrementally learn behaviors in varying environments. Local -greedy policy-based Q-learning is faster than existing Q-learning algorithms.

The energy-efficient train operation methodology is the focus of this paper, and a Q-Learning-based eco-driving approach is proposed. Firstly, the core idea of energy-distribution-based method (EDBM) that converts the eco-driving problem to the finite Markov decision process is presented. Secondly, Q-Learning approach is proposed to determine the optimal energy distribution policy. Specifically, two different state definitions, i.e., trip-time-relevant (TT) and energy-distribution-relevant (ED) state definitions, are introduced. Finally, the effectiveness of the proposed approach is verified in a deterministic and a stochastic environment. It is also illustrated that TT-state approach takes about 20 times more computation time compared with ED-state approach while the space complexity of TT-state approach is nearly constant. The hyperparameter sensitivity analysis demonstrates the robustness of the proposed approach.

In this article, we propose two model-free algorithms using state or output feedback for saturated discrete-time multiagent systems (SDTMASs) to attain global containment control. In most previous works, the control input can avoid saturation by utilizing the low gain feedback (LGF) method whereas requiring the knowledge of agent dynamics, and SDTMASs just can attain semi-global containment control. Distinct with the previous works, first, based on the Q-learning (QL) technique, this article defines a Q-function and deduces the corresponding QL Bellman equation, which is the most important part of the QL algorithm. Then, in order to solve the QL Bellman equation, we propose two iterative model-free algorithms using state and output feedback, and the LGF matrix can be acquired from that solution directly. Furthermore, under the state and output feedback control protocols with the feedback matrices obtained from the proposed model-free algorithms, the SDTMASs can achieve global containment control instead of semi-global containment control. Finally, we present some simulations to confirm the validity of the proposed algorithms.

Autoscaling strategies aim to exploit the elasticity, resource heterogeneity and varied prices options of a Cloud infrastructure to improve efficiency in the execution of resource-hungry applications such as scientific workflows. Scientific workflows represent a special type of Cloud application with task dependencies, high-performance computational requirements and fluctuating workloads. Hence, the amount and type of resources needed during workflow execution changes dynamically over time. The well-known autoscaling problem comprises (i) scaling decisions, for adjusting the computing capacity of a virtualized infrastructure to meet the current demand of the application and (ii) task scheduling decisions, for assigning tasks to specific acquired Cloud resources for execution. Both are highly complex sub-problems, even more because of the uncertainty inherent to the Cloud. Reinforcement Learning (RL) provides a solid framework for decision-making problems in stochastic environments. Therefore, RL offers a promising perspective for designing Cloud autoscaling strategies based on an online learning process. In this work, we propose a novel formulation for the problem of infrastructure scaling in the Cloud as a Markov Decision Process, and we use the Q-learning algorithm for learning scaling policies, while demonstrating that considering the specific characteristics of workflow applications when taking autoscaling decisions can lead to more efficient workflow executions. Thus, our RL-based scaling strategy exploits the information available about workflow dependency structures. Simulations performed on four well-known workflows demonstrate significant gains (25%–55%) of our proposal in comparison with a similar state-of-the-art proposal.

Flying ad hoc networks (FANETs) have emanated over the last few years for numerous civil and military applications. Owing to underlying attributes such as a dynamic topology, node mobility in three-dimensional (3D) space, and the limited energy of unmanned aerial vehicles (UAVs), a routing protocol for FANETs is challenging to design. Exiting topology-based routing is unsuitable for highly dynamic FANETs. Location-based routing protocols can be preferred for FANETs owing to their scalability, but are based on one-hop neighbor information and do not contemplate the reachability of further appropriate nodes for forwarding. Owing to the rapid mobility of UAVs, the topology frequently changes; thus, some route entries in the routing table can become invalid and the next-hop nodes may be unavailable before a timeout. That is, the routing decision based on one-hop neighbors cannot assure a successful delivery. In this study, we propose a novel Q-learning-based topology-aware routing (QTAR) protocol for FANETs to provide reliable combinations between the source and destination. The proposed QTAR improves the routing decision by considering two-hop neighbor nodes, extending the local view of the network topology. With the Q-learning technique, QTAR adaptively adjusts the routing decision according to the network condition. Our simulation results reveal that QTAR outstrips the existing routing protocols in respect of various performance metrics under distinct scenarios.

A reasonably designed energy management strategy (EMS) can guarantee the safe and stable operation of fuel cell hybrid electric vehicle (FCHEV). To optimize the energy conversion efficiency of FCHEV, this research proposes a three-level efficiency optimized EMS based on the dual reward functions Q-learning algorithm. Focused on the system overall efficiency, a hardware-in-the-loop experimental platform was built first to compare the effectiveness between the proposed EMS and other existed methods. The results shown that compared with other state-of-art methods, the proposed strategy can effectively improve the energy efficiency of the system, and can slow down the aging of the fuel cell by reducing its operating stress. To further verify the effectiveness of the proposed strategy, varying driving loads profile were tested based on a 1.2-kW hybrid electric vehicle developed in the laboratory. The FCHEV real-time experiment results indicated that the proposed EMS can achieve the average load power matching error of 0.19 W and can optimize the system average overall system efficiency to 52%. The proposed method can help to contribute to the massive commercialization and implementation of the FCHEV.

In this paper, we address the chance-constrained safe Reinforcement Learning (RL) problem using the function approximators based on Stochastic Model Predictive Control (SMPC) and Distributionally Robust Model Predictive Control (DRMPC). We use Conditional Value at Risk (CVaR) to measure the probability of constraint violation and safety. In order to provide a safe policy by construction, we first propose using parameterized nonlinear DRMPC at each time step. DRMPC optimizes a finite-horizon cost function subject to the worst-case constraint violation in an ambiguity set. We use a statistical ball around the empirical distribution with a radius measured by the Wasserstein metric as the ambiguity set. Unlike the sample average approximation SMPC, DRMPC provides a probabilistic guarantee of the out-of-sample risk and requires lower samples from the disturbance. Then the Q-learning method is used to optimize the parameters in the DRMPC to achieve the best closed-loop performance. Wheeled Mobile Robot (WMR) path planning with obstacle avoidance will be considered to illustrate the efficiency of the proposed method.

Refereed/Peer-reviewed Investigation of value based Reinforcement Learning algorithms exhibited a resurgence into mainstream research in 2015 following demonstration of super-human performance when applied to Atari 2600 games. Since then, significant media attention and hype have accompanied this area, and the field of Artificial Intelligence generally, spread across distinct categories. This review paper is focused exclusively on the progression of value based Reinforcement Learning in the last five years. We aim to distill the incremental improvements to stability and performance in this period, highlighting the minimal changes to the base algorithm over this time. This holds true with all but the one exception of the Recurrent Experience Replay in Distributed Reinforcement Learning algorithm, representing a fundamental shift and increase in agent performance through an advanced memory representation. We suggest a new focus area for value based Reinforcement Learning research.

This paper examines a permutation flow-shop scheduling problem, which is a complex combinatorial problem encountered in many practical applications. The objective of the research is to reduce the maximum completion time, i.e., the makespan of all jobs. In order to increase productivity and to meet the demand, manufacturers are continuously under pressure to attain the shortest possible completion time. Estimation of accurate cycle time can tremendously help production planning and scheduling in manufacturing industries. Since production planning is characterised by NP-hardness and a wide range, traditional optimization methods and heuristic rules are unable to find satisfactory solutions. Q-learning, a type of reinforcement learning algorithm, is used in this paper to find a solution that is close to being optimal. Q-learning is a branch of machine learning referring to the way an intelligent agent should act in order to maximize the concept of cumulative reward in a given environment. To validate the performance of the algorithm, Taillard’s benchmark problems were solved and compared with the upper bound value. The results showed that the performance of the algorithm is better and has low computational time. Based on the performance of the proposed algorithm, two case studies were done and the solutions are compared with the performance of a metaheuristic algorithm. The result shows that the proposed algorithm can effectively and efficiently solve the problem stated above and that it is an interesting solution to resolving complex scheduling problems.

Load aggregators (LAs) play a key role in fully tapping the demand response (DR) resources of small and medium-sized end-users to enable a more flexible power grid. In the ancillary service market, the LA can provide DR to the system by aggregating the resources of its users. In response to the issued DR program, end-users offer to provide DR resources. To help optimize the user bidding strategy, an evolutionary game model is presented here in view of the bounded rationality of bidders. A combined Q-learning and compound differential evolution (CDE) algorithm is proposed to deal with the problems of incomplete information and uncertainties in the opponents' decision-making, and prevent the evolutionary stable strategy (ESS) from falling into a local optimum. Moreover, a cloud-computing-based framework is designed and agent servers are introduced to protect data privacy. Numerical results show that by adopting the proposed algorithm, the user's bidding price keeps slightly lower than the opponents' price which guarantees its revenue remains on a high level. This indicates that the proposed algorithm has good adaptability for addressing incomplete information and uncertainties in opponents' decision-making.

Spectrum access system (SAS) is a three-tier layered spectrum sharing architecture proposed by the Federal Communications Commission (FCC) for Citizens Broadband Radio Service (CBRS) 3.5 GHz band. The available 150 MHz spectrum is dynamically shared among Incumbent Access (IA), Primary Access Licensees (PAL) and General Authorized Access (GAA) users. IA users are the highest priority federal military users, PAL users are the licensed users and the GAA users are the least priority unlicensed users. In this scenario, PAL operators are willing to give access to their idle spectrum to GAA users to generate extra revenue. SAS will ensure to protect IA users and PAL users from interference caused by lower-tier users. It is the responsibility of SAS to allocate resources to GAA users but the method to do so is left open. In this article, a novel auction algorithm based on Q-learning for dynamic spectrum access (SAS-QLA) is proposed. In SAS-QLA, multiple GAA users dynamically and intelligently bid using Q-learning to access PAL reserved idle channels. SAS will decide to allocate the channels to GAA users with maximum bidding offers. GAA users have their own quality of service (QoS) demands i.e., transmission rate, packet loss, bidding efficiency, and maintain the preference of available PAL reserved idle channels based on Q-learning considering the available QoS. The proposed scenario is also modeled as a knapsack NP-hard problem and solved using dynamic programming and distributed relaxation method. Numerical results demonstrate the effectiveness of the SAS-QLA algorithm in improving the bidding efficiency, maximizing the data rate per unit cost and spectrum utilization. Web of Science 10 60804 60790

This paper presents a novel Reinforcement Learning (RL)-based control approach that uses a combination of a Deep Q-Learning (DQL) algorithm and a metaheuristic Gravitational Search Algorithm (GSA). The GSA is employed to initialize the weights and the biases of the Neural Network (NN) involved in DQL in order to avoid the instability, which is the main drawback of the traditional randomly initialized NNs. The quality of a particular set of weights and biases is measured at each iteration of the GSA-based initialization using a fitness function aiming to achieve the predefined optimal control or learning objective. The data generated during the RL process is used in training a NN-based controller that will be able to autonomously achieve the optimal reference tracking control objective. The proposed approach is compared with other similar techniques which use different algorithms in the initialization step, namely the traditional random algorithm, the Grey Wolf Optimizer algorithm, and the Particle Swarm Optimization algorithm. The NN-based controllers based on each of these techniques are compared using performance indices specific to optimal control as settling time, rise time, peak time, overshoot, and minimum cost function value. Real-time experiments are conducted in order to validate and test the proposed new approach in the framework of the optimal reference tracking control of a nonlinear position servo system. The experimental results show the superiority of this approach versus the other three competing approaches.

Machine-type communications (MTC) should account for half the connections to the internet by 2030. The use case massive MTC (mMTC) allows for applications to connect a massive number of low-power and low-complexity devices, leading to challenges in resource allocation. Not only that, mMTC networks suffer under rigid random access schemes due to mMTC ultra-dense nature resulting in poor performance. In this sense, this paper proposes a Q -Learning-based random access method for massive machine-type communications, with device clustering and non-orthogonal multiple access (NOMA). The traditional NOMA implementation increases spectral efficiency, but at the same time, demands a larger Q -Table, thus slowing down convergence, which is known to be a highly detrimental effect on massive networks. We use pre-clustering through short-range device-to-device technology to mitigate this drawback, allowing devices to operate with a smaller Q -Table. Furthermore, the previous selection of partner devices allows us to implement a full-feedback-based reward mechanism so that clusters avoid time slots already successfully allocated. Additionally, to cope with the negative impact of system overload, we propose an adaptive frame size algorithm to run in the base station (BS). It allows adjusting the frame size to the network load, preventing idle slots in an underloaded scenario, and providing extra slots when the network is overloaded. The results show the great benefits in terms of throughput of the proposed method. In addition, the impact of the use of clustering and the size of the clusters, as well as the frame size adaptation, are analyzed.

We focus on the important problem of emergency evacuation, which clearly could benefit from reinforcement learning that has been largely unaddressed. Emergency evacuation is a complex task which is difficult to solve with reinforcement learning, since an emergency situation is highly dynamic, with a lot of changing variables and complex constraints that makes it difficult to train on. In this paper, we propose the first fire evacuation environment to train reinforcement learning agents for evacuation planning. The environment is modelled as a graph capturing the building structure. It consists of realistic features like fire spread, uncertainty and bottlenecks. We have implemented the environment in the OpenAI gym format, to facilitate future research. We also propose a new reinforcement learning approach that entails pretraining the network weights of a DQN based agents to incorporate information on the shortest path to the exit. We achieved this by using tabular Q-learning to learn the shortest path on the building model's graph. This information is transferred to the network by deliberately overfitting it on the Q-matrix. Then, the pretrained DQN model is trained on the fire evacuation environment to generate the optimal evacuation path under time varying conditions. We perform comparisons of the proposed approach with state-of-the-art reinforcement learning algorithms like PPO, VPG, SARSA, A2C and ACKTR. The results show that our method is able to outperform state-of-the-art models by a huge margin including the original DQN based models. Finally, we test our model on a large and complex real building consisting of 91 rooms, with the possibility to move to any other room, hence giving 8281 actions. We use an attention based mechanism to deal with large action spaces. Our model achieves near optimal performance on the real world emergency environment., 21 pages, 14 figures, 4 tables

In a mobile edge computing (MEC) network, mobile devices could selectively offload tasks to the edge server(s) to save time and energy. However, we should consider many dynamic factors in task offloading optimization, which increases the complexity of this problem. The traditional optimization approaches could require solving complex models to derive the optimal solution. This type of optimization problems are often NP-hard and will cause a considerable overhead on optimization. In contrast, a well-trained empirical model, such as an artificial neural network could be more efficient in decision making. In this research, considering the potential uneven spatial distribution of mobile devices in a MEC network with multiple wireless edge gateways, we allow an edge gateway to offload tasks to a nearby edge gateway further. We propose a deep Q-learning-based joint optimization approach for both device-level and edge-level task offloading. We also design a centralized mathematical programming solution for exploring the optimal trade-off performance. Simulation results show that the proposed approach achieves a satisfactory task delay performance and a better trade-off between the task delay and the energy consumption on tasks.

Edge computing is a promising paradigm that brings servers closer to users, leading to lower latencies and enabling latency-sensitive applications such as cloud gaming, virtual/augmented reality, telepresence, and telecollaboration. Due to the high number of possible edge servers and incoming user requests, the optimum choice of user-server matching has become a difficult challenge, especially in the 5G era where the network can offer very low latencies. In this paper, we introduce the problem of fair server selection as not only complying with an applications latency threshold but also reducing the variance of the latency among users in the same session. Due to the dynamic and rapidly evolving nature of such an environment and the capacity limitation of the servers, we propose as solution a Reinforcement Learning method in the form of a Quadruple QLearning model with action suppression, Q-value normalization, and a reward function that minimizes the variance of the latency. Our evaluations in the context of a cloud gaming application show that, compared to a existing methods, our proposed method not only better meets the applications latency threshold but is also more fair with a reduction of up to 35% in the standard deviation of the latencies experienced by users.

Information freshness is crucial for time-critical IoT applications, e.g., monitoring and control systems. We consider an IoT status update system with multiple users, multiple energy harvesting sensors, and a wireless edge node. The users receive time-sensitive information about physical quantities, each measured by a sensor. Users send requests to the edge node where a cache contains the most recently received measurements from each sensor. To serve a request, the edge node either commands the sensor to send a status update or retrieves the aged measurement from the cache. We aim at finding the best actions of the edge node to minimize the age of information of the served measurements. We model this problem as a Markov decision process and develop reinforcement learning (RL) algorithms: model-based value iteration and model-free Q-learning methods. We also propose a Q-learning method for the realistic case where the edge node is informed about the sensors' battery levels only via the status updates. The case under transmission limitations is also addressed. Furthermore, properties of an optimal policy are analytically characterized. Simulation results show that an optimal policy is a threshold-based policy and that the proposed RL methods significantly reduce the average cost compared to several baselines., 33 pages, 6 figures

Edge caching has become an effective solution to cope with the challenges brought by the massive content delivery in cellular networks. In device-to-device (D2D) enabled caching cellular networks with time-varying content popularity distribution and user terminal (UT) location, we model these dynamic networks as a stochastic game to design a cooperative cache placement policy. The cache placement reward of each UT is defined as the caching incentive minus the transmission power cost for content caching and sharing. We consider the long-term cache placement reward of all UTs in this stochastic game. In an effort to solve the stochastic game problem, we propose a multi-agent cooperative alternating Q-learning (CAQL) based cache placement algorithm. The caching control unit is defined to execute the proposed CAQL, in which, the cache placement policy of each UT is alternatively updated according to the stable policy of other UTs during the learning process, until the stable cache placement policy of all the UTs in the cell is obtained. We discuss the convergence and complexity of CAQL, which obtains the stable cache placement policy with low space complexity. Simulation results show that the proposed algorithm can effectively reduce the backhaul load and the average content access delay in dynamic networks.

To quickly manufacture multi-variety and low-volume products, manufacturing factories are increasingly sharing resources on collaborative production networks. However, the reliability of communicat...

Principles of typical motion planning algorithms are investigated and analyzed in this paper. These algorithms include traditional planning algorithms, classical machine learning algorithms, optimal value reinforcement learning, and policy gradient reinforcement learning. Traditional planning algorithms investigated include graph search algorithms, sampling-based algorithms, interpolating curve algorithms, and reaction-based algorithms. Classical machine learning algorithms include multiclass support vector machine, long short-term memory, Monte-Carlo tree search and convolutional neural network. Optimal value reinforcement learning algorithms include Q learning, deep Q-learning network, double deep Q-learning network, dueling deep Q-learning network. Policy gradient algorithms include policy gradient method, actor-critic algorithm, asynchronous advantage actor-critic, advantage actor-critic, deterministic policy gradient, deep deterministic policy gradient, trust region policy optimization and proximal policy optimization. New general criteria are also introduced to evaluate the performance and application of motion planning algorithms by analytical comparisons. The convergence speed and stability of optimal value and policy gradient algorithms are specially analyzed. Future directions are presented analytically according to principles and analytical comparisons of motion planning algorithms. This paper provides researchers with a clear and comprehensive understanding about advantages, disadvantages, relationships, and future of motion planning algorithms in robots, and paves ways for better motion planning algorithms in academia, engineering, and manufacturing.

Cloud computing provides computing resources on demand of users without their direct management. In this cloud paradigm, scheduling the tasks and allocating the resources become major aspect for cloud infrastructure as a service (IaaS). There are more existing algorithms and techniques suggested for task allocation problem. Still there is challenging research on efficient scheduling. To address this issue, many researches are in progress and all of them having their own drawbacks. In this paper, we are proposing a queue-based scheduling strategy with load balancing called as IQSLB and an extended IQSLB also proposed for dealing with critical situations. The proposed strategy calculates the placement value of the tasks in queue with the current status of the virtual machine (VM) in cluster and reshuffles the task accordingly. The extended IQSLB deals with handling deadlock situation where VM cannot adopt task for execution and task will be reshuffled with another task in another queue. The proposed strategy is compared with few existing systems, and the performance evaluation proves that IQSLB schedules tasks more efficiently than other systems. Our proposed IQSLB takes 75 s to allocate 1000 tasks by using 55 virtual machines which is much lesser than existing techniques.

openaire: EC/H2020/857031/EU//5G!Drones This paper considers the scenario of Unmanned Aerial Vehicles (UAVs) acting as flying base stations (UAV-BSs) to provide network connectivity to ground Internet of Things (IoT) devices. More precisely, we investigate the issue where a UAV-BS needs to be replaced by a new one in a seamless way. First, we formulate the issue as an optimization problem aiming to maximize the minimum transmission rate of the served IoT devices during the UAV-BS replacement process. This is translated into jointly optimizing the trajectory of the source UAV-BS (the one to be replaced) and the target UAV-BS (the replacing one), while pushing the IoT devices to seamlessly transfer their connections to the target UAV-BS. We therefore consider a target replacement zone where the UAV-BS replacement can happen, along with IoT connections transfer. Furthermore, we propose a solution based on Deep Reinforcement Learning (DRL). More precisely, we introduce a Multi-Heterogeneous Agent-based approach (MHA-DRL), where two types of agents are considered, namely the UAV-BS agents and the IoT agents. Each agent implements a DQN (Deep Q-Learning) algorithm, where UAV-BS agents learn optimal policies to perform replacement while IoT agents learn optimal policies to transfer their connections to the target UAV-BS. The conducted performance evaluations show that the proposed approach can achieve near optimal optimization.

When deploying infrastructure as a service (IaaS) cloud virtual machines using the existing algorithms, the deployment process cannot be simplified, and the algorithm is difficult to be applied. This leads to the problems of high energy consumption, high number of migrations, and high average service-level agreement (SLA) violation rate. In order to solve the above problems, an adaptive deployment algorithm for IaaS cloud virtual machines based on Q learning mechanism is proposed in this research. Based on the deployment principle, the deployment characteristics of the IaaS cloud virtual machines are analyzed. The virtual machine scheduling problem is replaced with the Markov process. The multistep Q learning algorithm is used to schedule the virtual machines based on the Q learning mechanism to complete the adaptive deployment of the IaaS cloud virtual machines. Experimental results show that the proposed algorithm has low energy consumption, small number of migrations, and low average SLA violation rate.

This paper aims to solve the path following problem for an underactuated unmanned-surface-vessel (USV) based on deep reinforcement learning (DRL). A smoothly-convergent DRL (SCDRL) method is proposed based on the deep Q network (DQN) and reinforcement learning. In this new method, an improved DQN structure was developed as a decision-making network to reduce the complexity of the control law for the path following of a three-degree of freedom USV model. An exploring function was proposed based on the adaptive gradient descent to extract the training knowledge for the DQN from the empirical data. In addition, a new reward function was designed to evaluate the output decisions of the DQN, and hence, to reinforce the decision-making network in controlling the USV path following. Numerical simulations were conducted to evaluate the performance of the proposed method. The analysis results demonstrate that the proposed SCDRL converges more smoothly than the traditional deep Q learning while the path following error of the SCDRL is comparable to existing methods. Thanks to good usability and generality of the proposed method for USV path following, it can be applied to practical applications.