Search

The Internet of vehicles (IoV) is a large information interaction network that collects information on vehicles, roads and pedestrians. One of the important uses of vehicle networks is to meet the entertainment needs of driving users through communication between vehicles and roadside units (RSUs). Due to the limited storage space of RSUs, determining the content cached in each RSU is a key challenge. With the development of 5G and video editing technology, short video systems have become increasingly popular. Current widely used cache update methods, such as partial file precaching and content popularity- and user interest-based determination, are inefficient for such systems. To solve this problem, this paper proposes a QoE-driven edge caching method for the IoV based on deep reinforcement learning. First, a class-based user interest model is established. Compared with the traditional file popularity- and user interest distribution-based cache update methods, the proposed method is more suitable for systems with a large number of small files. Second, a quality of experience (QoE)-driven RSU cache model is established based on the proposed class-based user interest model. Third, a deep reinforcement learning method is designed to address the QoE-driven RSU cache update issue effectively. The experimental results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

This paper is concerned with the efficient movement control of multiple drone cells for communication coverage. Although many works have been developed to cope with this problem, but only few of them have investigated the three-dimensional (3-D) continuous movement control of multiple drone cells. In this paper, a problem of 3-D continuous movement control of multiple drone cells is formulated with an objective of maximizing the energy-efficient communication coverage of drone-cell networks while preserving the network connectivity. To mitigate this challenging problem, an energy-efficiency and continuous-movement-control algorithm (E $^2$ CMC), which is based on an emerging deep reinforcement learning method, is proposed. In E $^2$ CMC, an energy-efficiency reward function considering the energy consumption, the sum of quality-of-service (QoS) requirements of users as well as the coverage fairness is first designed. Next, E $^2$ CMC learns to adjust drone cells’ locations continuously by interacting with an environment in a sequence of observations, actions, and rewards. Furthermore, E $^2$ CMC will reduce the reward drastically as long as the networks are disconnected. Simulation results verify the superiority of the proposed learning algorithm on deploying multiple drone cells to provide the communication coverage. [ABSTRACT FROM AUTHOR]

In this paper, we study the three-dimensional (3D) path planning problem for cellular-connected unmanned aerial vehicle (UAV) taking into account the impact of 3D antenna radiation patterns. The cellular-connected UAV has a mission to travel from an initial location to a destination. In this process, there is a trade-off between the flight time and the expected communication outages duration, which requires planning a suitable route to avoid the weak coverage region. However, as the existing cellular networks are primarily designed for ground coverage, the communication coverage in the sky tends to be intermittent and irregular due to the potential for severe interference and obstructions (such by buildings), which brings great challenges to 3D path planning. To address this challenge, we first construct a 3D coverage map that stores the expected outage probability over each locations. We then propose a multi-step dueling DDQN (multi-step D3QN) based algorithm to design the local optimal UAV path by leveraging the constructed coverage map. In this algorithm, the UAV acts as the agent to learn the appropriate action to complete the flight mission. Numerical results show the effectiveness of the proposed algorithm for connectivity-aware UAV path planning and the superiority of 3D path design over its 2D counterparts. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the problem of wireless service provisioning through a rotary-wing UAV which can serve as an aerial base station (BS) to communicate with multiple ground terminals (GTs) in a boost demand area. Our objective is to optimize the UAV control for maximizing the UAV.s energy efficiency, in which both aerodynamic energy and communication energy are considered while ensuring the communication requirements for each GT and backhaul link between the UAV and the terrestrial BS. The mobility of the UAV and GTs lead to time-varying channel conditions that make the environment dynamic. We formulate a nonconvex optimization for controlling the UAV considering the practical angle-dependent Rician fading channels between the UAV and GTs, and between the UAV and the terrestrial BS. Traditional optimization approaches are not able to handle the dynamic environment and high complexity of the problem in real-time. We propose to use a deep reinforcement learning-based approach namely Deep Deterministic Policy Gradient (DDPG) to solve the formulated nonconvex problem of UAV control with continuous action space that takes into account the real-time of the environment including time-varying UAV-ground channel conditions, available onboard energy of the UAV, and the communication requirement of the GTs. However, the DDPG method may not achieve good performance in an unstable environment and will face a large number of hyperparameters. We extend our approach to use the Trust Region Policy Optimization (TRPO) method that can improve the performance of the UAV compared to the DDPG method in such a dynamic environment. [ABSTRACT FROM AUTHOR]

The renewable energy source (RES)-powered small cell base station (SBS) is a promising technology for the next-generation networks because RESs provide sustainable energy without being depleted. In RES-assisted networks, joint optimization of user association (UA) and power control (PC) is required to enhance the sum-rate performance by reducing inter-cell interference between SBSs. This paper tackles the UA and PC optimization for the sum-rate maximization under quality-of-service (QoS) and backhaul constraints. We formulate the problem as mixed-integer non-linear programming, in which UA and PC variables of different time slots are tightly coupled due to the RES energy dynamics model. Hence, designing a dynamic policy-based UA and PC with consideration of the future environments such as the quantity of channel gain and energy harvesting remains a challenge. First, we propose a deep unsupervised learning (DUL)-based UA scheme for a fixed PC variable. To lower the computational complexity and accelerate the convergence of the proposed learning-based optimization, we relax the UA variable, which originally has an extremely high dimension, into a low-dimensional continuous variable inspired by the Lagrangian method. Next, we propose a deep reinforcement learning (DRL)-based PC scheme, in which the stringent QoS and backhaul constraints are considered penalty terms on the reward design. The proposed DRL-based PC scheme facilitates dynamic PC inferring the relationship between the future environment and current PC. Simulation results demonstrate that the proposed scheme enhances the sum-rate by 10%, accommodates 3.3 percent point (%p) more QoS-qualified users, and reduces the computation time by 20 times compared to the conventional optimization-based method. [ABSTRACT FROM AUTHOR]

In this paper, we develop centralized and distributed dynamic resource allocation schemes for an intelligent reflecting surface (IRS) aided energy harvesting (EH) system that optimize transmit power of a source node and phase-shift of passive reflecting elements of IRS. The source node randomly harvests renewable energy from the surrounding environment and performs data transmission with the harvested energy while satisfying statistical packet delay constraints in terms of maximum acceptable delay-outage probability. Our developed schemes do not require the statistical distributions of channel and energy profiles to be known and apply deep reinforcement learning (DRL) algorithm. Simulation results demonstrate the effectiveness of the proposed resource control scheme for IRS-aided EH system in different channel and energy conditions. [ABSTRACT FROM AUTHOR]

Buffer-aware video streaming, which exploits the available storage space in user device to store the prefetched video data in good channels for video data use in poor channels, has been proved to have the potential to reduce the impact of fluctuating wireless channels on user-perceived video performance. However, in practical wireless networks, due to the unknown channel state and video rate, providing buffer-aware video streaming service to wireless user is a challenging problem. In this paper, with the aim to design an autonomous wireless video streaming system, we apply the deep reinforcement learning approach to dynamic resource optimization for wireless buffer-aware video streaming under unknown channel state and video rate. Specifically, we define a reward function for buffer-aware video streaming as the effective video streaming time when neither video-playback overflow nor video-playback underflow occurs. We propose a Markov decision process based problem formulation of the joint bandwidth allocation and buffer management for maximizing the effective video streaming time of all users. The optimal bandwidth allocation and buffer management policy is learned from training a deep neural network based on a deep reinforcement learning algorithm. We simulate the proposed algorithm in Tensorflow. Simulation results verify that the proposed deep reinforcement learning approach is effective for buffer-aware video streaming in wireless networks. [ABSTRACT FROM AUTHOR]

Currently, self-driving, emerging as a key automatic application, has brought a huge potential for the provision of in-vehicle services (e.g., automatic path planning) to mitigate urban traffic congestion and enhance travel safety. To provide high-quality vehicular services with stringent delay constraints, edge computing (EC) enables resource-hungry self-driving vehicles (SDVs) to offload computation-intensive tasks to the edge servers (ESs). In addition, caching highly reusable contents decreases the redundant transmission time and improves the quality of services (QoS) of SDVs, which is envisioned as a supplement to the computation offloading. However, the high mobility and time-varying requests of SDVs make it challenging to provide reliable offloading decisions while guaranteeing the resource utilization of content caching. To this end, in this paper we propose a collaborative computation offloading and content caching method, named CoPace, by leveraging deep reinforcement learning (DRL) in EC for self-driving system. Specifically, we first introduce OSTP to predict the future time-varying content popularity, taking into account the temporal-spatial attributes of requests. Moreover, a DRL-based algorithm is developed to jointly optimize the offloading and caching decisions, as well as the resource allocation (i.e., computing and communication resources) strategies. Extensive experiments with real-world datasets in Shanghai, China, are conducted to evaluate the performance, which demonstrates that CoPace is both effective and well-performed. [ABSTRACT FROM AUTHOR]

In the multi-server multi-access edge computing (MEC) system, the computing capacity could be insufficient and the computing resource could be over-utilized when an excessive number of computing tasks are offloaded for execution. To alleviate the excessive burden in the multi-server MEC system and tap the underutilized resources of wired edge devices, which are called edge computers in this paper, we investigate the computation offloading and shunting scheme in the wireless wireline internetwork. We formulate an optimization problem to minimize the average total time delay and use the deep reinforcement learning method to obtain the optimal shunting policy. By sensing the state of the wireless wireline internetwork, the proposed scheme can adaptively adjust the shunting ratio and utilize the resources of MEC servers and edge computers intelligently and jointly. Finally, we evaluate the computational complexity and validate the convergence property of this scheme. The extensive simulation results demonstrate that this scheme can improve the resource productivity and reduce the average total time delay. [ABSTRACT FROM AUTHOR]

Advanced algorithms can promote the development of energy management strategies (EMSs) as a key technology in hybrid electric vehicles (HEVs). Reinforcement learning (RL) with distributed structure can significantly improve training efficiency in complex environments, and multi-threaded parallel computing provides a reliable algorithm basis for promoting adaptability. Dedicated to trying more efficient deep reinforcement learning (DRL) algorithms, this paper proposed a deep q-network (DQN)-based energy and emission management strategy (E&EMS) at first. Then, two distributed DRL algorithms, namely asynchronous advantage actor-critic (A3C) and distributed proximal policy optimization (DPPO), were adopted to propose EMSs, respectively. Finally, emission optimization was taken into account and then distributed DRL-based E&EMSs were proposed. Regarding dynamic programming (DP) as the optimal benchmark, simulation results show that three DRL-based control strategies can achieve near-optimal fuel economy and outstanding computational efficiency, and compared with DQN, two distributed DRL algorithms have improved the learning efficiency by four times. [ABSTRACT FROM AUTHOR]

Network slicing is a key paradigm in 5G and is expected to be inherited in future 6G networks for the concurrent provisioning of diverse quality of service (QoS). Unfortunately, effective slicing of Radio Access Networks (RAN) is still challenging due to time-varying network situations. This paper proposes a new intelligent RAN slicing strategy with two-layered control granularity, which aims at maximizing both the long-term QoS of services and spectrum efficiency (SE) of slices. The proposed method consists of an upper-level controller to ensure the QoS performance, which enforces loose control by performing adaptive slice configuration according to the long-term dynamics of service traffic. The lower-level controller is to improve SE of slices, by tightly scheduling radio resources to users at the small time-scale. To realize the proposed RAN slicing strategy, we propose a model-free deep reinforcement learning (DRL) framework, which is a hierarchical structure that collaboratively integrating the modified deep deterministic policy gradient (DDPG) and double deep-Q-network algorithm. Specifically, the lower-level control problem is a mixed-integer stochastic optimization problem with multiple constraints. This kind of problem is hard to be directly solved by the exiting DRL algorithms, since it involves searching for the solution in a vast set of mixed-integer action space, which will induce unbearable computational complexity. Thus, we propose a novel action space reducing approach, embedding the convex optimization tools into the DDPG algorithm, to speed up the lower-level control. Furthermore, simulation results confirm the effectiveness of our proposed intelligent RAN slicing scheme. [ABSTRACT FROM AUTHOR]

Network slicing (NS) has been widely identified as a key architectural technology for 5G-and-beyond systems by supporting divergent requirements in a sustainable way. In radio access network (RAN) slicing, due to the device-base station (BS)-NS three layer association relationship, device association (including access control and handoff management) becomes an essential yet challenging issue. With the increasing concerns on stringent data security and device privacy, exploiting local resources to solve device association problem while enforcing data security and device privacy becomes attractive. Fortunately, recently emerging federated learning (FL), a distributed learning paradigm with data protection, provides an effective tool to address this type of issues in mobile networks. In this paper, we propose an efficient device association scheme for RAN slicing by exploiting a hybrid FL reinforcement learning (HDRL) framework, with the aim to improve network throughput while reducing handoff cost. In our proposed framework, individual smart devices train a local machine learning model based on local data and then send the model features to the serving BS/encrypted party for aggregation, so as to efficiently reduce bandwidth consumption for learning while enforcing data privacy. Specifically, we use deep reinforcement learning to train the local model on smart devices under a hybrid FL framework, where horizontal FL is employed for parameter aggregation on BS, while vertical FL is employed for NS/BS pair selection aggregation on the encrypted party. Numerical results show that the proposed HDRL scheme can achieve significant performance gain in terms of network throughput and communication efficiency in comparison with some state-of-the-art solutions. [ABSTRACT FROM AUTHOR]

Resource allocation in ultra dense network (UDN) is an multi-objective optimization problem since it has to consider the tradeoff among spectrum efficiency (SE), energy efficiency (EE) and fairness. The existing methods can not effectively solve this NP-hard nonconvex problem, especially in the presence of limited channel state information (CSI). In this paper, we investigate a novel model-driven deep reinforcement learning assisted resource allocation method. We first design a novel deep neural network (DNN)-based optimization framework consisting of a series of Alternating Direction Method of Multipliers (ADMM) iterative procedures, which makes the CSI as the learned weights. Then a novel channel information absent Q-learning resource allocation (CIAQ) algorithm is proposed to train the DNN-based optimization framework without massive labeling data, where the SE, the EE, and the fairness can be jointly optimized by adjusting discount factor. Our simulation results show that, the proposed CIAQ with rapid convergence speed not only well characterizes the extent of optimization objective with partial CSI, but also significantly outperforms the current random initialization method of neural network and the other existing resource allocation algorithms in term of the tradeoff among the SE, EE and fairness. [ABSTRACT FROM AUTHOR]

The smart vehicles construct Internet of Vehicle (IoV), which can execute various intelligent services. Although the computation capability of a vehicle is limited, multi-type of edge computing nodes provide heterogeneous resources for intelligent vehicular services. When offloading the complex service to the vehicular edge computing node, the decision for its destination should be considered according to numerous factors. This paper mostly formulate the offloading decision as a resource scheduling problem with single or multiple objective function and constraints, where some customized heuristics algorithms are explored. However, offloading multiple data dependence tasks in a complex service is a difficult decision, as an optimal solution must understand the resource requirement, the access network, the user mobility, and importantly the data dependence. Inspired by recent advances in machine learning, we propose a knowledge driven (KD) service offloading decision framework for IoV, which provides the optimal policy directly from the environment. We formulate the offloading decision for the multiple tasks as a long-term planning problem, and explore the recent deep reinforcement learning to obtain the optimal solution. It can scruple the future data dependence of the following tasks when making decision for a current task from the learned offloading knowledge. Moreover, the framework supports the pre-training at the powerful edge computing node and continually online learning when the vehicular service is executed, so that it can adapt the environment changes and can learn policy that are sensible in foresight. The simulation results show that KD service offloading decision converges quickly, adapts to different conditions, and outperforms a greedy offloading decision algorithm. [ABSTRACT FROM AUTHOR]

With the increasing penetration of renewable energy (RE), the operations of distribution network are threatened and some issues may appear, i.e., large voltage deviation, deterioration of statistic voltage stability, high power loss, etc. In turn, RE accommodation would be significantly impacted. Therefore, we propose a many-objective distribution network reconfiguration (MDNR) model, with the consideration of RE curtailment, voltage deviation, power loss, statistic voltage stability, and generation cost. This aims to assess the trade-off among these objectives for better operations of distribution networks. As the proposed model is a non-convex, non-linear, many-objective optimization problem, it is difficult to be solved. We further propose a deep reinforcement learning (DRL) assisted multi-objective bacterial foraging optimization (DRL-MBFO) algorithm. This algorithm combines the advantages of DRL and MBFO, and is targeted to find the Pareto front of proposed MDNR model with better searching efficiency. Finally, we conduct case study on the modified IEEE 33-bus, 69-bus, and 118-bus power distribution systems, and results verify the effectiveness of the MDNR model and outperformance of the proposed DRL-MBFO. [ABSTRACT FROM AUTHOR]

We propose a learning-based scheme to investigate the dynamic multi-channel access (DMCA) problem in the fifth generation (5G) and beyond networks with fast time-varying channels wherein the channel parameters are unknown. The proposed learning-based scheme can maintain near-optimal performance for a long time, even in the sharp changing channels. This scheme greatly reduces processing delay, and effectively alleviates the error due to decision lag, which is cased by the non-immediacy of the information acquisition and processing. We first propose a psychology-based personalized quality of service model after introducing the network model with unknown channel parameters and the streaming model. Then, two access criteria are presented for the living streaming model and the buffered streaming model. Their corresponding optimization problems are also formulated. The optimization problems are solved by learning-based DMCA scheme, which combines the recurrent neural network with deep reinforcement learning. In the learning-based DMCA scheme, the agent mainly invokes the proposed prediction-based deep deterministic policy gradient algorithm as the learning algorithm. As a novel technical paradigm, our scheme has strong universality, since it can be easily extended to solve other problems in wireless communications. The real channel data-based simulation results validate that the performance of the learning-based scheme approaches that derived from the exhaustive search when making a decision at each time-slot, and is superior to the exhaustive search method when making a decision at every few time-slots. [ABSTRACT FROM AUTHOR]

Due to the dynamic nature of connectivity in terrestrial vehicular networks, it is of great benefit to deploy unmanned aerial vehicles (UAVs) in these networks to act as relays. As a result, a remarkable number of studies have exploited UAVs to bridge the communication gaps between terrestrial vehicles, and sometimes despite their unoptimized mobility, their restricted communication coverage, and their limited energy resources. However, it was noted that for an intermittently connected vehicular network, UAVs could not cover all sparse areas all the time. Even worse, when deploying enough UAVs to cover all these areas, the probability of inter-UAV collisions increases, and it will be complex to control their movements efficiently. Consequently, it is required to dispatch an organized and intelligent group of UAVs to perform communication relays in the long term while keeping their connectivity, minimizing their average energy consumption, and providing an efficient coverage strategy. To meet these requirements, we propose a deep reinforcement learning (DRL) framework, called DISCOUNT (Dispatch of UAVs for Urban VANETs). Extensive simulations have been conducted to evaluate the performance of the proposed framework. It has been shown that the proposed framework significantly outperforms two commonly-used baseline techniques and some reinforcement learning methods in terms of energy consumption, coverage, and routing performances. [ABSTRACT FROM AUTHOR]

For a battery swapping station (BSS), the stochastic operation of electric buses (EBs) and the uncertainty of electricity prices cause unnecessary economic losses. To minimize the operating costs of the BSS, this article applies the deep reinforcement learning (DRL) and proposes a BSS model to determine the optimal real-time charge/discharge power of the charging piles. The predictability of bus operation and the uncertainty of price can be directly captured from historical data without any assumption in the model. Moreover, deep deterministic policy gradient (DDPG), as the DRL algorithm, is implemented in the model to simultaneously control multiple charging piles. Numerical results illustrate that the proposed approach can bring less operating cost than the existing benchmark control methods for a BSS while providing adequate batteries ready for swapping. [ABSTRACT FROM AUTHOR]

The effective pricing of electric vehicle (EV) charging by aggregators constitutes a key problem toward the realization of the significant EV flexibility potential in deregulated electricity systems and has been addressed by previous work through bi-level optimization formulations. However, the solution approach adopted in previous work cannot capture the discrete nature of the EV charging/discharging levels. Although reinforcement learning (RL) can tackle this challenge, state-of-the-art RL methods require discretization of state and/or action spaces and thus exhibit limitations in terms of solution optimality and computational requirements. This article proposes a novel deep reinforcement learning (DRL) method to solve the examined EV pricing problem, combining deep deterministic policy gradient (DDPG) principles with a prioritized experience replay (PER) strategy and setting up the problem in multi-dimensional continuous state and action spaces. Case studies demonstrate that the proposed method outperforms state-of-the-art RL methods in terms of both solution optimality and computational requirements and comprehensively analyze the economic impacts of smart-charging and vehicle-to-grid (V2G) flexibility on both aggregators and EV owners. [ABSTRACT FROM AUTHOR]