Search

The explosive growth of Internet-of-Things (IoT) applications such as smart cities and Industry 4.0 have led to drastic increase in demand for wireless bandwidth, hence motivating the rapid development of new techniques for enhancing spectrum utilization needed by new generation wireless communication technologies. Among others, dynamic spectrum access (DSA) is one of the most widely accepted approaches. In this paper, as an enhancement of existing works, we take into consideration of inter-node collaborations in a dynamic spectrum environment. Typically, in such distributed circumstances, intelligent dynamic spectrum access almost invariably relies on self-learning to achieve dynamic spectrum access improvement. Whereas, this paper proposes a DSA scheme based on deep reinforcement learning to enhance spectrum and access efficiency. Unlike traditional Q-learning-based DSA, we introduce the following to enhance the spectrum efficiency in dynamic IoT spectrum environments. First, deep double Q-learning is adopted to perform local self-spectrum-learning for IoT terminals in order to achieve better dynamic access accuracy. Second, to accelerate learning convergence, federated learning (FL) in edge nodes is used to improve the self-learning. Third, multiple secondary users, who do not interfere with each other and have similar operation condition, are clustered for federated learning to enhance the efficiency of deep reinforcement learning. Comparing with the traditional distributed DSA with deep learning, the proposed scheme has faster access convergence speed due to the characteristic of global optimization for federated learning. Based on this, a framework of federated deep reinforcement learning (FDRL) for DSA is proposed. Furthermore, this scheme preserves privacy of IoT users in that FDRL only requires model parameters to be uploaded to edge servers. Simulations are performed to show the effectiveness of theproposed FDRL-based DSA framework. [ABSTRACT FROM AUTHOR]

This paper applies machine learning to optimize the transmission policies of cognitive radio inspired non-orthogonal multiple access (CR-NOMA) networks, where time-division multiple access (TDMA) is used to serve multiple primary users and an energy-constrained secondary user is admitted to the primary users’ time slots via NOMA. During each time slot, the secondary user performs the two tasks: data transmission and energy harvesting based on the signals received from the primary users. The goal of the paper is to maximize the secondary user’s long-term throughput, by optimizing its transmit power and the time-sharing coefficient for its two tasks. The long-term throughput maximization problem is challenging due to the need for making decisions that yield long-term gains but might result in short-term losses. For example, when in a given time slot, a primary user with large channel gains transmits, intuition suggests that the secondary user should not carry out data transmission due to the strong interference from the primary user but perform energy harvesting only, which results in zero data rate for this time slot but yields potential long-term benefits. In this paper, a deep reinforcement learning (DRL) approach is applied to emulate this intuition, where the deep deterministic policy gradient (DDPG) algorithm is employed together with convex optimization. Our simulation results demonstrate that the proposed DRL assisted NOMA transmission scheme can yield significant performance gains over two benchmark schemes. [ABSTRACT FROM AUTHOR]

This paper studies the allocation of shared resources between vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) links in vehicle-to-everything (V2X) communications. In existing algorithms, dynamic vehicular environments and quantization of continuous power become the bottlenecks for providing an effective and timely resource allocation policy. In this paper, we develop two algorithms to deal with these difficulties. First, we propose a deep reinforcement learning (DRL)-based resource allocation algorithm to improve the performance of both V2I and V2V links. Specifically, the algorithm uses deep Q-network (DQN) to solve the sub-band assignment and deep deterministic policy-gradient (DDPG) to solve the continuous power allocation problem. Second, we propose a meta-based DRL algorithm to enhance the fast adaptability of the resource allocation policy in the dynamic environment. Numerical results demonstrate that the proposed DRL-based algorithm can significantly improve the performance compared to the DQN-based algorithm that quantizes continuous power. In addition, the proposed meta-based DRL algorithm can achieve the required fast adaptation in the new environment with limited experiences. [ABSTRACT FROM AUTHOR]

As power systems are undergoing a significant transformation with more uncertainties, less inertia and closer to operation limits, there is increasing risk of large outages. Thus, there is an imperative need to enhance grid emergency control to maintain system reliability and security. Towards this end, great progress has been made in developing deep reinforcement learning (DRL) based grid control solutions in recent years. However, existing DRL-based solutions have two main limitations: 1) they cannot handle well with a wide range of grid operation conditions, system parameters, and contingencies; 2) they generally lack the ability to fast adapt to new grid operation conditions, system parameters, and contingencies, limiting their applicability for real-world applications. In this paper, we mitigate these limitations by developing a novel deep meta-reinforcement learning (DMRL) algorithm. The DMRL combines the meta strategy optimization together with DRL, and trains policies modulated by a latent space that can quickly adapt to new scenarios. We test the developed DMRL algorithm on the IEEE 300-bus system. We demonstrate fast adaptation of the meta-trained DRL polices with latent variables to new operating conditions and scenarios using the proposed method, which achieves superior performance compared to the state-of-the-art DRL and model predictive control (MPC) methods. [ABSTRACT FROM AUTHOR]

The Internet of vehicles (IoV) is a large information interaction network that collects information on vehicles, roads and pedestrians. One of the important uses of vehicle networks is to meet the entertainment needs of driving users through communication between vehicles and roadside units (RSUs). Due to the limited storage space of RSUs, determining the content cached in each RSU is a key challenge. With the development of 5G and video editing technology, short video systems have become increasingly popular. Current widely used cache update methods, such as partial file precaching and content popularity- and user interest-based determination, are inefficient for such systems. To solve this problem, this paper proposes a QoE-driven edge caching method for the IoV based on deep reinforcement learning. First, a class-based user interest model is established. Compared with the traditional file popularity- and user interest distribution-based cache update methods, the proposed method is more suitable for systems with a large number of small files. Second, a quality of experience (QoE)-driven RSU cache model is established based on the proposed class-based user interest model. Third, a deep reinforcement learning method is designed to address the QoE-driven RSU cache update issue effectively. The experimental results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

With the new feature of multi-energy coupling and the advancement of the energy market, Energy Internet (EI) has higher requirements for the efficiency and applicability of integrated energy response. This paper proposes an indirect multi-energy transaction (IMET) to promote multi-energy collaborative optimization in local energy market (LEM) and improve energy utilization through personalized responses from We-Energies (WEs). Firstly, an indirect customer-to-customer multi-energy transaction is modeled for local multi-energy coupling market which can satisfy privacy, preference and autonomy of users. The efficiency of energy matching can be promoted through the participation of conversion devices. In addition, multi-time scale hybrid trading mechanism is constructed with the consideration of the transmission speed of different energy sources. Meanwhile, energy transaction process is built as a Markov decision process (MDP) with deep reinforcement learning algorithm so that the system modeling error can be successfully avoided. Furthermore, a distributed training structure is utilized to obtain more experience for a wider range of scenarios. The results of numerical simulations demonstrate the performance of the proposed method. [ABSTRACT FROM AUTHOR]

This paper is concerned with the efficient movement control of multiple drone cells for communication coverage. Although many works have been developed to cope with this problem, but only few of them have investigated the three-dimensional (3-D) continuous movement control of multiple drone cells. In this paper, a problem of 3-D continuous movement control of multiple drone cells is formulated with an objective of maximizing the energy-efficient communication coverage of drone-cell networks while preserving the network connectivity. To mitigate this challenging problem, an energy-efficiency and continuous-movement-control algorithm (E $^2$ CMC), which is based on an emerging deep reinforcement learning method, is proposed. In E $^2$ CMC, an energy-efficiency reward function considering the energy consumption, the sum of quality-of-service (QoS) requirements of users as well as the coverage fairness is first designed. Next, E $^2$ CMC learns to adjust drone cells’ locations continuously by interacting with an environment in a sequence of observations, actions, and rewards. Furthermore, E $^2$ CMC will reduce the reward drastically as long as the networks are disconnected. Simulation results verify the superiority of the proposed learning algorithm on deploying multiple drone cells to provide the communication coverage. [ABSTRACT FROM AUTHOR]

Computation offloading is an effective method in mobile edge computing (MEC) to relieve user equipment (UE) from the limited computation resource and battery capacity. Meanwhile, simultaneous wireless information and power transmission (SWIPT) can be applied to MEC to extend the operating time of the equipment. However, in multi-user network environment, diverse computation task requirements and changeable network channel states make it challenging to obtain offloading strategy timely and accurately. To address the issue, we propose an intelligent computation offloading scheme (iCOS) based on enhanced priority deep deterministic policy gradient (EPDDPG) algorithm to minimize the energy consumption of all the UEs by jointly optimizing the offloading decision, the central processing unit (CPU) frequency and the power split ratio in a dynamic SWIPT-MEC network. In particular, we improve the traditional fully-connected network structure to obtain both discrete and continuous action outputs, and accelerate neural network parameter updates by using prioritized experience tuples. Furthermore, we use dynamic voltage and frequency scaling (DVFS) technology to dynamically adjust the CPU frequency of local computing, and employ SWIPT technology to balance the charging and communication according to the obtained strategy. Simulation results show that the algorithm proposed in this paper can effectively reduce the energy cost of UEs, and complete more computation tasks within the delay limit. [ABSTRACT FROM AUTHOR]

Many of the existing optimization-based transmit power control algorithms suffer from high computational complexity and require instantaneous global channel state information (CSI), both of which hinder their practical implementation. In this paper, we consider a wireless network with multiple transmitter-receiver pairs, where each transmitter only has access to its local CSI fed back from its intended receiver and does not require local CSI exchange with its neighboring transmitters. In such a network scenario, we propose a deep reinforcement learning based decentralized multi-agent power control (DEC-MAPC) algorithm for sum-rate maximization, where each transmitter acts as an intelligent agent. By leveraging the value decomposition technique, we establish a nonlinear mapping from the local reward of each agent to the global reward. Such a design allows each agent to independently control its transmit power based on its local CSI while enabling global collaboration among the agents. The proposed algorithm is scalable to large-scale networks as only local CSI is required, and is robust to the channel and interference variations via interacting with the environment. Simulation results show that the proposed DEC-MAPC algorithm with local CSI achieves comparable sum-rate performance with the centralized optimization algorithms with global CSI, while significantly reducing the computational complexity. [ABSTRACT FROM AUTHOR]

In this paper, we aim to propose a novel transmission control protocol (TCP) congestion control method from a cross-layer-based perspective and present a deep reinforcement learning (DRL)-driven method called DRL-3R (DRL for congestion control with Radio access network information and Reward Redistribution) so as to learn the TCP congestion control policy in a superior manner. In particular, we incorporate the RAN information to timely grasp the dynamics of RAN, and empower DRL to learn from the delayed RAN information feedback potentially induced by several consecutive actions. Meanwhile, we relax the implicit assumption (that the feedback to one specific action returns at a round-trip-time (RTT) after the action is applied) in previous researches, by redistributing the rewards and evaluating the merits of actions more accurately. Experiment results show that besides maintaining a reasonable fairness, DRL-3R significantly outperforms classical congestion control methods (e.g., TCP Reno, Westwood, Cubic, BBR and DRL-CC) on network utility by achieving a higher throughput while reducing delay in various network environments. [ABSTRACT FROM AUTHOR]

Knowledge transfer is a promising concept to achieve real-time decision-making for autonomous vehicles. This paper constructs a transfer deep reinforcement learning (RL) framework to transform the driving tasks in the intersection environments. The driving missions at the unsignalized intersection are cast into a left turn, right turn, and running straight for automated vehicles. The goal of the autonomous ego vehicle (AEV) is to drive through the intersection situation efficiently and safely. This objective promotes the studied vehicle to increase its speed and avoid crashing other vehicles. The decision-making policy learned from one driving task is transferred through three transfer rules in another driving mission and evaluated. Simulation results reveal that the decision-making strategies related to similar tasks are transferable and have a high success rate. It indicates that the presented control framework could reduce time consumption and realize online implementation. Therefore, the transfer RL concept is helpful for establishing the real-time decision-making policy for autonomous vehicles. [ABSTRACT FROM AUTHOR]

In this paper, we study the three-dimensional (3D) path planning problem for cellular-connected unmanned aerial vehicle (UAV) taking into account the impact of 3D antenna radiation patterns. The cellular-connected UAV has a mission to travel from an initial location to a destination. In this process, there is a trade-off between the flight time and the expected communication outages duration, which requires planning a suitable route to avoid the weak coverage region. However, as the existing cellular networks are primarily designed for ground coverage, the communication coverage in the sky tends to be intermittent and irregular due to the potential for severe interference and obstructions (such by buildings), which brings great challenges to 3D path planning. To address this challenge, we first construct a 3D coverage map that stores the expected outage probability over each locations. We then propose a multi-step dueling DDQN (multi-step D3QN) based algorithm to design the local optimal UAV path by leveraging the constructed coverage map. In this algorithm, the UAV acts as the agent to learn the appropriate action to complete the flight mission. Numerical results show the effectiveness of the proposed algorithm for connectivity-aware UAV path planning and the superiority of 3D path design over its 2D counterparts. [ABSTRACT FROM AUTHOR]

Due to the deregulation of power systems worldwide, bidding behavior simulation research has gained prominence. One crucial element in these studies is accurately defining and modelling the individual reward function (or objective function). Considering the ubiquitous information barriers between market participants and researchers, the common way is to develop reward functions based on theoretical assumptions, which will inevitably cause deviations from the real world. However, since market data have gradually become transparent in recent years, especially data regarding historical bidding behaviors, it is feasible to introduce data-driven methods to identify the individual reward functions that are hidden in raw bidding data. Thus, this paper proposes a data-driven bidding objective function identification framework with three procedures. First, the bidding decision processes of participants are formulated as a standard Markov decision process. Second, a deep inverse reinforcement learning method that is based on maximum entropy is introduced to identify individual reward functions, whose high-dimensional nonlinearity could be saved in multilayer perceptions (MLPs). Third, a deep Q-network method is customized to simulate the individual bidding behaviors based on the obtained MLP-based objective functions. The effectiveness and feasibility of the proposed framework and methods are tested based on real market data from the Australian electricity market. [ABSTRACT FROM AUTHOR]

In this paper, we propose a channel-wise interaction based binary convolutional neural networks (CI-BCNN) approach for efficient inference. Conventional binary convolutional neural networks usually apply the xnor and bitcount operations in the binary convolution with notable quantization errors, which obtain opposite signs of pixels in binary feature maps compared to their full-precision counterparts and lead to significant information loss. In our proposed CI-BCNN method, we exploit the channel-wise interactions with the prior knowledge which aims to alleviate inconsistency of signs in binary feature maps and preserves the information of input samples during inference. Specifically, we mine the channel-wise interactions by using a reinforcement learning model, and impose channel-wise priors on the intermediate feature maps to correct inconsistent signs through the interacted bitcount. Since CI-BCNN mines the channel-wise interactions in a large search space where each channel may correlate with others, the search deficiency caused by sparse interactions obstacles the agent to obtain the optimal policy. To address this, we further present a hierarchical channel-wise interaction based binary convolutional neural networks (HCI-BCNN) method to shrink the search space via hierarchical reinforcement learning. Moreover, we propose a denoising interacted bitcount operation in binary convolution by smoothing the channel-wise interactions, so that noise in channel-wise priors can be alleviated. Extensive experimental results on the CIFAR-10 and ImageNet datasets demonstrate the effectiveness of the proposed CI-BCNN and HCI-BCNN. [ABSTRACT FROM AUTHOR]

Resilience of a distribution system can be enhanced by efficient restoration of critical load following a major outage. Existing models include optimization approaches that consider available information without incorporating the inherent asynchrony of data arrival during execution of the restoration plan. Failure to consider the asynchronous nature of information arrival can lead to underutilization of critical resources. Moreover, analytical models become computationally inefficient for large scale systems. On the other hand, artificial intelligence (AI)-based tools have demonstrated efficient results for power system applications. In this paper, it is proposed a Reinforcement Learning (RL) model that learns how to efficiently restore a distribution system after a major outage. The proposed approach is based on a Monte Carlo Tree Search to expedite the training process. The proposed model strategy provides a robust decision-making tool for asynchronous and partial information scenarios. The results, validated with the IEEE 13-bus test feeder and IEEE 8500-node distribution test feeder, demonstrate the effectiveness and scalability of the proposed method. [ABSTRACT FROM AUTHOR]

To meet the execution requirements of delay-sensitive services in vehicular edge computing (VEC) networks, vehicular services need to be offloaded to edge computing nodes. For complex, large-scale services, the services need to be migrated if the services are not completed before the vehicles leave the coverage of edge computing nodes. Trust and resource matching between areas thus become major problems. This paper studies the decision model of vehicular service offloading and migration. First, software-defined network (SDN) technology is introduced into the traditional network architecture, and a two-layer distributed SDN-controlled VEC network architecture is designed, which is divided into a domain control layer and an area control layer. In this framework, we use the consortium blockchain as a carrier to share network topology information between SDN controllers to prevent information leakage. We then established a service offloading and migration optimization problem model to minimize service execution delay, reduce energy consumption and maximize the throughput of the blockchain system. We describe the problem model as a Markov Decision Process (MDP), introduce a deep reinforcement learning (DRL) algorithm named asynchronous advantage actor-critic (A3C) and design a dynamic service offloading and migration algorithm (DSOMA) based on A3C to solve the problem. Simulation results show that DSOMA can increase the throughput of the blockchain system, and DSOMA is superior to the deep Q-learning (DQN) algorithm and greedy offloading algorithm in reducing service execution delay and system energy consumption. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the problem of wireless service provisioning through a rotary-wing UAV which can serve as an aerial base station (BS) to communicate with multiple ground terminals (GTs) in a boost demand area. Our objective is to optimize the UAV control for maximizing the UAV.s energy efficiency, in which both aerodynamic energy and communication energy are considered while ensuring the communication requirements for each GT and backhaul link between the UAV and the terrestrial BS. The mobility of the UAV and GTs lead to time-varying channel conditions that make the environment dynamic. We formulate a nonconvex optimization for controlling the UAV considering the practical angle-dependent Rician fading channels between the UAV and GTs, and between the UAV and the terrestrial BS. Traditional optimization approaches are not able to handle the dynamic environment and high complexity of the problem in real-time. We propose to use a deep reinforcement learning-based approach namely Deep Deterministic Policy Gradient (DDPG) to solve the formulated nonconvex problem of UAV control with continuous action space that takes into account the real-time of the environment including time-varying UAV-ground channel conditions, available onboard energy of the UAV, and the communication requirement of the GTs. However, the DDPG method may not achieve good performance in an unstable environment and will face a large number of hyperparameters. We extend our approach to use the Trust Region Policy Optimization (TRPO) method that can improve the performance of the UAV compared to the DDPG method in such a dynamic environment. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) has seen an uptick in research interest in recent years, with many papers published in a plethora of different fields, topics and applications. A lot of that can be attributed to the recent advancements in machine learning (ML) and deep learning (DL) as a whole, the power of deep neural networks and the incorporation of them into reinforcement learning algorithms and techniques. This marriage of deep neural networks and traditional reinforcement learning methods has enabled reinforcement learning algorithms to achieve incredible results in use-cases that were previously deemed to be too complex. One of these highly complex scenarios is the realm of flight. In the last few years the number of papers published in this field has increased dramatically, but there is a fairly big divide between two major categories (quadcopters vs. fixed-wing aircraft). There is published work focusing on reinforcement learning using quadcopters, also known as drones, and there is published work focusing on fixed-wing aircraft, the more traditional aircraft, commonly referred to as airplanes. This paper focuses on the latter and will give an overview over the current state of reinforcement learning in the field of fixed-wing aviation. It will look into available toolkits usable for research as well as give an overview of different fields and tasks within reinforcement learning for fixed-wing aircraft control tasks. It contributes by providing detailed information about the methodologies used in each paper, while pointing out each paper’s contributions as well. A total of 48 papers on fixed-wing RL and airplane control are discussed. With the main focus lying on papers published post 2015, after the publication of Mnih et al.’s impactful deep RL paper.

With the development of artificial intelligence, there have been many attempts to incorporate artificial intelligence into algorithmic trading. In particular, reinforcement learning, which aims to solve dynamic decision-making problems, is attracting attention because of its high utilization in algorithmic trading. In this paper, we will implement a simple Deep Reinforcement Learning (DRL) trading robot to check the performance of DRL. In addition, we tried to find out how much performance improvement can be achieved by comparing a robot that learned a single stock data with a robot that learned stock data, market index, and interest rate data. This paper aims to develop a stock investment robot using a Proximal Policy Optimization (PPO) reinforcement learning algorithm and analyze the performance of the robot. The first robot used only the stock data of APPL INC, a single stock, as input, and the second robot used stock data of APPL INC and the S&P 500 index together with US interest rate data as input data. Afterward, the stock investment performance of the two robots for APPL INC was comparatively analyzed using the test data.

The Internet of Vehicles (IoV) as a promising application of Internet of Things (IoT) has played a significant role in autonomous driving, by connecting intelligent vehicles. Autonomous driving needs to process the mass environmental sensing data in coordination with surrounding vehicles, and makes an accurate driving judgment accordingly. Since the vehicles always have limited computing resources, processing these data in parallel with efficient task scheduling is one of the most important topics. Most current work focuses on formulating special scenarios and service requirements as optimization problems. However, the complicated and dynamic environment of vehicular computing is hard to model, predict and control, making those previous methods unscalable and unable to reflect the real scenario. In this paper, a Multi-task Deep reinforcement learning approach for scalable parallel Task Scheduling (MDTS) is firstly devised. For avoiding the curse of dimensionality when coping with complex parallel computing environments and jobs with diverse properties, we extend the action selection in Deep Reinforcement Learning (DRL) to a multi-task decision, where the output branches of multi- task learning are fine-matched to parallel scheduling tasks. Child tasks of a job are accordingly assigned to distributed nodes without any human knowledge while the resource competition among parallel tasks is leveraged through shared neural network layers. Moreover, we design an appropriate reward function to optimize multiple metrics simultaneously, with emphasis on specific scenarios. Extensive experiments show that the MDTS significantly increases the overall reward compared with least- connection scheduling and particle swarm optimization algorithm from −16.71, −0.67 to 2.93, respectively. [ABSTRACT FROM AUTHOR]

Dynamic topology, fast-changing channels and the time sensitivity of safety-related services present challenges to the status quo of resource allocation for cellular-underlaying vehicle-to-vehicle (V2V) communications. In this paper, we investigate a novel federated multi-agent deep reinforcement learning (FedMARL) approach for the decentralized joint optimization of channel selection and power control for V2V communication. The approach takes advantage of both deep reinforcement learning (DRL) and federated learning (FL), satisfying the reliability and delay requirements of V2V communication while maximizing the transmit rates of cellular links. Specifically, we elaborately construct individual V2V agent implement by the dueling double deep Q-network (D3QN), and design the reward function to train V2V agents collaboratively. As a result, each agent individually optimizes channel selection and power level based on its local observations, including the instantaneous channel state information (CSI) of corresponding V2V link, the instantaneous co-channel interference from the cellular link, the previous channels selections of nearby V2V pairs, and the queue backlog at the V2V transmitter. Another important aspect is that we incorporate FL to alleviate the training instability problem induced by cooperative multi-agent environment. The local DRL models of different V2V agents are federated periodically, addressing the limitations of partial observability on the entire network status for individual agent, and accelerating the training process of multi-agent learning. Validated via simulations, the proposed FedMARL scheme shows superiority to the baselines in terms of the cellular sum-rate and the V2V packet delivery rate. [ABSTRACT FROM AUTHOR]

The anti-jamming communication of the heterogeneous Internet of Satellites (IoS) has drawn increasing attentions due to the smart jamming and high dynamics. This paper investigates a spatial anti-jamming scheme for IoS, with the aim of minimizing anti-jamming routing cost via Stackelberg game and reinforcement learning. Firstly, we formulate the routing anti-jamming problem as a hierarchical anti-jamming Stackelberg game. It has been proved that there is a Stackelberg equilibrium (SE) in the proposed game. Secondly, the spatial anti-jamming scheme for IoS consists of two stages: the available routing selection and the fast anti-jamming decision. To tackle the high dynamics caused by the unknown interrupts and the unexpected congestion, we propose a deep reinforcement learning based routing algorithm (DRLR) to obtain an available routing subset; Furthermore, to make a fast anti-jamming decision, we propose a fast response anti-jamming algorithm (FRA) based on the available routing subset. The user utilizes DRLR and FRA algorithms to empirically analyze the jammer's strategies and adaptively make an anti-jamming decision according to the dynamic and unknown jamming environment. Finally, the simulations have shown that the proposed algorithm has lower routing cost and better anti-jamming performance than existing approaches, and the anti-jamming policies converge to the SE. [ABSTRACT FROM AUTHOR]

Adaptive bitrate (ABR) streaming has been used in wireless networks to deal with the time-varying wireless channels. Traditionally, wireless video is fetched from remote Internet server. However, wireless video streaming from Internet server faces challenges such as congestion and long latency. ABR streaming and video transcoding at the radio access network (RAN) edge have shown the potential to overcome such problem and provided better video streaming service. In the paper, we consider joint computation and communication for ABR streaming based on mobile edge computing (MEC) under time-varying wireless channels. We propose a joint video transcoding and quality adaptation framework for ABR streaming by enabling RAN with computing capability. By modeling the wireless channel as a finite state Markov channel, we formulate the optimization problem as a stochastic optimization problem of joint computational resource assignment and video quality adaptation for maximizing the average reward, which is defined as the tradeoff between user perceived quality of experience and the cost of performing transcoding at edge server. By using deep reinforcement learning (DRL) algorithm, we develop an automatic algorithm to perform the computational resource assignment and video quality adaptation without any prior knowledge of channel statistics. Simulation results using Tensorflow show the effectiveness of the designed MEC-enabled ABR streaming system and DRL algorithm. [ABSTRACT FROM AUTHOR]

In recent years, the growing development of Connected Autonomous Vehicles (CAV), Intelligent Transport Systems (ITS), and 5G communication networks have led to the advent of Autonomous Intersection Management (AIM) systems. AIMs present a new paradigm for CAV control in future cities, taking control of CAVs in scenarios where cooperation is necessary and allowing safe and efficient traffic flows, eliminating traffic signals. So far, the development of AIM algorithms has been based on basic control algorithms, without the ability to adapt or keep learning new situations. To solve this, in this paper we present a new advanced AIM approach based on end-to-end Multi-Agent Deep Reinforcement Learning (MADRL) and trained using Curriculum through Self-Play, called advanced Reinforced AIM (adv.RAIM). adv.RAIM enables the control of CAVs at intersections in a collaborative way, autonomously learning complex real-life traffic dynamics. In addition, adv.RAIM provides a new way to build smarter AIMs capable of proactively controlling CAVs in other highly complex scenarios. Results show remarkable improvements when compared to traffic light control techniques (reducing travel time by 59% or reducing time lost due to congestion by 95%), as well as outperforming other recently proposed AIMs (reducing waiting time by 56%), highlighting the advantages of using MADRL. [ABSTRACT FROM AUTHOR]

The renewable energy source (RES)-powered small cell base station (SBS) is a promising technology for the next-generation networks because RESs provide sustainable energy without being depleted. In RES-assisted networks, joint optimization of user association (UA) and power control (PC) is required to enhance the sum-rate performance by reducing inter-cell interference between SBSs. This paper tackles the UA and PC optimization for the sum-rate maximization under quality-of-service (QoS) and backhaul constraints. We formulate the problem as mixed-integer non-linear programming, in which UA and PC variables of different time slots are tightly coupled due to the RES energy dynamics model. Hence, designing a dynamic policy-based UA and PC with consideration of the future environments such as the quantity of channel gain and energy harvesting remains a challenge. First, we propose a deep unsupervised learning (DUL)-based UA scheme for a fixed PC variable. To lower the computational complexity and accelerate the convergence of the proposed learning-based optimization, we relax the UA variable, which originally has an extremely high dimension, into a low-dimensional continuous variable inspired by the Lagrangian method. Next, we propose a deep reinforcement learning (DRL)-based PC scheme, in which the stringent QoS and backhaul constraints are considered penalty terms on the reward design. The proposed DRL-based PC scheme facilitates dynamic PC inferring the relationship between the future environment and current PC. Simulation results demonstrate that the proposed scheme enhances the sum-rate by 10%, accommodates 3.3 percent point (%p) more QoS-qualified users, and reduces the computation time by 20 times compared to the conventional optimization-based method. [ABSTRACT FROM AUTHOR]

Autonomous vehicles have been highlighted as a major growth area for future transportation systems and the deployment of large numbers of these vehicles is expected when safety and legal challenges are overcome. To meet the necessary safety standards, effective collision avoidance technologies are required to ensure that the number of accidents are kept to a minimum. As large numbers of autonomous vehicles, operating together on roads, can be regarded as a swarm system, we propose a bio-inspired collision avoidance strategy using virtual pheromones; an approach that has evolved effectively in nature over many millions of years. Previous research using virtual pheromones showed the potential of pheromone-based systems to maneuver a swarm of robots. However, designing an individual controller to maximise the performance of the entire swarm is a major challenge. In this paper, we propose a novel deep reinforcement learning (DRL) based approach that is able to train a controller that introduces collision avoidance behaviour. To accelerate training, we propose a novel sampling strategy called Highlight Experience Replay and integrate it with a Deep Deterministic Policy Gradient algorithm with noise added to the weights and biases of the artificial neural network to improve exploration. To evaluate the performance of the proposed DRL-based controller, we applied it to navigation and collision avoidance tasks in three different traffic scenarios. The experimental results showed that the proposed DRL-based controller outperformed the manually-tuned controller in terms of stability, effectiveness, robustness and ease of tuning process. Furthermore, the proposed Highlight Experience Replay method outperformed than the popular Prioritized Experience Replay sampling strategy by taking 27% of training time average over three stages. [ABSTRACT FROM AUTHOR]

Autonomous pilot is crucial in integrally promoting the autonomy of an unmanned surface vehicle (USV). However, the integration mechanism of decision and control is still unclear within the entire autonomy. In this paper, by organically bridging path planning and tracking, an autonomous pilot framework with waypoints generation, path smoothing and policy guidance of a USV in congested waters is established, for the first time. Incorporating elite and diversity operations into the genetic algorithm (GA), an elite-duplication GA (EGA) strategy is devised to optimally generate sparse waypoints in a constrained space. The B-spline technique is further deployed to make flexibly smooth interpolation facilitating path smoothing supported by optimal sparse-waypoints. Seamlessly bridged by the parametric smooth path, deep reinforcement learning (DRL) technique is resorted to continuously extract in-depth pilotage policies, i.e., mappings from path tracking errors, collision risks and control constraints to continuous control forces/torques. Eventually, the entire spline-bridged EGA-DRL (SED) framework merits autonomous global-pilotage and local-reaction in an organically modular manner. Comprehensive validations and comparisons in various real-world geographies demonstrate the effectiveness and superiority of the proposed SED autonomous pilot framework. [ABSTRACT FROM AUTHOR]

Tile-based rate adaption can improve the quality of experience (QoE) for adaptive 360-degree video streaming under constrained network conditions, which, however, is a challenging problem due to the requirements of accurate prediction for users’ viewports and optimal bitrate allocation for tiles. In this paper, we propose a strategy that deploys reinforcement learning-based Rate Adaptation with adaptive Prediction and Tiling for 360-degree video streaming, named RAPT360, to address these challenges. Specifically, to improve the accuracy of the state-of-the-art viewport prediction approaches, we fit the time-varying Laplace distribution-based probability density function of the prediction error for different prediction lengths. On the basis of that, we develop a viewport identification method to determine the viewport area of a user depending on the buffer occupancy, where the obtained viewport can cover the real viewport with any given probability confidence level. We then propose a viewport-aware adaptive tiling scheme to improve the bandwidth efficiency, where three types of tile granularities are allocated according to the shape and position of the 2-D projection of that viewport. By establishing an adaptive streaming model and QoE metric specific to 360-degree videos, we finally formulate the rate adaptation problem for tile-based 360-degree video streaming as a non-linear discrete optimization problem that targets at maximizing the long-term user QoE under a bandwidth-constrained network. To efficiently solve this problem, we model the rate adaptation logic as a Markov decision process (MDP) and employ the deep reinforcement learning (DRL)-based algorithm to dynamically learn the optimal bitrate allocation of tiles. Extensive experimental results show that RAPT360 achieves a performance gain of at least 1.47 dB on average chunk QoE, including a video quality improvement of at least 1.33 dB, in comparison to the existing strategies for tile-based adaptive 360-degree video streaming. [ABSTRACT FROM AUTHOR]

In this paper, we develop centralized and distributed dynamic resource allocation schemes for an intelligent reflecting surface (IRS) aided energy harvesting (EH) system that optimize transmit power of a source node and phase-shift of passive reflecting elements of IRS. The source node randomly harvests renewable energy from the surrounding environment and performs data transmission with the harvested energy while satisfying statistical packet delay constraints in terms of maximum acceptable delay-outage probability. Our developed schemes do not require the statistical distributions of channel and energy profiles to be known and apply deep reinforcement learning (DRL) algorithm. Simulation results demonstrate the effectiveness of the proposed resource control scheme for IRS-aided EH system in different channel and energy conditions. [ABSTRACT FROM AUTHOR]

Offloading computation-intensive tasks (e.g., blockchain consensus processes and data processing tasks) to the edge/cloud is a promising solution for blockchain-empowered mobile edge computing. However, the traditional offloading approaches (e.g., auction-based and game-theory approaches) fail to adjust the policy according to the changing environment and cannot achieve long-term performance. Moreover, the existing deep reinforcement learning-based offloading approaches suffer from the slow convergence caused by high-dimensional action space. In this paper, we propose a new model-free deep reinforcement learning-based online computation offloading approach for blockchain-empowered mobile edge computing in which both mining tasks and data processing tasks are considered. First, we formulate the online offloading problem as a Markov decision process by considering both the blockchain mining tasks and data processing tasks. Then, to maximize long-term offloading performance, we leverage deep reinforcement learning to accommodate highly dynamic environments and address the computational complexity. Furthermore, we introduce an adaptive genetic algorithm into the exploration of deep reinforcement learning to effectively avoid useless exploration and speed up the convergence without reducing performance. Finally, our experimental results demonstrate that our algorithm can converge quickly and outperform three benchmark policies. [ABSTRACT FROM AUTHOR]

It is essential to develop proper energy management strategies (EMSs) with broad adaptability for hybrid electric vehicles (HEVs). This paper utilizes deep reinforcement learning (DRL) to develop EMSs for a series HEV due to DRL's advantages of requiring no future driving information in derivation and good generalization in solving energy management problem formulated as a Markov decision process. History cumulative trip information is also integrated for effective state of charge guidance in DRL-based EMSs. The proposed method is systematically introduced from offline training to online applications; its learning ability, optimality, and generalization are validated by comparisons with fuel economy benchmark optimized by dynamic programming, and real-time EMSs based on model predictive control (MPC). Simulation results indicate that without a priori knowledge of future trip, original DRL-based EMS achieves an average 3.5% gap from benchmark, superior to MPC-based EMS with accurate prediction; after further applying output frequency adjustment, a mean gap of 8.7%, which is comparable with MPC-based EMS with mean prediction error of 1 m/s, is maintained with concurrently noteworthy improvement in reducing engine start times. Besides, its impressive computation speed of about 0.001 s per simulation step proves its practical application potential, and this method is independent of powertrain topology such that it is applicative for any type of HEVs even when future driving information is unavailable. [ABSTRACT FROM AUTHOR]

Buffer-aware video streaming, which exploits the available storage space in user device to store the prefetched video data in good channels for video data use in poor channels, has been proved to have the potential to reduce the impact of fluctuating wireless channels on user-perceived video performance. However, in practical wireless networks, due to the unknown channel state and video rate, providing buffer-aware video streaming service to wireless user is a challenging problem. In this paper, with the aim to design an autonomous wireless video streaming system, we apply the deep reinforcement learning approach to dynamic resource optimization for wireless buffer-aware video streaming under unknown channel state and video rate. Specifically, we define a reward function for buffer-aware video streaming as the effective video streaming time when neither video-playback overflow nor video-playback underflow occurs. We propose a Markov decision process based problem formulation of the joint bandwidth allocation and buffer management for maximizing the effective video streaming time of all users. The optimal bandwidth allocation and buffer management policy is learned from training a deep neural network based on a deep reinforcement learning algorithm. We simulate the proposed algorithm in Tensorflow. Simulation results verify that the proposed deep reinforcement learning approach is effective for buffer-aware video streaming in wireless networks. [ABSTRACT FROM AUTHOR]

Recently, the rapid advancement of high computing platforms has accelerated the development and applications of artificial intelligence techniques. Deep learning, which has been regarded as the next paradigm to revolutionize users' experiences, has attracted networking researchers' interests to relieve the burden due to the exponentially growing traffic and increasing complexities. Various intelligent packet transmission strategies have been proposed to tackle different network problems. However, most of the existing research just focuses on the network related improvements and neglects the analysis about the computation consumptions. In this paper, we propose a Value Iteration Architecture based Deep Learning (VIADL) method to conduct routing design to address the limitations of existing deep learning based routing algorithms in dynamic networks. Besides the network performance analysis, we also study the complexity of our proposal as well as the resource consumptions in different deployment manners. Moreover, we adopt the Heterogeneous Computing Platform (HCP) to conduct the training and running of the proposed VIADL since the theoretical analysis demonstrates the significant reduction of the time complexity with the multiple GPUs in HCPs. Furthermore, simulation results demonstrate that compared with the existing deep learning based method, our proposal can guarantee more stable network performance when network topology changes. [ABSTRACT FROM AUTHOR]

Device-to-Device (D2D) communication with short communication distance is an efficient way to improve spectrum efficiency and mitigate interference. To realize the optimal resource configuration including wireless channel matching and power allocation, a distributed resource matching scheme is proposed based on deep reinforcement learning(DRL). The reward is defined as the difference of achieve rate of D2D users and the consumed power, which is limited by the Signal to Interference plus Noise Ratio (SINR) of the other cellular users on the current channel. The proposed algorithm maximizes the D2D throughput and energy efficiency in a distributed manner, without online coordination and message exchange between users. The considered resource allocation problem is formulated as a random non-cooperative game with multiple players (D2D pairs), where each player is a learning agent, whose task is to learn its best strategy based on locally observed information, multi-user communication resource matching algorithm is proposed based on a Double Deep Q-network (DDQN), where the total cellular throughput and user energy efficiency could converge to the Nash equilibrium (NE) under the mixed strategy. Simulation results show that the proposed algorithm can improve the communication rate and energy efficiency of each user by selecting the optimal strategy, and has better convergence performance compared with existing schemes. [ABSTRACT FROM AUTHOR]

The development of Autonomous Vehicles (AVs) envisions the promising technology of future Intelligent Transportation Systems (ITS). However, the complex road structures and increased vehicles cause traffic congestion and road safety, which eventually leads to horrible accidents. Cooperative driving of AVs, a groundbreaking initiative of vehicle platooning, epitomizes the next wave in vehicular technology through minimizing accident risks, transport times, costs, energy, and fuel consumption. However, the traditional machine learning-based platooning approaches fail to regulate the policy with the dynamic feature of AVs. This paper proposes a hybrid Deep Reinforcement learning and Genetic algorithm for Smart-Platooning (DRG-SP) the AVs. The leverage of the deep reinforcement learning mechanism addresses the computational complexity and accommodates the high dynamic platoon environments. Adopting the Genetic Algorithm in Deep Reinforcement learning overcomes the slow convergence problem and offers long-term performance. The simulation results reveal that the Smart-Platooning effectively forms and maintains the platoons by minimizing traffic congestion and fuel consumption. [ABSTRACT FROM AUTHOR]

Currently, self-driving, emerging as a key automatic application, has brought a huge potential for the provision of in-vehicle services (e.g., automatic path planning) to mitigate urban traffic congestion and enhance travel safety. To provide high-quality vehicular services with stringent delay constraints, edge computing (EC) enables resource-hungry self-driving vehicles (SDVs) to offload computation-intensive tasks to the edge servers (ESs). In addition, caching highly reusable contents decreases the redundant transmission time and improves the quality of services (QoS) of SDVs, which is envisioned as a supplement to the computation offloading. However, the high mobility and time-varying requests of SDVs make it challenging to provide reliable offloading decisions while guaranteeing the resource utilization of content caching. To this end, in this paper we propose a collaborative computation offloading and content caching method, named CoPace, by leveraging deep reinforcement learning (DRL) in EC for self-driving system. Specifically, we first introduce OSTP to predict the future time-varying content popularity, taking into account the temporal-spatial attributes of requests. Moreover, a DRL-based algorithm is developed to jointly optimize the offloading and caching decisions, as well as the resource allocation (i.e., computing and communication resources) strategies. Extensive experiments with real-world datasets in Shanghai, China, are conducted to evaluate the performance, which demonstrates that CoPace is both effective and well-performed. [ABSTRACT FROM AUTHOR]

In the multi-server multi-access edge computing (MEC) system, the computing capacity could be insufficient and the computing resource could be over-utilized when an excessive number of computing tasks are offloaded for execution. To alleviate the excessive burden in the multi-server MEC system and tap the underutilized resources of wired edge devices, which are called edge computers in this paper, we investigate the computation offloading and shunting scheme in the wireless wireline internetwork. We formulate an optimization problem to minimize the average total time delay and use the deep reinforcement learning method to obtain the optimal shunting policy. By sensing the state of the wireless wireline internetwork, the proposed scheme can adaptively adjust the shunting ratio and utilize the resources of MEC servers and edge computers intelligently and jointly. Finally, we evaluate the computational complexity and validate the convergence property of this scheme. The extensive simulation results demonstrate that this scheme can improve the resource productivity and reduce the average total time delay. [ABSTRACT FROM AUTHOR]

Advanced algorithms can promote the development of energy management strategies (EMSs) as a key technology in hybrid electric vehicles (HEVs). Reinforcement learning (RL) with distributed structure can significantly improve training efficiency in complex environments, and multi-threaded parallel computing provides a reliable algorithm basis for promoting adaptability. Dedicated to trying more efficient deep reinforcement learning (DRL) algorithms, this paper proposed a deep q-network (DQN)-based energy and emission management strategy (E&EMS) at first. Then, two distributed DRL algorithms, namely asynchronous advantage actor-critic (A3C) and distributed proximal policy optimization (DPPO), were adopted to propose EMSs, respectively. Finally, emission optimization was taken into account and then distributed DRL-based E&EMSs were proposed. Regarding dynamic programming (DP) as the optimal benchmark, simulation results show that three DRL-based control strategies can achieve near-optimal fuel economy and outstanding computational efficiency, and compared with DQN, two distributed DRL algorithms have improved the learning efficiency by four times. [ABSTRACT FROM AUTHOR]

Device-to-device (D2D) communication affords the benefits of improved network spectral efficiency, throughput, energy efficiency, and delay performance. However, D2D communication underlaid cellular uplinks are degraded by interference between D2D user equipment (DUE) and cellular user equipment (CUE) that use the same frequency band. In this context, researchers have proposed the optimization of the transmit power of the devices to mitigate interference. This power allocation problem is generally modeled as a NP-hard combinatorial optimization problem with linear constraints; therefore, traditional optimization methods are ineffective in addressing the problem. In this paper, we apply a deep reinforcement learning (DRL) algorithm to optimize the transmit power for both DUEs and CUEs in the context of D2D communication underlaid uplink cellular networks, which can solve the optimization problem via optimal decision-making together with efficient deep network training. Our simulation results show that the proposed algorithm affords a near-global-optimum solution with lower computation complexity than exhaustive search, and outperforms the fixed-power, weighted minimum mean square error (WMMSE), gradient descent method, and Newton-Raphson approaches. [ABSTRACT FROM AUTHOR]

Network slicing is a key paradigm in 5G and is expected to be inherited in future 6G networks for the concurrent provisioning of diverse quality of service (QoS). Unfortunately, effective slicing of Radio Access Networks (RAN) is still challenging due to time-varying network situations. This paper proposes a new intelligent RAN slicing strategy with two-layered control granularity, which aims at maximizing both the long-term QoS of services and spectrum efficiency (SE) of slices. The proposed method consists of an upper-level controller to ensure the QoS performance, which enforces loose control by performing adaptive slice configuration according to the long-term dynamics of service traffic. The lower-level controller is to improve SE of slices, by tightly scheduling radio resources to users at the small time-scale. To realize the proposed RAN slicing strategy, we propose a model-free deep reinforcement learning (DRL) framework, which is a hierarchical structure that collaboratively integrating the modified deep deterministic policy gradient (DDPG) and double deep-Q-network algorithm. Specifically, the lower-level control problem is a mixed-integer stochastic optimization problem with multiple constraints. This kind of problem is hard to be directly solved by the exiting DRL algorithms, since it involves searching for the solution in a vast set of mixed-integer action space, which will induce unbearable computational complexity. Thus, we propose a novel action space reducing approach, embedding the convex optimization tools into the DDPG algorithm, to speed up the lower-level control. Furthermore, simulation results confirm the effectiveness of our proposed intelligent RAN slicing scheme. [ABSTRACT FROM AUTHOR]

Unmanned aerial vehicles (UAVs) have emerged as a promising candidate solution for data collection of large-scale wireless sensor networks (WSNs). In this paper, we investigate a UAV-aided WSN, where cluster heads (CHs) receive data from their member nodes, and a UAV is dispatched to collect data from CHs. We aim to minimize the total energy consumption of the UAV-WSN system in a complete round of data collection. Toward this end, we formulate the energy consumption minimization problem as a constrained combinatorial optimization problem by jointly selecting CHs from clusters and planning the UAV's visiting order to the selected CHs. The formulated energy consumption minimization problem is NP-hard, and hence, hard to solve optimally. To tackle this challenge, we propose a novel deep reinforcement learning (DRL) technique, pointer network-A* (Ptr-A*), which can efficiently learn the UAV trajectory policy for minimizing the energy consumption. The UAV's start point and the WSN with a set of pre-determined clusters are fed into the Ptr-A*, and the Ptr-A* outputs a group of CHs and the visiting order of CHs, i.e., the UAV's trajectory. The parameters of the Ptr-A* are trained on small-scale clusters problem instances for faster training by using the actor-critic algorithm in an unsupervised manner. Simulation results show that the trained models based on 20-clusters and 40-clusters have a good generalization ability to solve the UAV's trajectory planning problem in WSNs with different numbers of clusters, without retraining the models. Furthermore, the results show that our proposed DRL algorithm outperforms two baseline techniques. [ABSTRACT FROM AUTHOR]

Due to the inherent openness of wireless channels, data transmission in user-centric ultra-dense networks (UUDNs) is vulnerable to eavesdropping and malicious jamming attacks (EJA). This paper addresses the user-centric clustering, beamforming, and artificial noise design to combat with the EJA, and an accelerated deep reinforcement learning (DRL) based approach has been proposed for the downlink of UUDNs from a security, reliability and energy efficiency (SREE) perspective. In contrast to the existing approaches, the proposed approach divides the UE's communication behavior into association phase and transmission phase to perform a fast and adaptive response to the EJA. For the association phase, the proposed approach aims at achieving a fast response to EJA by optimization algorithms from a short-term perspective. For the transmission phase, we propose a DRL based algorithm to achieve optimal adaptive response by dealing with the dynamic and persistent effects of EJA from a long-term perspective. To accelerate the training process, fast response achieved in the association phase will be regarded as the initial experience to update the parameters of the deep neural networks adopted by the DRL based algorithm for transmission phase. Comparing with the existing methods, simulation results show that our proposed accelerated DRL approach achieves better performance in terms of SREE and convergence speed. [ABSTRACT FROM AUTHOR]

Performing deep neural network (DNN) inference in real time requires excessive network resources, which poses a big challenge to the resource-limited industrial Internet of things (IIoT) networks. To address the challenge, in this paper, we introduce an end-edge-cloud orchestration architecture, in which the inference task assignment and DNN model placement are flexibly coordinated. Specifically, the DNN models, trained and pre-stored in the cloud, are properly placed at the end and edge to perform DNN inference. To achieve efficient DNN inference, a multi-dimensional resource management problem is formulated to maximize the average inference accuracy while satisfying the strict delay requirements of inference tasks. Due to the mix-integer decision variables, it is difficult to solve the formulated problem directly. Thus, we transform the formulated problem into a Markov decision process which can be solved efficiently. Furthermore, a deep reinforcement learning based resource management scheme is proposed to make real-time optimal resource allocation decisions. Simulation results are provided to demonstrate that the proposed scheme can efficiently allocate the available spectrum, caching, and computing resources, and improve average inference accuracy by 31.4 % compared with the deep deterministic policy gradient benchmark. [ABSTRACT FROM AUTHOR]

Vehicular networks are vulnerable to various attacks from malicious vehicles or infrastructures within a network. The collaborative misbehavior detection system can be used to detect these internal or insider attacks. However, in a collaborative misbehavior detection system, an attacker may lower the detection accuracy by sending false feedback. A trust model can be used to stimulate vehicles to send true feedbacks. However, an attacker can take advantage of weak or strong reputation update methods. A dynamic trust or reputation update policy can be used to stimulate vehicles to send true feedbacks. In this paper, we propose a deep reinforcement learning based dynamic reputation update policy. In the proposed scheme, feedbacks from vehicles are combined in vehicular edge computing (VEC) servers using Dempster-Shafer theory and the results are used to predict the average number of true messages. VEC then uses deep reinforcement learning to determine the optimum reputation update policy to stimulate vehicles to send true feedbacks. In addition, through extensive simulations, we show that the proposed dynamic reputation-policy is better in terms of the average number of true feedbacks compared to the existing reputation update policy. [ABSTRACT FROM AUTHOR]

In this paper, we consider a multi-access edge computing (MEC) network with multiple servers. Due to the low latency constraints, the wireless data transmission/offloading is carried by finite blocklength codes. We characterize the reliability of the transmission phase in the finite blocklength regime and investigate the extreme event of queue length violation in the computation phase by applying extreme value theory. Under the assumption of perfect channel state information (CSI), we follow the obtained characterizations and provide an optimal framework design including server selection and time allocation aiming to minimize the overall error probability. Moreover, when only the outdated CSI is available, a deep reinforcement learning based design is proposed applying the deep deterministic policy gradient method. Via simulations, we validate the convexity proven in our analytical model and show the performance advantage of proposed analytical solution and learning-based solution comparing to the benchmark for perfect CSI and outdated CSI, respectively. [ABSTRACT FROM AUTHOR]

Autonomous exploration is an important application of multi-vehicle systems, where a team of networked robots are coordinated to explore an unknown environment collaboratively. This technique has earned significant research interest due to its usefulness in search and rescue, fault detection and monitoring, localization and mapping, etc. In this paper, a novel cooperative exploration strategy is proposed for multiple mobile robots, which reduces the overall task completion time and energy costs compared to conventional methods. To efficiently navigate the networked robots during the collaborative tasks, a hierarchical control architecture is designed which contains a high-level decision making layer and a low-level target tracking layer. The proposed cooperative exploration approach is developed using dynamic Voronoi partitions, which minimizes duplicated exploration areas by assigning different target locations to individual robots. To deal with sudden obstacles in the unknown environment, an integrated deep reinforcement learning based collision avoidance algorithm is then proposed, which enables the control policy to learn from human demonstration data and thus improve the learning speed and performance. Finally, simulation and experimental results are provided to demonstrate the effectiveness of the proposed scheme. [ABSTRACT FROM AUTHOR]

Network slicing (NS) has been widely identified as a key architectural technology for 5G-and-beyond systems by supporting divergent requirements in a sustainable way. In radio access network (RAN) slicing, due to the device-base station (BS)-NS three layer association relationship, device association (including access control and handoff management) becomes an essential yet challenging issue. With the increasing concerns on stringent data security and device privacy, exploiting local resources to solve device association problem while enforcing data security and device privacy becomes attractive. Fortunately, recently emerging federated learning (FL), a distributed learning paradigm with data protection, provides an effective tool to address this type of issues in mobile networks. In this paper, we propose an efficient device association scheme for RAN slicing by exploiting a hybrid FL reinforcement learning (HDRL) framework, with the aim to improve network throughput while reducing handoff cost. In our proposed framework, individual smart devices train a local machine learning model based on local data and then send the model features to the serving BS/encrypted party for aggregation, so as to efficiently reduce bandwidth consumption for learning while enforcing data privacy. Specifically, we use deep reinforcement learning to train the local model on smart devices under a hybrid FL framework, where horizontal FL is employed for parameter aggregation on BS, while vertical FL is employed for NS/BS pair selection aggregation on the encrypted party. Numerical results show that the proposed HDRL scheme can achieve significant performance gain in terms of network throughput and communication efficiency in comparison with some state-of-the-art solutions. [ABSTRACT FROM AUTHOR]

In this paper, we propose an identity-preserving face hallucination (IPFH) method via deep reinforcement learning. Most existing methods ultra-resolve facial visual information in guidance of appearance similarity which rarely attend to recovering the semantic property, undermining further face analysis (e.g., recognition). We present a visual-semantic hallucinator relying on deep reinforcement learning to adaptively repair local details for the restoration of both identity and appearance characteristics. Specifically, we first capture the facial global topology structure to roughly recover the visual information with the pixel-wise similarity constraint. To super-resolve more photo-realistic faces, we explore the contextual interdependency to reconstruct facial local textural details (e.g., over-smoothed edges) with the constraints of visual and identity similarity. In terms of the visual similarity constraint, we develop the dual domain network with bidirectional consistency on both HR domain and LR domain to improve the appearance quality. Moreover, we introduce the identity constraint to encourage hallucinated faces to satisfy the identity property. Experimental results on several benchmarks demonstrate our method achieves promising performance on the recovery of visual and semantic information. [ABSTRACT FROM AUTHOR]

This paper proposes a demand response method to reduce the long-term charging cost of single plug-in electric vehicles (PEV) while overcoming obstacles such as the stochastic nature of the user's driving behaviour, traffic condition, energy usage, and energy price. The problem is formulated as a Markov Decision Process (MDP) with an unknown transition probability matrix and solved using deep reinforcement learning (RL) techniques. The proposed method does not require any initial data on the PEV driver's behaviour and shows improvement on learning speed when compared to a pure model-free reinforcement learning method. A combination of model-based and model-free learning methods called Dyna-Q reinforcement learning is utilized in our strategy. Every time a real experience is obtained, the model is updated, and the RL agent will learn from both the real experience and “imagined” experiences from the model. Due to the vast amount of state space, a table-lookup method is impractical, and a value approximation method using deep neural networks is employed for estimating the long-term expected reward of all state-action pairs. An average of historical price and a long short-term memory (LSTM) network are used to predict future price. Simulation results demonstrate the effectiveness of this approach and its ability to reach an optimal policy quicker while avoiding state of charge (SOC) depletion during trips when compared to existing PEV charging schemes. [ABSTRACT FROM AUTHOR]

The increasing number of low power communication devices in the era of Internet of Things (IoT) has resulted in challenges in maintenance and continuous energy supply. In this paper, the ambient energy harvesting is integrated with wireless local area network (WLAN), where low-power users adopt a distributed coordination strategy for their random access, namely ambient energy harvesting carrier-sense-multiple-access with collision avoidance (AEH-CSMA/CA). All users harvest ambient energy to power their data uploading. In order to increase the uplink throughput and to reduce the energy outage probability, a classic deep reinforcement learning (DRL) algorithm, namely deep Q-learning (DQL), is adopted by a smart user for intelligently adjusting its initial backoff window size during the backoff process. Simulation results demonstrate that the proposed DQL algorithm is capable of substantially improving the throughput performance, while gauranteeing the energy outage probability of the smart user lower than a certain threshold. [ABSTRACT FROM AUTHOR]