Search

Bi-Level Optimization (BLO) is originated from the area of economic game theory and then introduced into the optimization community. BLO is able to handle problems with a hierarchical structure, involving two levels of optimization tasks, where one task is nested inside the other. In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems, such as hyper-parameter optimization, multi-task and meta learning, neural architecture search, adversarial learning and deep reinforcement learning, actually all contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of BLO. Then we construct a best-response-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies, covering aspects ranging from fundamental automatic differentiation schemes to various accelerations, simplifications, extensions and their convergence and complexity properties. Last but not least, we discuss the potentials of our unified BLO framework for designing new algorithms and point out some promising directions for future research. A list of important papers discussed in this survey, corresponding codes, and additional resources on BLOs are publicly available at: https://github.com/vis-opt-group/BLO. [ABSTRACT FROM AUTHOR]

This paper proposes a novel scalable reinforcement learning approach for simultaneous routing and spectrum access in wireless ad-hoc networks. In most previous works on reinforcement learning for network optimization, the network topology is assumed to be fixed, and a different agent is trained for each transmission node—this limits scalability and generalizability. Further, routing and spectrum access are typically treated as separate tasks. Moreover, the optimization objective is usually a cumulative metric along the route, e.g., number of hops or delay. In this paper, we account for the physical-layer signal-to-interference-plus-noise ratio (SINR) in a wireless network and further show that bottleneck objective such as the minimum SINR along the route can also be optimized effectively using reinforcement learning. Specifically, we propose a scalable approach in which a single agent is associated with each flow and makes routing and spectrum access decisions as it moves along the frontier nodes. The agent is trained according to the physical-layer characteristics of the environment using a novel rewarding scheme based on the Monte Carlo estimation of the future bottleneck SINR. It learns to avoid interference by intelligently making joint routing and spectrum allocation decisions based on the geographical location information of the neighbouring nodes. [ABSTRACT FROM AUTHOR]

Ride-sourcing services are now reshaping the way people travel by effectively connecting drivers and passengers through mobile internets. Online matching between idle drivers and waiting passengers is one of the most key components in a ride-sourcing system. The average pickup distance or time is an important measurement of system efficiency since it affects both passengers’ waiting time and drivers’ utilization rate. It is naturally expected that a more effective bipartite matching (with smaller average pickup time) can be implemented if the platform accumulates more idle drivers and waiting passengers in the matching pool. A specific passenger request can also benefit from a delayed matching since he/she may be matched with closer idle drivers after waiting for a few seconds. Motivated by the potential benefits of delayed matching, this paper establishes a two-stage framework which incorporates a combinatorial optimization and multi-agent deep reinforcement learning methods. The multi-agent reinforcement learning methods are used to dynamically determine the delayed time for each passenger request (or the time at which each request enters the matching pool), while the combinatorial optimization conducts an optimal bipartite matching between idle drivers and waiting passengers in the matching pool. Four tailored reinforcement learning methods, delayed multi-agent deep Q learning (Delayed-M-DQN), delayed multi-agent actor-critic (Delayed-M-A2C), delayed multi-agent Proximal Policy Optimization (Delayed-M-PPO), and delayed multi-agent actor-critic with experience replay (Delayed-M-ACER), are developed. Through extensive empirical experiments with a well-designed simulator, we show that the proposed framework is able to remarkably improve system performances, by well balancing the trade-off among pick-up time, matching time, successful matching rate. [ABSTRACT FROM AUTHOR]

This paper proposes a deep reinforcement learning-based video streaming scheme for mobility-aware vehicular networks, e.g., vehicles on the highway. We consider infrastructure-assisted and mmWave-based scenarios in which the macro base station (MBS) cannot directly provide the streaming service to vehicles due to the short range of mmWave beams so that small mmWave base stations (mBSs) along the road deliver the desired videos to users. Motivated by the fact that a video stream consists of sequential chunks and smooth streaming has to be provided while users pass through multiple mBSs at high speed, the MBS proactively pushes some video chunks of the desired contents to mBSs. This is done to support vehicles that are currently covered and/or will be by each mBS. We formulate the dynamic video delivery scheme that adaptively determines 1) which content, 2) what quality and 3) how many chunks to be proactively delivered from the MBS to mBSs using Markov decision process (MDP). Since it is difficult for the MBS to track all the channel conditions and the network states have extensive dimensions, we adopt the deep deterministic policy gradient (DDPG) algorithm for the DRL-based video delivery scheme. This paper finally shows that the DRL agentlearns a streaming policy that pursues high average quality while limiting packet drops, avoiding playback stalls, reducing quality fluctuations and saving backhaul usage. [ABSTRACT FROM AUTHOR]

The Internet of vehicles (IoV) is a large information interaction network that collects information on vehicles, roads and pedestrians. One of the important uses of vehicle networks is to meet the entertainment needs of driving users through communication between vehicles and roadside units (RSUs). Due to the limited storage space of RSUs, determining the content cached in each RSU is a key challenge. With the development of 5G and video editing technology, short video systems have become increasingly popular. Current widely used cache update methods, such as partial file precaching and content popularity- and user interest-based determination, are inefficient for such systems. To solve this problem, this paper proposes a QoE-driven edge caching method for the IoV based on deep reinforcement learning. First, a class-based user interest model is established. Compared with the traditional file popularity- and user interest distribution-based cache update methods, the proposed method is more suitable for systems with a large number of small files. Second, a quality of experience (QoE)-driven RSU cache model is established based on the proposed class-based user interest model. Third, a deep reinforcement learning method is designed to address the QoE-driven RSU cache update issue effectively. The experimental results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

This paper proposes an effective and novel multi-agent deep reinforcement learning (MADRL)-based method for solving the joint virtual network function (VNF) placement and routing (P&R), where multiple service requests with differentiated demands are delivered at the same time. The differentiated demands of the service requests are reflected by their delay- and cost-sensitive factors. We first construct a VNF P&R problem to jointly minimize a weighted sum of service delay and resource consumption cost, which is NP-complete. Then, the joint VNF P&R problem is decoupled into two iterative subtasks: placement subtask and routing subtask. Each subtask consists of multiple concurrent parallel sequential decision processes. By invoking the deep deterministic policy gradient method and multi-agent technique, an MADRL-P&R framework is designed to perform the two subtasks. The new joint reward and internal rewards mechanism is proposed to match the goals and constraints of the placement and routing subtasks. We also propose the parameter migration-based model-retraining method to deal with changing network topologies. Corroborated by experiments, the proposed MADRL-P&R framework is superior to its alternatives in terms of service cost and delay, and offers higher flexibility for personalized service demands. The parameter migration-based model-retraining method can efficiently accelerate convergence under moderate network topology changes. [ABSTRACT FROM AUTHOR]

This paper is concerned with the efficient movement control of multiple drone cells for communication coverage. Although many works have been developed to cope with this problem, but only few of them have investigated the three-dimensional (3-D) continuous movement control of multiple drone cells. In this paper, a problem of 3-D continuous movement control of multiple drone cells is formulated with an objective of maximizing the energy-efficient communication coverage of drone-cell networks while preserving the network connectivity. To mitigate this challenging problem, an energy-efficiency and continuous-movement-control algorithm (E $^2$ CMC), which is based on an emerging deep reinforcement learning method, is proposed. In E $^2$ CMC, an energy-efficiency reward function considering the energy consumption, the sum of quality-of-service (QoS) requirements of users as well as the coverage fairness is first designed. Next, E $^2$ CMC learns to adjust drone cells’ locations continuously by interacting with an environment in a sequence of observations, actions, and rewards. Furthermore, E $^2$ CMC will reduce the reward drastically as long as the networks are disconnected. Simulation results verify the superiority of the proposed learning algorithm on deploying multiple drone cells to provide the communication coverage. [ABSTRACT FROM AUTHOR]

In this paper, we study the three-dimensional (3D) path planning problem for cellular-connected unmanned aerial vehicle (UAV) taking into account the impact of 3D antenna radiation patterns. The cellular-connected UAV has a mission to travel from an initial location to a destination. In this process, there is a trade-off between the flight time and the expected communication outages duration, which requires planning a suitable route to avoid the weak coverage region. However, as the existing cellular networks are primarily designed for ground coverage, the communication coverage in the sky tends to be intermittent and irregular due to the potential for severe interference and obstructions (such by buildings), which brings great challenges to 3D path planning. To address this challenge, we first construct a 3D coverage map that stores the expected outage probability over each locations. We then propose a multi-step dueling DDQN (multi-step D3QN) based algorithm to design the local optimal UAV path by leveraging the constructed coverage map. In this algorithm, the UAV acts as the agent to learn the appropriate action to complete the flight mission. Numerical results show the effectiveness of the proposed algorithm for connectivity-aware UAV path planning and the superiority of 3D path design over its 2D counterparts. [ABSTRACT FROM AUTHOR]

In this paper, we consider an underwater linear network, where an autonomous underwater vehicle (AUV) gathers data from a set of underwater devices. The AUV monitors a set of physical processes, where the status of each process can be sensed by one or more devices and each device is capable of sensing one or more processes. The AUV needs to maintain freshness of its information status about the monitored processes. To quantify the freshness of the information at the AUV, we consider the concept of the age of information (AoI), which represents the amount of time elapsed since the most recently delivered update information was generated. A framework is proposed to optimize the AUV's linear movement trajectory and scheduling of process status updates with the objective of minimizing the normalized weighted sum of the average AoI of the monitored physical processes. The formulated optimization problem is a non-convex mixed integer problem, which cannot be solved by the standard optimization techniques. We develop a solution approach based on the technique of deep reinforcement learning (DRL). Specifically, we leverage an actor-critic DRL approach to find the optimum locations and stopping time of the data gathering points. Simulation results illustrate that the proposed framework maintains robustness under different scenarios and provides better performance when compared with baseline and $K$ -means clustering approaches. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the problem of wireless service provisioning through a rotary-wing UAV which can serve as an aerial base station (BS) to communicate with multiple ground terminals (GTs) in a boost demand area. Our objective is to optimize the UAV control for maximizing the UAV.s energy efficiency, in which both aerodynamic energy and communication energy are considered while ensuring the communication requirements for each GT and backhaul link between the UAV and the terrestrial BS. The mobility of the UAV and GTs lead to time-varying channel conditions that make the environment dynamic. We formulate a nonconvex optimization for controlling the UAV considering the practical angle-dependent Rician fading channels between the UAV and GTs, and between the UAV and the terrestrial BS. Traditional optimization approaches are not able to handle the dynamic environment and high complexity of the problem in real-time. We propose to use a deep reinforcement learning-based approach namely Deep Deterministic Policy Gradient (DDPG) to solve the formulated nonconvex problem of UAV control with continuous action space that takes into account the real-time of the environment including time-varying UAV-ground channel conditions, available onboard energy of the UAV, and the communication requirement of the GTs. However, the DDPG method may not achieve good performance in an unstable environment and will face a large number of hyperparameters. We extend our approach to use the Trust Region Policy Optimization (TRPO) method that can improve the performance of the UAV compared to the DDPG method in such a dynamic environment. [ABSTRACT FROM AUTHOR]

The flow fluctuations in the highly dynamic Internet of Vehicles (IoV) make the IoV difficult to provide reliable and scalable wireless network services for the emerging applications in the 5 G and beyond era. The software-defined networks (SDN) could feasibly manage and optimize the network according to the network load. Controller placement is a critical problem in SDN to achieve its robustness and flexibility with the changes of network status. Motivated by the advantages of SDN and Mobile-edge computing (MEC), this paper aims at enhancing the performance of IoV with the assistance of these two. Specifically, we consider a three-layer hierarchical SDN control plane for the IoV, where the SDN controllers are placed at the edge of networks. Under this framework, we investigate a multi-objective optimization problem on controller placement problem including delay, load balancing, and path reliability. To efficiently solve the formulated NP-hard problems, we develop an algorithm based on multi-agent deep Q-learning networks (MADQN) because of its advantages for large-scale combinatorial optimization. At last, we use multi-process technology to accelerate the operation of the algorithm, so as to complete the algorithm iteration faster. Numerical results show that the proposed methods achieve better performances than three baselines. [ABSTRACT FROM AUTHOR]

The vehicular edge computing is considered an enabling technology for intelligent and connected vehicles since the optimization of communication and computing on edge has a significant impact on driving safety and efficiency. In this paper, with the road traffic assignment to “proactively” reshape the spatiotemporal distribution of resource demands, we investigate the joint service migration and mobility optimization problem for vehicular edge computing. The goal is to meet the service delay requirements of vehicular edge computing with minimum migration cost and travel time. As service migration and mobility optimization are coupled, the joint scheduling problem suffers from the curse of dimensionality, which cannot be solved in real time by centralized algorithms. To this end, a multi-agent deep reinforcement learning (MADRL) algorithm is proposed to maximize the composite utility of communication, computing, and route planning in a distributed way. In the MADRL algorithm, a two-branch convolution based deep Q-network is constructed to coordinate migration action and routing action. Extensive experimental results show that the proposed algorithm is scalable and substantially reduces service delay, migration cost and travel time as compared with the existing baselines. [ABSTRACT FROM AUTHOR]

For vehicle-to-network communications, handover (HO) management enables vehicles to maintain the connection with the network while transiting through coverage areas of different base stations (BSs). However, the high mobility of vehicles means shorter connection periods with each BS that leads to frequent HOs, hence raises the necessity for optimal HO decision making for high quality infotainment services. Machine learning is capable of capturing underlying pattern via data driven methods to find optimal solutions to complex problems, and much learning-based HO optimization research has been conducted focusing on specific network setups. However, attention still needs to be paid to the actual deployment aspect and standardized datasets or simulation environments for evaluation. This paper proposes a deep reinforcement learning-based HO algorithm using the input parameters that are configurable in the existing measurement report of cellular networks. The performance of the proposed algorithm is evaluated using the well-known network simulator ns-3 with its official LTE module. A realistic network setup in the city center of Glasgow (U.K.) is configured with vehicle trajectories generated by the routes mobility model using Google Maps Directions API. Evaluation results reveal that the proposed algorithm significantly outperforms the A3 RSRP baseline with an average of 25.72% packet loss reduction per HO, suggesting significant improvement in quality of service of phone call and video streaming, etc. The proposed algorithm also has a small implementation cost compared to some state-of-the-art and should be deployed by a software update to a local BS controller. [ABSTRACT FROM AUTHOR]

The renewable energy source (RES)-powered small cell base station (SBS) is a promising technology for the next-generation networks because RESs provide sustainable energy without being depleted. In RES-assisted networks, joint optimization of user association (UA) and power control (PC) is required to enhance the sum-rate performance by reducing inter-cell interference between SBSs. This paper tackles the UA and PC optimization for the sum-rate maximization under quality-of-service (QoS) and backhaul constraints. We formulate the problem as mixed-integer non-linear programming, in which UA and PC variables of different time slots are tightly coupled due to the RES energy dynamics model. Hence, designing a dynamic policy-based UA and PC with consideration of the future environments such as the quantity of channel gain and energy harvesting remains a challenge. First, we propose a deep unsupervised learning (DUL)-based UA scheme for a fixed PC variable. To lower the computational complexity and accelerate the convergence of the proposed learning-based optimization, we relax the UA variable, which originally has an extremely high dimension, into a low-dimensional continuous variable inspired by the Lagrangian method. Next, we propose a deep reinforcement learning (DRL)-based PC scheme, in which the stringent QoS and backhaul constraints are considered penalty terms on the reward design. The proposed DRL-based PC scheme facilitates dynamic PC inferring the relationship between the future environment and current PC. Simulation results demonstrate that the proposed scheme enhances the sum-rate by 10%, accommodates 3.3 percent point (%p) more QoS-qualified users, and reduces the computation time by 20 times compared to the conventional optimization-based method. [ABSTRACT FROM AUTHOR]

In this paper, we investigate an unmanned aerial vehicle (UAV) communication system, where the trajectories of multi-UAVs are designed for the data collection mission of IoT nodes. We aim at minimizing the mission time with constraints of UAV’s maximum speed and acceleration, the collision avoidance, and communication interference among UAVs. We propose a three-step approach to solve this problem, which is based on the K-means algorithm, and Deep Reinforcement Learning (DRL) with a distributed manner and a centralized manner. The mutual influences like collision avoidance and interference among UAVs are explicitly expressed in our algorithm. Numerical results show the advantage of our proposed approach. [ABSTRACT FROM AUTHOR]

In this paper, we develop centralized and distributed dynamic resource allocation schemes for an intelligent reflecting surface (IRS) aided energy harvesting (EH) system that optimize transmit power of a source node and phase-shift of passive reflecting elements of IRS. The source node randomly harvests renewable energy from the surrounding environment and performs data transmission with the harvested energy while satisfying statistical packet delay constraints in terms of maximum acceptable delay-outage probability. Our developed schemes do not require the statistical distributions of channel and energy profiles to be known and apply deep reinforcement learning (DRL) algorithm. Simulation results demonstrate the effectiveness of the proposed resource control scheme for IRS-aided EH system in different channel and energy conditions. [ABSTRACT FROM AUTHOR]

Buffer-aware video streaming, which exploits the available storage space in user device to store the prefetched video data in good channels for video data use in poor channels, has been proved to have the potential to reduce the impact of fluctuating wireless channels on user-perceived video performance. However, in practical wireless networks, due to the unknown channel state and video rate, providing buffer-aware video streaming service to wireless user is a challenging problem. In this paper, with the aim to design an autonomous wireless video streaming system, we apply the deep reinforcement learning approach to dynamic resource optimization for wireless buffer-aware video streaming under unknown channel state and video rate. Specifically, we define a reward function for buffer-aware video streaming as the effective video streaming time when neither video-playback overflow nor video-playback underflow occurs. We propose a Markov decision process based problem formulation of the joint bandwidth allocation and buffer management for maximizing the effective video streaming time of all users. The optimal bandwidth allocation and buffer management policy is learned from training a deep neural network based on a deep reinforcement learning algorithm. We simulate the proposed algorithm in Tensorflow. Simulation results verify that the proposed deep reinforcement learning approach is effective for buffer-aware video streaming in wireless networks. [ABSTRACT FROM AUTHOR]

In this paper, we study the outage probability minimizing problem in a two-hop cooperative relay network. To reduce outage probability, existing studies propose many schemes for relay selection and power allocation, which are usually based on the assumption of exact channel state information (CSI). However, it is difficult to obtain perfect instantaneous CSI in practical situations where channel states change rapidly, and thus traditional methods would not perform well. Considering these factors, we turn to the emerging reinforcement learning (RL) methods for solutions. RL methods do not need any prior knowledge of CSI, but use neural network for approximation and decision after interacting with communication environment. Nevertheless, conventional RL methods, including most deep reinforcement learning (DRL) methods, cannot perform well when the search space is too large. In addition, non-stationarity is a common problem when using hierarchical reinforcement learning (HRL), which is caused by the changing behavior in different hierarchies. Therefore, we first propose a DRL framework with an outage-based reward function, which is then used as a baseline. Then, we further design an HRL framework and training algorithm. By decomposing relay selection and power allocation into two hierarchical optimization objectives, and combining on- policy and off-policy methods in the HRL framework, our method successfully address the sparse reward and non-stationary problem. Simulation results reveal that compared with traditional DRL method, the proposed HRL training algorithm can converge faster and reduce the outage probability by 8% in two-hop relay network with the same outage threshold. [ABSTRACT FROM AUTHOR]

Currently, self-driving, emerging as a key automatic application, has brought a huge potential for the provision of in-vehicle services (e.g., automatic path planning) to mitigate urban traffic congestion and enhance travel safety. To provide high-quality vehicular services with stringent delay constraints, edge computing (EC) enables resource-hungry self-driving vehicles (SDVs) to offload computation-intensive tasks to the edge servers (ESs). In addition, caching highly reusable contents decreases the redundant transmission time and improves the quality of services (QoS) of SDVs, which is envisioned as a supplement to the computation offloading. However, the high mobility and time-varying requests of SDVs make it challenging to provide reliable offloading decisions while guaranteeing the resource utilization of content caching. To this end, in this paper we propose a collaborative computation offloading and content caching method, named CoPace, by leveraging deep reinforcement learning (DRL) in EC for self-driving system. Specifically, we first introduce OSTP to predict the future time-varying content popularity, taking into account the temporal-spatial attributes of requests. Moreover, a DRL-based algorithm is developed to jointly optimize the offloading and caching decisions, as well as the resource allocation (i.e., computing and communication resources) strategies. Extensive experiments with real-world datasets in Shanghai, China, are conducted to evaluate the performance, which demonstrates that CoPace is both effective and well-performed. [ABSTRACT FROM AUTHOR]

In the multi-server multi-access edge computing (MEC) system, the computing capacity could be insufficient and the computing resource could be over-utilized when an excessive number of computing tasks are offloaded for execution. To alleviate the excessive burden in the multi-server MEC system and tap the underutilized resources of wired edge devices, which are called edge computers in this paper, we investigate the computation offloading and shunting scheme in the wireless wireline internetwork. We formulate an optimization problem to minimize the average total time delay and use the deep reinforcement learning method to obtain the optimal shunting policy. By sensing the state of the wireless wireline internetwork, the proposed scheme can adaptively adjust the shunting ratio and utilize the resources of MEC servers and edge computers intelligently and jointly. Finally, we evaluate the computational complexity and validate the convergence property of this scheme. The extensive simulation results demonstrate that this scheme can improve the resource productivity and reduce the average total time delay. [ABSTRACT FROM AUTHOR]

Advanced algorithms can promote the development of energy management strategies (EMSs) as a key technology in hybrid electric vehicles (HEVs). Reinforcement learning (RL) with distributed structure can significantly improve training efficiency in complex environments, and multi-threaded parallel computing provides a reliable algorithm basis for promoting adaptability. Dedicated to trying more efficient deep reinforcement learning (DRL) algorithms, this paper proposed a deep q-network (DQN)-based energy and emission management strategy (E&EMS) at first. Then, two distributed DRL algorithms, namely asynchronous advantage actor-critic (A3C) and distributed proximal policy optimization (DPPO), were adopted to propose EMSs, respectively. Finally, emission optimization was taken into account and then distributed DRL-based E&EMSs were proposed. Regarding dynamic programming (DP) as the optimal benchmark, simulation results show that three DRL-based control strategies can achieve near-optimal fuel economy and outstanding computational efficiency, and compared with DQN, two distributed DRL algorithms have improved the learning efficiency by four times. [ABSTRACT FROM AUTHOR]

Network slicing is a key paradigm in 5G and is expected to be inherited in future 6G networks for the concurrent provisioning of diverse quality of service (QoS). Unfortunately, effective slicing of Radio Access Networks (RAN) is still challenging due to time-varying network situations. This paper proposes a new intelligent RAN slicing strategy with two-layered control granularity, which aims at maximizing both the long-term QoS of services and spectrum efficiency (SE) of slices. The proposed method consists of an upper-level controller to ensure the QoS performance, which enforces loose control by performing adaptive slice configuration according to the long-term dynamics of service traffic. The lower-level controller is to improve SE of slices, by tightly scheduling radio resources to users at the small time-scale. To realize the proposed RAN slicing strategy, we propose a model-free deep reinforcement learning (DRL) framework, which is a hierarchical structure that collaboratively integrating the modified deep deterministic policy gradient (DDPG) and double deep-Q-network algorithm. Specifically, the lower-level control problem is a mixed-integer stochastic optimization problem with multiple constraints. This kind of problem is hard to be directly solved by the exiting DRL algorithms, since it involves searching for the solution in a vast set of mixed-integer action space, which will induce unbearable computational complexity. Thus, we propose a novel action space reducing approach, embedding the convex optimization tools into the DDPG algorithm, to speed up the lower-level control. Furthermore, simulation results confirm the effectiveness of our proposed intelligent RAN slicing scheme. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the adaptive video delivery for multiple users over time-varying and mutually interfering multi-cell wireless networks. The key research challenge is to jointly design the physical-layer resource allocation scheme and application-layer rate adaptation logic, such that the users’ long-term fair quality of experience (QoE) can be maximized. Due to the timescale mismatch between these two layers and the asynchrony of user requests, however, it is difficult to directly model the cross-layer stochastic control problem by using a reinforcement learning framework. To address this difficulty, we propose a novel two-level decision framework where an optimization-based beamforming scheme (performed at the base stations) and a deep reinforcement learning (DRL)-based rate adaptation scheme (performed at the user terminals) are, respectively, developed, such that a highly complex long-term multi-user QoE fairness problem is decomposed into some relatively simple problems and solved effectively. Our strategy represents a significant departure from the existing schemes with consideration of either a short-term multi-user QoE maximization or a long-term single-user point-to-point QoE maximization. Extensive simulations demonstrate that the proposed cross-layer design is effective and promising. [ABSTRACT FROM AUTHOR]

Network slicing (NS) has been widely identified as a key architectural technology for 5G-and-beyond systems by supporting divergent requirements in a sustainable way. In radio access network (RAN) slicing, due to the device-base station (BS)-NS three layer association relationship, device association (including access control and handoff management) becomes an essential yet challenging issue. With the increasing concerns on stringent data security and device privacy, exploiting local resources to solve device association problem while enforcing data security and device privacy becomes attractive. Fortunately, recently emerging federated learning (FL), a distributed learning paradigm with data protection, provides an effective tool to address this type of issues in mobile networks. In this paper, we propose an efficient device association scheme for RAN slicing by exploiting a hybrid FL reinforcement learning (HDRL) framework, with the aim to improve network throughput while reducing handoff cost. In our proposed framework, individual smart devices train a local machine learning model based on local data and then send the model features to the serving BS/encrypted party for aggregation, so as to efficiently reduce bandwidth consumption for learning while enforcing data privacy. Specifically, we use deep reinforcement learning to train the local model on smart devices under a hybrid FL framework, where horizontal FL is employed for parameter aggregation on BS, while vertical FL is employed for NS/BS pair selection aggregation on the encrypted party. Numerical results show that the proposed HDRL scheme can achieve significant performance gain in terms of network throughput and communication efficiency in comparison with some state-of-the-art solutions. [ABSTRACT FROM AUTHOR]

This paper studies multi-agent deep reinforcement learning (MADRL) based resource allocation methods for multi-cell wireless powered communication networks (WPCNs) where multiple hybrid access points (H-APs) wirelessly charge energy-limited users to collect data from them. We design a distributed reinforcement learning strategy where H-APs individually determine time and power allocation variables. Unlike traditional centralized optimization algorithms which require global information collected at a central unit, the proposed MADRL technique models an H-AP as an agent producing its action based only on its own locally observable states. Numerical results verify that the proposed approach can achieve comparable performance of the centralized algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we study the handover (HO), and power allocation problem in a two-tier heterogeneous network (HetNet), which consists of a macro base station, and some millimeter-wave (mmWave) small base stations. We establish an HO management, and power allocation scheme to maximize the overall throughput while reducing the HO frequency. In particular, considering the interrelationship among decisions made by different user equipments (UEs), we first model the HO, and power allocation problem as a fully cooperative multi-agent task, in which all agents, i.e., UEs, have the same target. Then, to solve the multi-agent task, and get decentralized policies for each UE, we develop a multi-agent reinforcement learning (MARL) algorithm based on the proximal policy optimization (PPO) method, by introducing the centralized training with decentralized execution framework. That is, we use global information to train policies for each UE, and after the training is finished, each UE obtains a decentralized policy, which can be implemented only based on each UE's local observation. Specially, we introduce the counterfactual baseline to address the credit assignment problem in centralized learning. Due to the centralized training, the decentralized polices learned by multi-agent PPO (MAPPO) can work more cooperatively. Finally, the simulation results demonstrate that our method can achieve better performance comparing with other existing works. [ABSTRACT FROM AUTHOR]

Resource allocation in ultra dense network (UDN) is an multi-objective optimization problem since it has to consider the tradeoff among spectrum efficiency (SE), energy efficiency (EE) and fairness. The existing methods can not effectively solve this NP-hard nonconvex problem, especially in the presence of limited channel state information (CSI). In this paper, we investigate a novel model-driven deep reinforcement learning assisted resource allocation method. We first design a novel deep neural network (DNN)-based optimization framework consisting of a series of Alternating Direction Method of Multipliers (ADMM) iterative procedures, which makes the CSI as the learned weights. Then a novel channel information absent Q-learning resource allocation (CIAQ) algorithm is proposed to train the DNN-based optimization framework without massive labeling data, where the SE, the EE, and the fairness can be jointly optimized by adjusting discount factor. Our simulation results show that, the proposed CIAQ with rapid convergence speed not only well characterizes the extent of optimization objective with partial CSI, but also significantly outperforms the current random initialization method of neural network and the other existing resource allocation algorithms in term of the tradeoff among the SE, EE and fairness. [ABSTRACT FROM AUTHOR]

The smart vehicles construct Internet of Vehicle (IoV), which can execute various intelligent services. Although the computation capability of a vehicle is limited, multi-type of edge computing nodes provide heterogeneous resources for intelligent vehicular services. When offloading the complex service to the vehicular edge computing node, the decision for its destination should be considered according to numerous factors. This paper mostly formulate the offloading decision as a resource scheduling problem with single or multiple objective function and constraints, where some customized heuristics algorithms are explored. However, offloading multiple data dependence tasks in a complex service is a difficult decision, as an optimal solution must understand the resource requirement, the access network, the user mobility, and importantly the data dependence. Inspired by recent advances in machine learning, we propose a knowledge driven (KD) service offloading decision framework for IoV, which provides the optimal policy directly from the environment. We formulate the offloading decision for the multiple tasks as a long-term planning problem, and explore the recent deep reinforcement learning to obtain the optimal solution. It can scruple the future data dependence of the following tasks when making decision for a current task from the learned offloading knowledge. Moreover, the framework supports the pre-training at the powerful edge computing node and continually online learning when the vehicular service is executed, so that it can adapt the environment changes and can learn policy that are sensible in foresight. The simulation results show that KD service offloading decision converges quickly, adapts to different conditions, and outperforms a greedy offloading decision algorithm. [ABSTRACT FROM AUTHOR]

With the increasing penetration of renewable energy (RE), the operations of distribution network are threatened and some issues may appear, i.e., large voltage deviation, deterioration of statistic voltage stability, high power loss, etc. In turn, RE accommodation would be significantly impacted. Therefore, we propose a many-objective distribution network reconfiguration (MDNR) model, with the consideration of RE curtailment, voltage deviation, power loss, statistic voltage stability, and generation cost. This aims to assess the trade-off among these objectives for better operations of distribution networks. As the proposed model is a non-convex, non-linear, many-objective optimization problem, it is difficult to be solved. We further propose a deep reinforcement learning (DRL) assisted multi-objective bacterial foraging optimization (DRL-MBFO) algorithm. This algorithm combines the advantages of DRL and MBFO, and is targeted to find the Pareto front of proposed MDNR model with better searching efficiency. Finally, we conduct case study on the modified IEEE 33-bus, 69-bus, and 118-bus power distribution systems, and results verify the effectiveness of the MDNR model and outperformance of the proposed DRL-MBFO. [ABSTRACT FROM AUTHOR]

Content caching is an effective technique to alleviate the burden on backhaul links and reduce the traffic in cellular networks. In converged networks, broadcasting networks can push non-real-time popular services to different types of terminals during off-peak hours, while cellular networks are deployed to meet personalized needs. However, the storage capacity of terminals are usually limited. In this context, we study on the converged networks to push and cache the popular services in the router nodes close to the terminators. In this scheme, the most popular services are transmitted by the broadcasting base station and cached in the router nodes in a distributed cache network. Then, users can access the cached services in a more energy efficient manner. Due to the limited storage capacity of the router node, we assume that the user can access the cached services within two hops. Then, we formulate the service scheduling problem as a Markov Decision Process, aiming to maximize equivalent throughput (ET). Due to the large state space involved in the distributed cache network, it is quite challenging to obtain a tractable solution by the classical optimization algorithms. To handle this problem, a deep reinforcement learning based framework is proposed to tackle this problem. The simulation results show that the proposed algorithms are very effective; and they outperform the conventional one in term of the ET, especially when the users in the network subject to Poisson point process distribution. [ABSTRACT FROM AUTHOR]

We propose a learning-based scheme to investigate the dynamic multi-channel access (DMCA) problem in the fifth generation (5G) and beyond networks with fast time-varying channels wherein the channel parameters are unknown. The proposed learning-based scheme can maintain near-optimal performance for a long time, even in the sharp changing channels. This scheme greatly reduces processing delay, and effectively alleviates the error due to decision lag, which is cased by the non-immediacy of the information acquisition and processing. We first propose a psychology-based personalized quality of service model after introducing the network model with unknown channel parameters and the streaming model. Then, two access criteria are presented for the living streaming model and the buffered streaming model. Their corresponding optimization problems are also formulated. The optimization problems are solved by learning-based DMCA scheme, which combines the recurrent neural network with deep reinforcement learning. In the learning-based DMCA scheme, the agent mainly invokes the proposed prediction-based deep deterministic policy gradient algorithm as the learning algorithm. As a novel technical paradigm, our scheme has strong universality, since it can be easily extended to solve other problems in wireless communications. The real channel data-based simulation results validate that the performance of the learning-based scheme approaches that derived from the exhaustive search when making a decision at each time-slot, and is superior to the exhaustive search method when making a decision at every few time-slots. [ABSTRACT FROM AUTHOR]

Coarse-grained reconfigurable architectures (CGRAs) have drawn increasing attention due to their flexibility and energy efficiency. Data flow graphs (DFGs) are often mapped onto CGRAs for acceleration. The problem of DFG mapping is challenging due to the diverse structures from DFGs and constrained hardware from CGRAs. Consequently, it is difficult to find a valid and high quality solution simultaneously. Inspired from the great progress in deep reinforcement learning (RL) for AI problems, we consider building methods that learn to map DFGs onto spatially programmed CGRAs directly from experiences. We propose RLMap, a solution that formulates DFG mapping on CGRA as an agent in RL, which unifies placement, routing and processing element insertion by interchange actions of the agent. Experimental results show that RLMap performs comparably to state-of-the-art heuristics in mapping quality, adapts to different architecture, and converges quickly. [ABSTRACT FROM AUTHOR]

Although design automation is a key enabler of modern large-scale digital systems, automating the transistor-level circuit design process still remains a challenge. Some recent works suggest that deep learning algorithms could be adopted to find optimal transistor dimensions in relatively small circuitry such as analog amplifiers. However, those approaches are not capable of exploring different circuit structures to meet the given design constraints. In this work, we propose an automatic circuit design framework that can generate practical circuit structures from scratch as well as optimize the size of each transistor, considering performance and reliability. We employ the framework to design level shifter circuits, and the experimental results show that the framework produces novel level shifter circuit topologies and the automatically optimized designs achieve $2.8\times $ – $5.3\times $ lower power-delay product (PDP) than prior arts designed by human experts. [ABSTRACT FROM AUTHOR]

The field of meta-learning, or learning-to-learn, has seen a dramatic rise in interest in recent years. Contrary to conventional approaches to AI where tasks are solved from scratch using a fixed learning algorithm, meta-learning aims to improve the learning algorithm itself, given the experience of multiple learning episodes. This paradigm provides an opportunity to tackle many conventional challenges of deep learning, including data and computation bottlenecks, as well as generalization. This survey describes the contemporary meta-learning landscape. We first discuss definitions of meta-learning and position it with respect to related fields, such as transfer learning and hyperparameter optimization. We then propose a new taxonomy that provides a more comprehensive breakdown of the space of meta-learning methods today. We survey promising applications and successes of meta-learning such as few-shot learning and reinforcement learning. Finally, we discuss outstanding challenges and promising areas for future research. [ABSTRACT FROM AUTHOR]

We present a method for empirically modeling and optimizing variations in sputtering deposition processes using Gaussian Process (GP) machine learning methods. Our predictive models can be trained with limited training data to enable rapid sputtering process tuning. As a first case, we model the effect of process recipe parameters such as chamber pressure and power on sputtered film thickness uniformity. A second more challenging case is also demonstrated: modeling film thickness spatial uniformity as a function of equipment configuration parameters. The effects of the chamber configuration variables are complex, motivating incorporation of prior process knowledge into the GP framework by utilizing a physics-based solver. Because adjusting equipment configuration parameters and obtaining corresponding wafer fabrication data is costly, a key metric is the expected number of tunes required until process constraints are met. Using past experimental data, we show that tunes using the GP-based predictive model are expected to converge in significantly fewer iterations compared to tunes using polynomial, gradient boosted regression tree, multivariate spline, and deep learning based modeling methods. [ABSTRACT FROM AUTHOR]

Packet routing is one of the fundamental problems in computer networks in which a router determines the next-hop of each packet in the queue to get it as quickly as possible to its destination. Reinforcement learning (RL) has been introduced to design autonomous packet routing policies with local information of stochastic packet arrival and service. However, the curse of dimensionality of RL prohibits the more comprehensive representation of dynamic network states, thus limiting its potential benefit. In this article, we propose a novel packet routing framework based on multiagent deep RL (DRL) in which each router possess an independent long short term memory (LSTM) recurrent neural network (RNN) for training and decision making in a fully distributed environment. The LSTM RNN extracts routing features from rich information regarding backlogged packets and past actions, and effectively approximates the value function of Q-learning. We further allow each route to communicate periodically with direct neighbors so that a broader view of network state can be incorporated. The experimental results manifest that our multiagent DRL policy can strike the delicate balance between congestion-aware and shortest routes, and significantly reduce the packet delivery time in general network topologies compared with its counterparts. [ABSTRACT FROM AUTHOR]

The data center market grows rapidly with the increase of data and its corresponding applications (e.g., machine learning, cloud storage, Internet of Things, and so on). The growth is boosted recently due to the shift of activities online during the COVID-19 pandemic. Reducing the energy consumption of data centers faces various challenges that are further aggravated by the tropical conditions with high temperature and humidity in the tropics like Singapore. The prevailing siloed approach of operating the information technology (IT) and the facility systems separately has resulted in wasteful over-provisioning. The recently proposed approaches for energy usage minimization under various constraints including thermal safety scale poorly with the data center size and often result in non-optimal solutions. To advance the state of the art, we apply deep reinforcement learning (DRL) to address the scalability problem and achieve optimality over a long time horizon in reducing data center energy usage. In particular, we deploy the data-driven deep model and physical rule based model in lieu of the physical data center during the training and validation phases to manage the thermal safety risks caused by DRL's strategy of learning from errors. [ABSTRACT FROM AUTHOR]

Due to the dynamic nature of connectivity in terrestrial vehicular networks, it is of great benefit to deploy unmanned aerial vehicles (UAVs) in these networks to act as relays. As a result, a remarkable number of studies have exploited UAVs to bridge the communication gaps between terrestrial vehicles, and sometimes despite their unoptimized mobility, their restricted communication coverage, and their limited energy resources. However, it was noted that for an intermittently connected vehicular network, UAVs could not cover all sparse areas all the time. Even worse, when deploying enough UAVs to cover all these areas, the probability of inter-UAV collisions increases, and it will be complex to control their movements efficiently. Consequently, it is required to dispatch an organized and intelligent group of UAVs to perform communication relays in the long term while keeping their connectivity, minimizing their average energy consumption, and providing an efficient coverage strategy. To meet these requirements, we propose a deep reinforcement learning (DRL) framework, called DISCOUNT (Dispatch of UAVs for Urban VANETs). Extensive simulations have been conducted to evaluate the performance of the proposed framework. It has been shown that the proposed framework significantly outperforms two commonly-used baseline techniques and some reinforcement learning methods in terms of energy consumption, coverage, and routing performances. [ABSTRACT FROM AUTHOR]

The interconnection of vehicles in the future fifth generation (5G) wireless ecosystem forms the so-called Internet of Vehicles (IoV). IoV offers new kinds of applications requiring delay-sensitive, compute-intensive, and bandwidth-hungry services. Mobile edge computing (MEC) and network slicing are two of the key enabler technologies in 5G networks that can be used to optimize the allocation of the network resources and guarantee the diverse requirements of IoV applications. As traditional model-based optimization techniques generally end up with NP-hard and strongly non-convex and nonlinear mathematical programming formulations, in this article, we introduce a model-free approach based on deep reinforcement learning (DRL) to solve the resource allocation problem in MEC-enabled IoV networks based on network slicing. Furthermore, the solution uses non-orthogonal multiple access (NOMA) to enable better exploitation of the scarce channel resources. The considered problem addresses jointly the channel and power allocation, the slice selection, and the vehicle selection (vehicle grouping). We model the problem as a single-agent Markov decision process. Then we solve it using DRL with the well-known deep Q learning (DQL) algorithm. We show that our approach is robust and effective under different network conditions compared to benchmark solutions. [ABSTRACT FROM AUTHOR]

We investigate the benefits of integrating unmanned aerial vehicles (UAVs) with reconfigurable intelligent surface (RIS) elements to passively relay information sampled by Internet of Things devices (IoTDs) to the base station (BS). In order to maintain the freshness of relayed information, an optimization problem with the objective of minimizing the expected sum Age-of-Information (AoI) is formulated to optimize the altitude of the UAV, the communication schedule, and phases-shift of RIS elements. In the absence of prior knowledge of the activation pattern of the IoTDs, proximal policy optimization algorithm is developed to solve this mixed-integer non-convex optimization problem. Numerical results show that our proposed algorithm outperforms all others in terms of AoI. [ABSTRACT FROM AUTHOR]

Vehicular fog computing (VFC) has been expected as a promising scheme that can increase the computational capability of vehicles without relying on servers. Comparing with accessing the remote cloud, VFC is suitable for delay-sensitive tasks because of its low-latency vehicle-to-vehicle (V2V) transmission. However, due to the dynamic vehicular environment, how to motivate vehicles to share their idle computing resource while simultaneously evaluating the service availability of vehicles in terms of vehicle mobility and vehicular computational capability in heterogeneous vehicular networks is a main challenge. Meanwhile, tasks with different priorities of a vehicle should be processed with different efficiencies. In this work, we propose a task offloading scheme in the context of VFC, where vehicles are incentivized to share their idle computing resource by dynamic pricing, which comprehensively considers the mobility of vehicles, the task priority, and the service availability of vehicles. Given that the policy of task offloading depends on the state of the dynamic vehicular environment, we formulate the task offloading problem as a Markov decision process (MDP) aiming at maximizing the mean latency-aware utility of tasks in a period. To solve this problem, we develop a soft actor-critic (SAC) based deep reinforcement learning (DRL) algorithm for the sake of maximizing both the expected reward and the entropy of policy. Finally, extensive simulation results validate the effectiveness and superiority of our proposed scheme benchmarked with traditional algorithms. [ABSTRACT FROM AUTHOR]

Wireless network optimization has been becoming very challenging as the problem size and complexity increase tremendously, due to close couplings among network entities with heterogeneous service and resource requirements. By continuously interacting with the environment, DRL provides a mechanism for different network entities to build knowledge and make autonomous decisions to improve network performance. In this article, we first review typical DRL approaches and recent enhancements. We then discuss the applications of DRL for mobile edge computing (MEC), which can be used for user devices to offload computation workload to MEC servers. However, for the low-power user devices, for example, wireless sensors, MEC can be costly as data offloading also consumes high power in RF communications. To balance the energy consumption in local computation and data offloading, we propose a novel hybrid offloading model that exploits the complementary operations of active RF communications and low-power backscatter communications. To maximize the energy efficiency in MEC offloading, the DRL framework is customized to learn the optimal transmission scheduling and workload allocation in two communications technologies. Numerical results show that the hybrid offloading scheme can improve the energy efficiency over 20 percent compared to existing schemes. [ABSTRACT FROM AUTHOR]

For a battery swapping station (BSS), the stochastic operation of electric buses (EBs) and the uncertainty of electricity prices cause unnecessary economic losses. To minimize the operating costs of the BSS, this article applies the deep reinforcement learning (DRL) and proposes a BSS model to determine the optimal real-time charge/discharge power of the charging piles. The predictability of bus operation and the uncertainty of price can be directly captured from historical data without any assumption in the model. Moreover, deep deterministic policy gradient (DDPG), as the DRL algorithm, is implemented in the model to simultaneously control multiple charging piles. Numerical results illustrate that the proposed approach can bring less operating cost than the existing benchmark control methods for a BSS while providing adequate batteries ready for swapping. [ABSTRACT FROM AUTHOR]

The effective pricing of electric vehicle (EV) charging by aggregators constitutes a key problem toward the realization of the significant EV flexibility potential in deregulated electricity systems and has been addressed by previous work through bi-level optimization formulations. However, the solution approach adopted in previous work cannot capture the discrete nature of the EV charging/discharging levels. Although reinforcement learning (RL) can tackle this challenge, state-of-the-art RL methods require discretization of state and/or action spaces and thus exhibit limitations in terms of solution optimality and computational requirements. This article proposes a novel deep reinforcement learning (DRL) method to solve the examined EV pricing problem, combining deep deterministic policy gradient (DDPG) principles with a prioritized experience replay (PER) strategy and setting up the problem in multi-dimensional continuous state and action spaces. Case studies demonstrate that the proposed method outperforms state-of-the-art RL methods in terms of both solution optimality and computational requirements and comprehensively analyze the economic impacts of smart-charging and vehicle-to-grid (V2G) flexibility on both aggregators and EV owners. [ABSTRACT FROM AUTHOR]

Resource allocation and transceivers in wireless networks are usually designed by solving optimization problems subject to specific constraints, which can be formulated as variable or functional optimization. If the objective and constraint functions of a variable optimization problem can be derived, standard numerical algorithms can be applied for finding the optimal solution, which however incurs high computational cost when the dimension of the variable is high. To reduce the on-line computational complexity, learning the optimal solution as a function of the environment's status by deep neural networks (DNNs) is an effective approach. DNNs can be trained under the supervision of optimal solutions, which however, is not applicable to the scenarios without models or for functional optimization where the optimal solutions are hard to obtain. If the objective and constraint functions are unavailable, reinforcement learning can be applied to find the solution of a functional optimization problem, which is however not tailored to optimization problems in wireless networks. In this article, we introduce unsupervised and reinforced-unsupervised learning frameworks for solving both variable and functional optimization problems without the supervision of the optimal solutions. When the mathematical model of the environment is completely known and the distribution of the environment's status is known or unknown, we can invoke an unsupervised learning algorithm. When the mathematical model of the environment is incomplete, we introduce reinforced- unsupervised learning algorithms that learn the model by interacting with the environment. Our simulation results confirm the applicability of these learning frameworks by taking a user association problem as an example. [ABSTRACT FROM AUTHOR]