Search

The explosive growth of Internet-of-Things (IoT) applications such as smart cities and Industry 4.0 have led to drastic increase in demand for wireless bandwidth, hence motivating the rapid development of new techniques for enhancing spectrum utilization needed by new generation wireless communication technologies. Among others, dynamic spectrum access (DSA) is one of the most widely accepted approaches. In this paper, as an enhancement of existing works, we take into consideration of inter-node collaborations in a dynamic spectrum environment. Typically, in such distributed circumstances, intelligent dynamic spectrum access almost invariably relies on self-learning to achieve dynamic spectrum access improvement. Whereas, this paper proposes a DSA scheme based on deep reinforcement learning to enhance spectrum and access efficiency. Unlike traditional Q-learning-based DSA, we introduce the following to enhance the spectrum efficiency in dynamic IoT spectrum environments. First, deep double Q-learning is adopted to perform local self-spectrum-learning for IoT terminals in order to achieve better dynamic access accuracy. Second, to accelerate learning convergence, federated learning (FL) in edge nodes is used to improve the self-learning. Third, multiple secondary users, who do not interfere with each other and have similar operation condition, are clustered for federated learning to enhance the efficiency of deep reinforcement learning. Comparing with the traditional distributed DSA with deep learning, the proposed scheme has faster access convergence speed due to the characteristic of global optimization for federated learning. Based on this, a framework of federated deep reinforcement learning (FDRL) for DSA is proposed. Furthermore, this scheme preserves privacy of IoT users in that FDRL only requires model parameters to be uploaded to edge servers. Simulations are performed to show the effectiveness of theproposed FDRL-based DSA framework. [ABSTRACT FROM AUTHOR]

Co-salient object detection (CoSOD) aims to detect common salient objects sharing the same attributes in an image group. The key issue of CoSOD is how to model the inter-saliency relations within an image group. The major limitation of previous methods is that they pre-define the group-to-one relations within an image group. In this paper, we propose a new concept of structural inter-saliency relations and solve the CoSOD with deep reinforcement learning framework. Firstly, we design a semantic relation graph (SRG) to model the structural inter-saliency relations. Then the feature selecting agent (FS-agent) aims to select the informative features, which can help the SRG effectively model structural inter-saliency relations. Finally, relation updating agent (RU-agent) progressively updates the SRG to focus on the co-salient relations like human decision-making process. Extensive experiments on co-saliency datasets show that because of well modeling inter-saliency relations in image group, our proposed method achieves superior performance compared to the state-of-the-art methods. We hope that this paper can motivate future research for visual co-analysis tasks. [ABSTRACT FROM AUTHOR]

As more and more people choose high-speed rail (HSR) as a means of transportation for short trips, there is ever growing demand of high quality of multimedia services. With its rich spectrum resources, millimeter wave (mm-wave) communications can satisfy the high network capacity requirements for HSR. Also, it is possible for receivers (RXs) to be equipped with antenna arrays in mm-wave communication systems due to its short wavelength. However, as HSRs run with high speed, the received signal power (RSP) varies rapidly over a cell and it is the lowest at the edge of the cell compared to other locations. Consequently, it is necessary to conduct research on RX beamforming for HSR in mm-wave band to improve the quality of the received signal. In this paper, we focus on RX beamforming for a mm-wave train-ground communication system. To improve the RSP, we propose an effective RX beamforming scheme based on deep reinforcement learning (DRL), and develop a deep Q-network (DQN) algorithm to train and determine the optimal RX beam direction with the purpose of maximizing average RSP. Through extensive simulations, we demonstrate that the proposed scheme has better performance than the four baseline schemes in terms of average RSP at most positions on the railway. [ABSTRACT FROM AUTHOR]

Cooperative Intelligent Transportation Systems (C-ITS) will change the modes of road safety and traffic management, especially at intersections without traffic lights, namely unsignalized intersections. Existing researches focus on vehicle control within a small area around an unsignalized intersection. This paper expands the control domain to a large area with multiple intersections. In particular, a Multi-intersection Vehicular Cooperative Control (MiVeCC) is proposed to enable cooperation among vehicles in a large area with multiple unsignalized intersections. Firstly, a vehicular end-edge-cloud computing framework is proposed to facilitate end-edge-cloud vertical cooperation and horizontal cooperation among vehicles. Under the framework, a two-stage reinforcement learning is implemented to obtain the optimal policy for vehicle control. To support RL in the proposed framework, a multi-vehicle state representation method and a safety-oriented value representation method are designed. The former structures the collected vehicle information, and the latter evaluates the traffic condition. A multi-intersection simulation platform is developed to evaluate the proposed scheme. Simulation results show that the proposed MiVeCC can improve travel efficiency at multiple intersections by up to 4.59 times without collision compared with benchmark methods. [ABSTRACT FROM AUTHOR]

The ongoing growth of Internet of Things (IoT) traffic increasesthe emergence of Multi-Access Edge Computing (MEC), which can serve IoT devices to process edge missions properly. The power supply for IoT devices is addressed by energy harvesting (EH) technology. However, the energy arrival model obeys the unknown energy arrival process. In addition, due to the dynamical changes of network state, it is crucial to study the access control in the MEC-based green IoT system. In this paper, we propose an MEC-based access control strategy to maximize the system utility for the green IoT system, considering the spectral efficiency and the successful access ratio from the perspective of operators and users respectively. Specifically, we model the access control problem as a Constrained Markov Decision Process (CMDP), and develop a novel centralized reward-based experience replay deep convolutional Q network algorithm (RCQN) to attain the optimal access control for EH IoT devices. Additionally, to enrich the prior knowledge of access control strategy, we design a Long Short-Term Memory (LSTM) based energy prediction module to predict the energy state of IoT devices by learning the historical energy arrival information, whose output is taken as the state information of RCQN. Simulation results demonstrate that the proposed LSTM-based RCQN access control algorithm significantly improves the system utility with a proper reward design and training mechanism. [ABSTRACT FROM AUTHOR]

This paper applies machine learning to optimize the transmission policies of cognitive radio inspired non-orthogonal multiple access (CR-NOMA) networks, where time-division multiple access (TDMA) is used to serve multiple primary users and an energy-constrained secondary user is admitted to the primary users’ time slots via NOMA. During each time slot, the secondary user performs the two tasks: data transmission and energy harvesting based on the signals received from the primary users. The goal of the paper is to maximize the secondary user’s long-term throughput, by optimizing its transmit power and the time-sharing coefficient for its two tasks. The long-term throughput maximization problem is challenging due to the need for making decisions that yield long-term gains but might result in short-term losses. For example, when in a given time slot, a primary user with large channel gains transmits, intuition suggests that the secondary user should not carry out data transmission due to the strong interference from the primary user but perform energy harvesting only, which results in zero data rate for this time slot but yields potential long-term benefits. In this paper, a deep reinforcement learning (DRL) approach is applied to emulate this intuition, where the deep deterministic policy gradient (DDPG) algorithm is employed together with convex optimization. Our simulation results demonstrate that the proposed DRL assisted NOMA transmission scheme can yield significant performance gains over two benchmark schemes. [ABSTRACT FROM AUTHOR]

This paper studies the allocation of shared resources between vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) links in vehicle-to-everything (V2X) communications. In existing algorithms, dynamic vehicular environments and quantization of continuous power become the bottlenecks for providing an effective and timely resource allocation policy. In this paper, we develop two algorithms to deal with these difficulties. First, we propose a deep reinforcement learning (DRL)-based resource allocation algorithm to improve the performance of both V2I and V2V links. Specifically, the algorithm uses deep Q-network (DQN) to solve the sub-band assignment and deep deterministic policy-gradient (DDPG) to solve the continuous power allocation problem. Second, we propose a meta-based DRL algorithm to enhance the fast adaptability of the resource allocation policy in the dynamic environment. Numerical results demonstrate that the proposed DRL-based algorithm can significantly improve the performance compared to the DQN-based algorithm that quantizes continuous power. In addition, the proposed meta-based DRL algorithm can achieve the required fast adaptation in the new environment with limited experiences. [ABSTRACT FROM AUTHOR]

As power systems are undergoing a significant transformation with more uncertainties, less inertia and closer to operation limits, there is increasing risk of large outages. Thus, there is an imperative need to enhance grid emergency control to maintain system reliability and security. Towards this end, great progress has been made in developing deep reinforcement learning (DRL) based grid control solutions in recent years. However, existing DRL-based solutions have two main limitations: 1) they cannot handle well with a wide range of grid operation conditions, system parameters, and contingencies; 2) they generally lack the ability to fast adapt to new grid operation conditions, system parameters, and contingencies, limiting their applicability for real-world applications. In this paper, we mitigate these limitations by developing a novel deep meta-reinforcement learning (DMRL) algorithm. The DMRL combines the meta strategy optimization together with DRL, and trains policies modulated by a latent space that can quickly adapt to new scenarios. We test the developed DMRL algorithm on the IEEE 300-bus system. We demonstrate fast adaptation of the meta-trained DRL polices with latent variables to new operating conditions and scenarios using the proposed method, which achieves superior performance compared to the state-of-the-art DRL and model predictive control (MPC) methods. [ABSTRACT FROM AUTHOR]

This paper proposes a hybrid data-driven method for fast solutions of preventive security-constrained optimal power flow (SCOPF) of power systems. The proposed method formulates the SCOPF problem as constraints-satisfying training of a deep reinforcement learning (DRL) agent, where the action-value function of DRL is augmented by contingency security constraints. In the training process, the proposed method hybridizes the primal-dual deep deterministic policy gradient (PD-DDPG) and the classic SCOPF model. Instead of building reward critic networks and cost critic networks via interacting with the environment (i.e., power flow equations), the actor gradients are approximated by solving KKT conditions of the Lagrangian. Finally, with the formulated sparse Jacobians of constraints and sparse Hessians of Lagrangians, the interior point method is incorporated in PD-DDPG to derive the parameters updating rule of the DRL agent. Numerical tests are carried out on a modified IEEE 57-bus system and a modified IEEE 300-bus system for critical contingencies. The results show that the well-trained DRL agent can rapidly (real-time) obtain high-quality SCOPF solutions that satisfy the security constraints. [ABSTRACT FROM AUTHOR]

The Internet of vehicles (IoV) is a large information interaction network that collects information on vehicles, roads and pedestrians. One of the important uses of vehicle networks is to meet the entertainment needs of driving users through communication between vehicles and roadside units (RSUs). Due to the limited storage space of RSUs, determining the content cached in each RSU is a key challenge. With the development of 5G and video editing technology, short video systems have become increasingly popular. Current widely used cache update methods, such as partial file precaching and content popularity- and user interest-based determination, are inefficient for such systems. To solve this problem, this paper proposes a QoE-driven edge caching method for the IoV based on deep reinforcement learning. First, a class-based user interest model is established. Compared with the traditional file popularity- and user interest distribution-based cache update methods, the proposed method is more suitable for systems with a large number of small files. Second, a quality of experience (QoE)-driven RSU cache model is established based on the proposed class-based user interest model. Third, a deep reinforcement learning method is designed to address the QoE-driven RSU cache update issue effectively. The experimental results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

With the new feature of multi-energy coupling and the advancement of the energy market, Energy Internet (EI) has higher requirements for the efficiency and applicability of integrated energy response. This paper proposes an indirect multi-energy transaction (IMET) to promote multi-energy collaborative optimization in local energy market (LEM) and improve energy utilization through personalized responses from We-Energies (WEs). Firstly, an indirect customer-to-customer multi-energy transaction is modeled for local multi-energy coupling market which can satisfy privacy, preference and autonomy of users. The efficiency of energy matching can be promoted through the participation of conversion devices. In addition, multi-time scale hybrid trading mechanism is constructed with the consideration of the transmission speed of different energy sources. Meanwhile, energy transaction process is built as a Markov decision process (MDP) with deep reinforcement learning algorithm so that the system modeling error can be successfully avoided. Furthermore, a distributed training structure is utilized to obtain more experience for a wider range of scenarios. The results of numerical simulations demonstrate the performance of the proposed method. [ABSTRACT FROM AUTHOR]

This paper is concerned with the efficient movement control of multiple drone cells for communication coverage. Although many works have been developed to cope with this problem, but only few of them have investigated the three-dimensional (3-D) continuous movement control of multiple drone cells. In this paper, a problem of 3-D continuous movement control of multiple drone cells is formulated with an objective of maximizing the energy-efficient communication coverage of drone-cell networks while preserving the network connectivity. To mitigate this challenging problem, an energy-efficiency and continuous-movement-control algorithm (E $^2$ CMC), which is based on an emerging deep reinforcement learning method, is proposed. In E $^2$ CMC, an energy-efficiency reward function considering the energy consumption, the sum of quality-of-service (QoS) requirements of users as well as the coverage fairness is first designed. Next, E $^2$ CMC learns to adjust drone cells’ locations continuously by interacting with an environment in a sequence of observations, actions, and rewards. Furthermore, E $^2$ CMC will reduce the reward drastically as long as the networks are disconnected. Simulation results verify the superiority of the proposed learning algorithm on deploying multiple drone cells to provide the communication coverage. [ABSTRACT FROM AUTHOR]

As for fuel cell hybrid electric vehicle equipped with battery (BAT) and ultracapacitor (UC), its dynamic topology structure is complex and different characteristics of three power sources induce challenges in energy management for fuel economy, power sources lifespan, and dynamic performance of the vehicle. In this paper, an energy management strategy (EMS) based on a hierarchical power splitting structure and deep reinforcement learning (DRL) is proposed. In the higher layer strategy of the proposed EMS, the UC is employed to supply peak power and recover braking energy through the adaptive filter based on fuzzy control. Then, the integrated DRL and equivalent consumption minimization strategy framework is proposed to optimize the power allocation of fuel cell (FC) and BAT in the lower layer, to ensure the highly efficient operation of FC and reduce hydrogen consumption. And the action trimming based on heuristic technique is proposed to further restrain the adverse effect of sudden peak power on FC lifespan. The simulation results show the proposed EMS can make the output of FC smoother, improve its working efficiency to alleviate the stress of BAT, and increase by 14.8% compared with the Q-learning strategy in fuel economy under WLTP driving cycle. Meanwhile, the obtained results under UDDSHDV show fuel economy of the proposed EMS can reach dynamic programming (DP) benchmark level of 89.7 $\%$. [ABSTRACT FROM AUTHOR]

Computation offloading is an effective method in mobile edge computing (MEC) to relieve user equipment (UE) from the limited computation resource and battery capacity. Meanwhile, simultaneous wireless information and power transmission (SWIPT) can be applied to MEC to extend the operating time of the equipment. However, in multi-user network environment, diverse computation task requirements and changeable network channel states make it challenging to obtain offloading strategy timely and accurately. To address the issue, we propose an intelligent computation offloading scheme (iCOS) based on enhanced priority deep deterministic policy gradient (EPDDPG) algorithm to minimize the energy consumption of all the UEs by jointly optimizing the offloading decision, the central processing unit (CPU) frequency and the power split ratio in a dynamic SWIPT-MEC network. In particular, we improve the traditional fully-connected network structure to obtain both discrete and continuous action outputs, and accelerate neural network parameter updates by using prioritized experience tuples. Furthermore, we use dynamic voltage and frequency scaling (DVFS) technology to dynamically adjust the CPU frequency of local computing, and employ SWIPT technology to balance the charging and communication according to the obtained strategy. Simulation results show that the algorithm proposed in this paper can effectively reduce the energy cost of UEs, and complete more computation tasks within the delay limit. [ABSTRACT FROM AUTHOR]

In this paper, we consider an intelligent reflecting surface (IRS)-assisted cognitive radio system and maximize the secondary user (SU) rate by jointly optimizing the transmit power of secondary transmitter (ST) and the IRS’s reflect beamforming, subject to the constraints of the minimum required signal-to-interference-plus-noise ratio at the primary receiver, the ST’s maximum transmit power, and the unit modulus of the IRS reflect beamforming vector. This joint optimization problem can be solved suboptimally by the non-convex optimization techniques, which however usually require complicated mathematical transformations and are computationally intensive. To address this challenge, we propose an algorithm based on the deep deterministic policy gradient (DDPG) method. To achieve a higher learning efficiency and a lower reward variance, we propose another algorithm based on the soft actor-critic (SAC) method. In these proposed algorithms, a reward impact adjustment approach is proposed to improve their learning efficiency and stability. Simulation results show that the two proposed algorithms can achieve comparable SU rate performance with much shorter running time, as compared to the existing non-convex optimization-based benchmark algorithm, and that the proposed SAC-based algorithm learns faster and achieves a higher average reward with lower variance, as compared to the proposed DDPG-based algorithm. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (RL) has recently led to many breakthroughs on a range of complex control tasks. However, the agent’s decision-making process is generally not transparent. The lack of interpretability hinders the applicability of RL in safety-critical scenarios. While several methods have attempted to interpret vision-based RL, most come without detailed explanation for the agent’s behavior. In this paper, we propose a self-supervised interpretable framework, which can discover interpretable features to enable easy understanding of RL agents even for non-experts. Specifically, a self-supervised interpretable network (SSINet) is employed to produce fine-grained attention masks for highlighting task-relevant information, which constitutes most evidence for the agent’s decisions. We verify and evaluate our method on several Atari 2600 games as well as Duckietown, which is a challenging self-driving car simulator environment. The results show that our method renders empirical evidences about how the agent makes decisions and why the agent performs well or badly, especially when transferred to novel scenes. Overall, our method provides valuable insight into the internal decision-making process of vision-based RL. In addition, our method does not use any external labelled data, and thus demonstrates the possibility to learn high-quality mask through a self-supervised manner, which may shed light on new paradigms for label-free vision learning such as self-supervised segmentation and detection. [ABSTRACT FROM AUTHOR]

Traffic awareness is the prerequisite of autonomous driving. Given the limitation of on-board sensors (e.g., precision and price), remote measurement from either infrastructure or other vehicles can improve traffic safety. However, the wireless communication carrying the measurement result undergoes fading, noise and interference and has a certain probability of outage. When the communication fails, the vehicle state can only be predicted by Bayesian filtering with a low precision. Higher communication resource utilization (e.g., transmission power) reduces the outage probability and hence results in an improved estimation precision. The power control subject to an estimate variance constraint is a difficult problem due to the complicated mapping from transmit power to vehicle-state estimate variance. In this paper, we develop an estimator consisting of several Kalman filters (KFs) or extended Kalman filters (EKFs) and an interacting multiple model (IMM) to estimate and predict the vehicle state. We propose to apply deep reinforcement learning (DRL) for the transmit power optimization. In particular, we consider an intersection and a lane-changing scenario and apply proximal policy optimization (PPO) and soft actor-critic (SAC) to train the DRL model. Testing results show satisfactory power control strategies confining estimate variances below given threshold. SAC achieves higher performance compared to PPO. [ABSTRACT FROM AUTHOR]

With the rapid development of intelligent transportation, automated traffic surveillance is considered as an important component. In the field of traffic surveillance, it is particularly important to achieve robust and real-time tracking of vehicles in complex scenes. In this paper, a robust real-time vehicle tracking model named NoisyOTNet is proposed, which formulates tracking as reinforcement learning with parameter space noise. In this formulation, the exploration ability of the model is enhanced to improve the robustness of tracking. Specifically, we develop a new implementation for noisy network based on deep deterministic policy gradients (DDPGs) with parameter noise, which can better cope with the tracking task and directly predict the tracking result. To improve the tracking accuracy in complex conditions, e.g. fast motion and large deformation, this paper presents an adaptive update strategy that can exploit the vehicle spatial-temporal information based on Upper Confidence Bound (UCB) algorithm by exploiting. Moreover, as for the recovery of the lost target, a relocation algorithm based on incremental learning is developed. The results of extensive experiments demonstrate that the proposed NoisyOTNet can effectively track vehicles in complex scenes and achieve competitive performance compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Reconfigurable intelligent surface (RIS) technology is promising to enhance wireless communications services by providing smart radio environment. In this paper, we investigate the RIS aided cellular networks with device-to-device (D2D) users, and maximize the sum of the transmission rate of the D2D communications and the cellular networks from a new perspective. In addition to solving the typical resource allocation problems for D2D communications, this paper further optimize the wireless environment by adjusting the position and phase shift of the RIS. To solve this non-convex problem, we propose a novel decentralized double deep Q-network (D3QN) framework for the resource allocation at users and a centralized DDQN for RIS optimization at the base station (BS), which are verified to achieve the near-optimal performance with lower complexity and enhanced robustness. Simulation results illustrate that the proposed framework can achieve higher transmission rates compared to benchmarks, meanwhile meeting the quality of service (QoS) requirements at the BS and D2D users. [ABSTRACT FROM AUTHOR]

Many of the existing optimization-based transmit power control algorithms suffer from high computational complexity and require instantaneous global channel state information (CSI), both of which hinder their practical implementation. In this paper, we consider a wireless network with multiple transmitter-receiver pairs, where each transmitter only has access to its local CSI fed back from its intended receiver and does not require local CSI exchange with its neighboring transmitters. In such a network scenario, we propose a deep reinforcement learning based decentralized multi-agent power control (DEC-MAPC) algorithm for sum-rate maximization, where each transmitter acts as an intelligent agent. By leveraging the value decomposition technique, we establish a nonlinear mapping from the local reward of each agent to the global reward. Such a design allows each agent to independently control its transmit power based on its local CSI while enabling global collaboration among the agents. The proposed algorithm is scalable to large-scale networks as only local CSI is required, and is robust to the channel and interference variations via interacting with the environment. Simulation results show that the proposed DEC-MAPC algorithm with local CSI achieves comparable sum-rate performance with the centralized optimization algorithms with global CSI, while significantly reducing the computational complexity. [ABSTRACT FROM AUTHOR]

In this paper, we aim to propose a novel transmission control protocol (TCP) congestion control method from a cross-layer-based perspective and present a deep reinforcement learning (DRL)-driven method called DRL-3R (DRL for congestion control with Radio access network information and Reward Redistribution) so as to learn the TCP congestion control policy in a superior manner. In particular, we incorporate the RAN information to timely grasp the dynamics of RAN, and empower DRL to learn from the delayed RAN information feedback potentially induced by several consecutive actions. Meanwhile, we relax the implicit assumption (that the feedback to one specific action returns at a round-trip-time (RTT) after the action is applied) in previous researches, by redistributing the rewards and evaluating the merits of actions more accurately. Experiment results show that besides maintaining a reasonable fairness, DRL-3R significantly outperforms classical congestion control methods (e.g., TCP Reno, Westwood, Cubic, BBR and DRL-CC) on network utility by achieving a higher throughput while reducing delay in various network environments. [ABSTRACT FROM AUTHOR]

Knowledge transfer is a promising concept to achieve real-time decision-making for autonomous vehicles. This paper constructs a transfer deep reinforcement learning (RL) framework to transform the driving tasks in the intersection environments. The driving missions at the unsignalized intersection are cast into a left turn, right turn, and running straight for automated vehicles. The goal of the autonomous ego vehicle (AEV) is to drive through the intersection situation efficiently and safely. This objective promotes the studied vehicle to increase its speed and avoid crashing other vehicles. The decision-making policy learned from one driving task is transferred through three transfer rules in another driving mission and evaluated. Simulation results reveal that the decision-making strategies related to similar tasks are transferable and have a high success rate. It indicates that the presented control framework could reduce time consumption and realize online implementation. Therefore, the transfer RL concept is helpful for establishing the real-time decision-making policy for autonomous vehicles. [ABSTRACT FROM AUTHOR]

In this paper, we study the three-dimensional (3D) path planning problem for cellular-connected unmanned aerial vehicle (UAV) taking into account the impact of 3D antenna radiation patterns. The cellular-connected UAV has a mission to travel from an initial location to a destination. In this process, there is a trade-off between the flight time and the expected communication outages duration, which requires planning a suitable route to avoid the weak coverage region. However, as the existing cellular networks are primarily designed for ground coverage, the communication coverage in the sky tends to be intermittent and irregular due to the potential for severe interference and obstructions (such by buildings), which brings great challenges to 3D path planning. To address this challenge, we first construct a 3D coverage map that stores the expected outage probability over each locations. We then propose a multi-step dueling DDQN (multi-step D3QN) based algorithm to design the local optimal UAV path by leveraging the constructed coverage map. In this algorithm, the UAV acts as the agent to learn the appropriate action to complete the flight mission. Numerical results show the effectiveness of the proposed algorithm for connectivity-aware UAV path planning and the superiority of 3D path design over its 2D counterparts. [ABSTRACT FROM AUTHOR]

Due to the deregulation of power systems worldwide, bidding behavior simulation research has gained prominence. One crucial element in these studies is accurately defining and modelling the individual reward function (or objective function). Considering the ubiquitous information barriers between market participants and researchers, the common way is to develop reward functions based on theoretical assumptions, which will inevitably cause deviations from the real world. However, since market data have gradually become transparent in recent years, especially data regarding historical bidding behaviors, it is feasible to introduce data-driven methods to identify the individual reward functions that are hidden in raw bidding data. Thus, this paper proposes a data-driven bidding objective function identification framework with three procedures. First, the bidding decision processes of participants are formulated as a standard Markov decision process. Second, a deep inverse reinforcement learning method that is based on maximum entropy is introduced to identify individual reward functions, whose high-dimensional nonlinearity could be saved in multilayer perceptions (MLPs). Third, a deep Q-network method is customized to simulate the individual bidding behaviors based on the obtained MLP-based objective functions. The effectiveness and feasibility of the proposed framework and methods are tested based on real market data from the Australian electricity market. [ABSTRACT FROM AUTHOR]

In this paper, we propose a channel-wise interaction based binary convolutional neural networks (CI-BCNN) approach for efficient inference. Conventional binary convolutional neural networks usually apply the xnor and bitcount operations in the binary convolution with notable quantization errors, which obtain opposite signs of pixels in binary feature maps compared to their full-precision counterparts and lead to significant information loss. In our proposed CI-BCNN method, we exploit the channel-wise interactions with the prior knowledge which aims to alleviate inconsistency of signs in binary feature maps and preserves the information of input samples during inference. Specifically, we mine the channel-wise interactions by using a reinforcement learning model, and impose channel-wise priors on the intermediate feature maps to correct inconsistent signs through the interacted bitcount. Since CI-BCNN mines the channel-wise interactions in a large search space where each channel may correlate with others, the search deficiency caused by sparse interactions obstacles the agent to obtain the optimal policy. To address this, we further present a hierarchical channel-wise interaction based binary convolutional neural networks (HCI-BCNN) method to shrink the search space via hierarchical reinforcement learning. Moreover, we propose a denoising interacted bitcount operation in binary convolution by smoothing the channel-wise interactions, so that noise in channel-wise priors can be alleviated. Extensive experimental results on the CIFAR-10 and ImageNet datasets demonstrate the effectiveness of the proposed CI-BCNN and HCI-BCNN. [ABSTRACT FROM AUTHOR]

Resilience of a distribution system can be enhanced by efficient restoration of critical load following a major outage. Existing models include optimization approaches that consider available information without incorporating the inherent asynchrony of data arrival during execution of the restoration plan. Failure to consider the asynchronous nature of information arrival can lead to underutilization of critical resources. Moreover, analytical models become computationally inefficient for large scale systems. On the other hand, artificial intelligence (AI)-based tools have demonstrated efficient results for power system applications. In this paper, it is proposed a Reinforcement Learning (RL) model that learns how to efficiently restore a distribution system after a major outage. The proposed approach is based on a Monte Carlo Tree Search to expedite the training process. The proposed model strategy provides a robust decision-making tool for asynchronous and partial information scenarios. The results, validated with the IEEE 13-bus test feeder and IEEE 8500-node distribution test feeder, demonstrate the effectiveness and scalability of the proposed method. [ABSTRACT FROM AUTHOR]

To meet the execution requirements of delay-sensitive services in vehicular edge computing (VEC) networks, vehicular services need to be offloaded to edge computing nodes. For complex, large-scale services, the services need to be migrated if the services are not completed before the vehicles leave the coverage of edge computing nodes. Trust and resource matching between areas thus become major problems. This paper studies the decision model of vehicular service offloading and migration. First, software-defined network (SDN) technology is introduced into the traditional network architecture, and a two-layer distributed SDN-controlled VEC network architecture is designed, which is divided into a domain control layer and an area control layer. In this framework, we use the consortium blockchain as a carrier to share network topology information between SDN controllers to prevent information leakage. We then established a service offloading and migration optimization problem model to minimize service execution delay, reduce energy consumption and maximize the throughput of the blockchain system. We describe the problem model as a Markov Decision Process (MDP), introduce a deep reinforcement learning (DRL) algorithm named asynchronous advantage actor-critic (A3C) and design a dynamic service offloading and migration algorithm (DSOMA) based on A3C to solve the problem. Simulation results show that DSOMA can increase the throughput of the blockchain system, and DSOMA is superior to the deep Q-learning (DQN) algorithm and greedy offloading algorithm in reducing service execution delay and system energy consumption. [ABSTRACT FROM AUTHOR]

This paper proposes to use deep reinforcement learning for the simulation of physics-based musculoskeletal models of both healthy subjects and transfemoral prostheses’ users during normal level-ground walking. The deep reinforcement learning algorithm is based on the proximal policy optimization approach in combination with imitation learning to guarantee a natural walking gait while reducing the computational time of the training. Firstly, the optimization algorithm is implemented for the OpenSim model of a healthy subject and validated with experimental data from a public data-set. Afterwards, the optimization algorithm is implemented for the OpenSim model of a generic transfemoral prosthesis’ user, which has been obtained by reducing the number of muscles around the knee and ankle joints and, specifically, by keeping only the uniarticular ones. The model of the transfemoral prosthesis’ user shows a stable gait, with a forward dynamic comparable to the healthy subject’s, yet using higher muscles’ forces. Even though the computed muscles’ forces could not be directly used as control inputs for muscle-like linear actuators due to their pattern, this study paves the way for using deep reinforcement learning for the design of the control architecture of transfemoral prostheses. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the problem of wireless service provisioning through a rotary-wing UAV which can serve as an aerial base station (BS) to communicate with multiple ground terminals (GTs) in a boost demand area. Our objective is to optimize the UAV control for maximizing the UAV.s energy efficiency, in which both aerodynamic energy and communication energy are considered while ensuring the communication requirements for each GT and backhaul link between the UAV and the terrestrial BS. The mobility of the UAV and GTs lead to time-varying channel conditions that make the environment dynamic. We formulate a nonconvex optimization for controlling the UAV considering the practical angle-dependent Rician fading channels between the UAV and GTs, and between the UAV and the terrestrial BS. Traditional optimization approaches are not able to handle the dynamic environment and high complexity of the problem in real-time. We propose to use a deep reinforcement learning-based approach namely Deep Deterministic Policy Gradient (DDPG) to solve the formulated nonconvex problem of UAV control with continuous action space that takes into account the real-time of the environment including time-varying UAV-ground channel conditions, available onboard energy of the UAV, and the communication requirement of the GTs. However, the DDPG method may not achieve good performance in an unstable environment and will face a large number of hyperparameters. We extend our approach to use the Trust Region Policy Optimization (TRPO) method that can improve the performance of the UAV compared to the DDPG method in such a dynamic environment. [ABSTRACT FROM AUTHOR]

Dependency-aware jobs, such as the big data analytic workflows, are commonly executed on the cloud. They are compiled to directed acyclic graphs, with tasks linked in regarding the dependency. The cloud scheduler, which maintains a large number of resources, is responsible to execute tasks in parallel. To resolve the complex dependencies, Deep Reinforcement Learning (DRL) based schedulers are widely applied. However, we find that the DRL-based schedulers are vulnerable to the perturbations in the input jobs and may generate falsified decisions to benefit a particular job while delaying the others. By perturbation, we mean a slight adjustment to the job's node features or dependencies, while not changing its functionality. In this paper, we first explore the vulnerability of DRL-based schedulers to job perturbations without accessing the information of the DRL models used in the scheduler. We devise the black-box perturbation system, in which, a proxy model is trained to mimic the DRL-based scheduling policy. We show that the high-faith proxy model can help to craft effective perturbations. The DRL-based schedulers can be as high as 60% likely to be badly affected by the perturbations. Then, we investigate the solution to improve the robustness of DRL-based schedulers to such perturbations. We propose an adversarial training framework to force the neural model to adapt to the perturbation patterns during training so as to eliminate the potential damage during applications. Experiments show that the adversarial-trained scheduler is more robust, reducing the chance of being affected to 3-fold less and the potential bad effects halved. [ABSTRACT FROM AUTHOR]

Network function virtualization redefines a network service as a softwarized chain of virtual network functions (VNFs), which decouples the specific network service from dedicated devices and greatly reduces the hardware cost. The VNFs are generally deployed on common off-the-shelf servers and statistically share computational resources. Due to the inherent integer constraints, the corresponding VNF mapping and scheduling issue is a highly challenging task. In this paper, we consider the mapping and scheduling of VNFs for a given network service, for which a flexible job shop scheduling problem is formulated to optimize the max-min fairness while ensuring the delay requirements of different service chains. Specifically, we propose a deep reinforcement learning method based on offline proximal policy optimization, which dynamically determines the mapping and scheduling decision based on the state of unfinished service chains. The proposed algorithm is scalable to the number of service chains and can be enhanced by Monte Carlo tree search. Numerical results show that the proposed algorithm outperforms the traditional random forest and the greedy algorithms in terms of both service fairness and acceptance ratio. [ABSTRACT FROM AUTHOR]

The dramatic increasing of data and demands for computing capabilities may result in excessive use of resources in cloud data centers, which not only causes the raising of energy consumption, but also leads to the violation of Service Level Agreement (SLA). Dynamic consolidation of virtual machines (VMs) is proven to be an efficient way to tackle this issue. In this paper, we present an Adaptive Deep Reinforcement Learning (DRL)-based Virtual Machine Consolidation (ADVMC) framework for energy-efficient cloud data centers. ADVMC has two phases. In the first phase, Influence Coefficient is introduced to measure the impact of a VM on producing host overload, and a dynamic Influence Coefficient-based VM selection algorithm (ICVMS) is proposed to preferentially choose those VMs with the greatest impact for migration in order to remove the excessive workloads of the overloaded host quickly and accurately. In the second phase, a Prediction Aware DRL-based VM placement method (PADRL) is further proposed to automatically find suitable hosts for VMs to be migrated, in which a state prediction network is designed based on LSTM to provide DRL-based model more reasonable environment states so as to accelerate the convergence of DRL. Simulation experiments on the real-world workload provided by Google Cluster Trace have shown that our ADVMC approach can largely cut down system energy consumption and reduce SLA violation of users as compared to many other VM consolidation policies. [ABSTRACT FROM AUTHOR]

Satellite mobile communication networks can provide communication services in areas beyond the coverage of terrestrial cellular networks. Ground relay stations for satellite networks is an effective solution to improve the coverage quality and to ensure connectivity between regular ground terminals and the satellite network. Without power grid coverage, the ground relay station can only be powered by green energy generation devices. So the problem of energy-saving under sleep strategy is worthy of studying.This paper proposes an energy-saving sleep control method for ground relay stations through Double Deep Q Network (DDQN) algorithm. Based on the traditional rule-based method that is only set for residual energy, it also considers the impact of weather factors on solar and wind power generation on the energy efficiency of relay stations. The research results show that by considering multiple influencing factors at the same time, the method used in this paper can effectively save non-communication energy consumption at low loads, ensure the satellite communication needs of most users, and improve the operational energy efficiency of the relay station over a period of time.

Reinforcement learning (RL) has seen an uptick in research interest in recent years, with many papers published in a plethora of different fields, topics and applications. A lot of that can be attributed to the recent advancements in machine learning (ML) and deep learning (DL) as a whole, the power of deep neural networks and the incorporation of them into reinforcement learning algorithms and techniques. This marriage of deep neural networks and traditional reinforcement learning methods has enabled reinforcement learning algorithms to achieve incredible results in use-cases that were previously deemed to be too complex. One of these highly complex scenarios is the realm of flight. In the last few years the number of papers published in this field has increased dramatically, but there is a fairly big divide between two major categories (quadcopters vs. fixed-wing aircraft). There is published work focusing on reinforcement learning using quadcopters, also known as drones, and there is published work focusing on fixed-wing aircraft, the more traditional aircraft, commonly referred to as airplanes. This paper focuses on the latter and will give an overview over the current state of reinforcement learning in the field of fixed-wing aviation. It will look into available toolkits usable for research as well as give an overview of different fields and tasks within reinforcement learning for fixed-wing aircraft control tasks. It contributes by providing detailed information about the methodologies used in each paper, while pointing out each paper’s contributions as well. A total of 48 papers on fixed-wing RL and airplane control are discussed. With the main focus lying on papers published post 2015, after the publication of Mnih et al.’s impactful deep RL paper.

The power grids nowadays are facing increasing complexity and uncertainty due to the continuously growing penetration of renewable energy sources, such as photovoltaic (PV) and wind power, as well as the emerging uncertain and dynamic nature of demand-side factors such as electric vehicles, portable energy storage, etc. To effectively manage these challenges, artificial intelligence (AI) technologies, particularly model-free deep reinforcement learning (DRL), have risen as a powerful tool for power system control. This paper presents an in-depth review of the state-of-the-art applications of model-free DRL in power system control. The review is focused on various model-free DRL approaches utilized in addressing uncertainty caused by stochastic factors from renewable generations, demand-side dynamics, and power system contingencies. Specifically, it investigates how model-free DRL techniques are employed to facilitate decision-making in the frequency control, voltage control, and optimal power flow control in the grid. The benefits, challenges, and limitations of these technologies are revealed, shedding light on recent advancements in the field and showcasing the performance of these methods across diverse power system scenarios. By synthesizing the findings from extensive research, this paper also highlights the limitations, key future research directions, and recommendations for deploying model-free DRL in power system control.

This paper proposes a novel curriculum learning approach for collision avoidance using feedback from the deep reinforcement learning (DRL) training process. Previous research on DRL-based collision avoidance algorithms has encountered challenges such as long training times and difficulty in convergence due to sparse rewards. To address these issues, curriculum learning has been used to divide the target task into multiple subtasks for training. However, manual or random curriculum design often generates unnecessary subtasks that do not improve performance. Furthermore, a standardized curriculum design method for collision avoidance has not yet been presented. Therefore, this paper introduces a curriculum-based collision avoidance learning method that utilizes feedback during the training phase. The proposed method differs from traditional curriculum learning in that the subtask is not predetermined before training. Instead, the curriculum is modified during training based on feedback obtained from validation environments. If a robot demonstrates high collision avoidance performance in a validation environment, it is then validated in more challenging environments for rigorous evaluation. Conversely, if collision avoidance performance is low in the validation environment, the robot is trained in a new environment to overcome frequent collision situations. Simulations and practical experiments were conducted for the proposed method, which showed better performance compared to the non-curriculum method.

Many studies on the application of deep reinforcement learning (DRL) in the field of traffic signal control do not fully consider the influence of vehicles approaching the intersection on traffic flow. In this paper, the convolutional block attention module (CBAM) is incorporated on the basis of the Dueling Double Deep Q Network (D3QN) method to improve the sensitivity of the model to the traffic situation, which can help the model to focus more on the distribution and dynamics of vehicles near intersections. To further improve the model performance, this paper introduces the traffic light phase variable time interval based on the original D3QN method, which helps the model to take into account the traffic requirements in all directions of the intersection. In addition, Double Deep Q Network (Double DQN) and Dueling Deep Q Network (Dueling DQN) technologies are used to further improve the performance of the model. The simulation experiments using the urban traffic simulator SUMO show that the proposed method has significant advantages over D3QN, Maximum Pressure algorithm and Fixed Timing Strategy for key indicators such as mean vehicle delay time, mean queue length and average number of stops. This shows that the method proposed in this paper has great potential in practical traffic signal control applications.

To support dramatically increasing services from internet of thing (IoT) devices with the sporadic and fluctuated generation of short packet traffic, this paper investigates joint dynamic time division duplexing (TDD) and radio resource control (RRC) connection management in a single-cell massive IoT network. Specifically, under the grant-free transmission incurring packet collision, this study models the factors affecting the time resource utilization (TRU) and energy consumption of IoT devices as a comprehensive system utility and further formulates the problem as a decision-making process aiming for balancing the long-term average TRU and energy consumption. To address the formulated problem, based on the deep reinforcement learning framework, this paper designs an experience-driven joint dynamic TDD and RRC connection management scheme that intelligently i) determines the TDD configuration based on the most recent downlink (DL)/uplink (UL) traffic demands and ii) adjusts the RRC state of each IoT device to control the maximum number of transmitting IoT devices. Finally, trace-driven simulation results demonstrate that the proposed scheme outperforms existing benchmarks, such as Static TDD and Dynamic TDD, in terms of transmission success ratio difference (TSRD) (up to 89% reduction), time resource utilization (TRU) (up to $17\times $ increase), and energy consumption (up to 70% reduction) of IoT devices.

With the development of artificial intelligence, there have been many attempts to incorporate artificial intelligence into algorithmic trading. In particular, reinforcement learning, which aims to solve dynamic decision-making problems, is attracting attention because of its high utilization in algorithmic trading. In this paper, we will implement a simple Deep Reinforcement Learning (DRL) trading robot to check the performance of DRL. In addition, we tried to find out how much performance improvement can be achieved by comparing a robot that learned a single stock data with a robot that learned stock data, market index, and interest rate data. This paper aims to develop a stock investment robot using a Proximal Policy Optimization (PPO) reinforcement learning algorithm and analyze the performance of the robot. The first robot used only the stock data of APPL INC, a single stock, as input, and the second robot used stock data of APPL INC and the S&P 500 index together with US interest rate data as input data. Afterward, the stock investment performance of the two robots for APPL INC was comparatively analyzed using the test data.

This paper leverages the OpenSim physics-based simulation environment for the forward dynamic simulation of an osseointegrated transfemoral amputee musculoskeletal model, wearing a generic prosthesis. A deep reinforcement learning architecture, which combines the proximal policy optimization algorithm with imitation learning, is designed to enable the model to walk by using three different observation states. The first is a complete state that includes the agent’s kinematics, ground reaction forces, and muscle data; the second is a reduced state that only includes the kinematics and ground reaction forces; the third is an augmented state that combines the kinematics and ground reaction forces with a prediction of the muscle data generated by a fully-connected feed-forward neural network. The empirical results demonstrate that the model trained with the augmented observation state can achieve walking patterns with rewards and gait symmetry ratings comparable to those of the model trained with the complete observation state, while there are no symmetric walking patterns when using the reduced observation state. This paper shows the importance of including muscle data in a deep reinforcement learning architecture for the forward dynamic simulation of musculoskeletal models of transfemoral amputees.

The Internet of Vehicles (IoV) as a promising application of Internet of Things (IoT) has played a significant role in autonomous driving, by connecting intelligent vehicles. Autonomous driving needs to process the mass environmental sensing data in coordination with surrounding vehicles, and makes an accurate driving judgment accordingly. Since the vehicles always have limited computing resources, processing these data in parallel with efficient task scheduling is one of the most important topics. Most current work focuses on formulating special scenarios and service requirements as optimization problems. However, the complicated and dynamic environment of vehicular computing is hard to model, predict and control, making those previous methods unscalable and unable to reflect the real scenario. In this paper, a Multi-task Deep reinforcement learning approach for scalable parallel Task Scheduling (MDTS) is firstly devised. For avoiding the curse of dimensionality when coping with complex parallel computing environments and jobs with diverse properties, we extend the action selection in Deep Reinforcement Learning (DRL) to a multi-task decision, where the output branches of multi- task learning are fine-matched to parallel scheduling tasks. Child tasks of a job are accordingly assigned to distributed nodes without any human knowledge while the resource competition among parallel tasks is leveraged through shared neural network layers. Moreover, we design an appropriate reward function to optimize multiple metrics simultaneously, with emphasis on specific scenarios. Extensive experiments show that the MDTS significantly increases the overall reward compared with least- connection scheduling and particle swarm optimization algorithm from −16.71, −0.67 to 2.93, respectively. [ABSTRACT FROM AUTHOR]

The developments of connected vehicles are heavily influenced by information and communications technologies, which have fueled a plethora of innovations in various areas, including networking, caching, and computing. Nevertheless, these important enabling technologies have traditionally been studied separately in the existing works on vehicular networks. In this paper, we propose an integrated framework that can enable dynamic orchestration of networking, caching, and computing resources to improve the performance of next generation vehicular networks. We formulate the resource allocation strategy in this framework as a joint optimization problem, where the gains of not only networking but also caching and computing are taken into consideration in the proposed framework. The complexity of the system is very high when we jointly consider these three technologies. Therefore, we propose a novel deep reinforcement learning approach in this paper. Simulation results with different system parameters are presented to show the effectiveness of the proposed scheme. [ABSTRACT FROM PUBLISHER]

Both caching and interference alignment (IA) are promising techniques for next-generation wireless networks. Nevertheless, most of the existing works on cache-enabled IA wireless networks assume that the channel is invariant, which is unrealistic considering the time-varying nature of practical wireless environments. In this paper, we consider realistic time-varying channels. Specifically, the channel is formulated as a finite-state Markov channel (FSMC). The complexity of the system is very high when we consider realistic FSMC models. Therefore, in this paper, we propose a novel deep reinforcement learning approach, which is an advanced reinforcement learning algorithm that uses a deep $Q$ network to approximate the $Q$ value-action function. We use Google TensorFlow to implement deep reinforcement learning in this paper to obtain the optimal IA user selection policy in cache-enabled opportunistic IA wireless networks. Simulation results are presented to show that the performance of cache-enabled opportunistic IA networks in terms of the network's sum rate and energy efficiency can be significantly improved by using the proposed approach. [ABSTRACT FROM PUBLISHER]

Dynamic topology, fast-changing channels and the time sensitivity of safety-related services present challenges to the status quo of resource allocation for cellular-underlaying vehicle-to-vehicle (V2V) communications. In this paper, we investigate a novel federated multi-agent deep reinforcement learning (FedMARL) approach for the decentralized joint optimization of channel selection and power control for V2V communication. The approach takes advantage of both deep reinforcement learning (DRL) and federated learning (FL), satisfying the reliability and delay requirements of V2V communication while maximizing the transmit rates of cellular links. Specifically, we elaborately construct individual V2V agent implement by the dueling double deep Q-network (D3QN), and design the reward function to train V2V agents collaboratively. As a result, each agent individually optimizes channel selection and power level based on its local observations, including the instantaneous channel state information (CSI) of corresponding V2V link, the instantaneous co-channel interference from the cellular link, the previous channels selections of nearby V2V pairs, and the queue backlog at the V2V transmitter. Another important aspect is that we incorporate FL to alleviate the training instability problem induced by cooperative multi-agent environment. The local DRL models of different V2V agents are federated periodically, addressing the limitations of partial observability on the entire network status for individual agent, and accelerating the training process of multi-agent learning. Validated via simulations, the proposed FedMARL scheme shows superiority to the baselines in terms of the cellular sum-rate and the V2V packet delivery rate. [ABSTRACT FROM AUTHOR]

The anti-jamming communication of the heterogeneous Internet of Satellites (IoS) has drawn increasing attentions due to the smart jamming and high dynamics. This paper investigates a spatial anti-jamming scheme for IoS, with the aim of minimizing anti-jamming routing cost via Stackelberg game and reinforcement learning. Firstly, we formulate the routing anti-jamming problem as a hierarchical anti-jamming Stackelberg game. It has been proved that there is a Stackelberg equilibrium (SE) in the proposed game. Secondly, the spatial anti-jamming scheme for IoS consists of two stages: the available routing selection and the fast anti-jamming decision. To tackle the high dynamics caused by the unknown interrupts and the unexpected congestion, we propose a deep reinforcement learning based routing algorithm (DRLR) to obtain an available routing subset; Furthermore, to make a fast anti-jamming decision, we propose a fast response anti-jamming algorithm (FRA) based on the available routing subset. The user utilizes DRLR and FRA algorithms to empirically analyze the jammer's strategies and adaptively make an anti-jamming decision according to the dynamic and unknown jamming environment. Finally, the simulations have shown that the proposed algorithm has lower routing cost and better anti-jamming performance than existing approaches, and the anti-jamming policies converge to the SE. [ABSTRACT FROM AUTHOR]

Adaptive bitrate (ABR) streaming has been used in wireless networks to deal with the time-varying wireless channels. Traditionally, wireless video is fetched from remote Internet server. However, wireless video streaming from Internet server faces challenges such as congestion and long latency. ABR streaming and video transcoding at the radio access network (RAN) edge have shown the potential to overcome such problem and provided better video streaming service. In the paper, we consider joint computation and communication for ABR streaming based on mobile edge computing (MEC) under time-varying wireless channels. We propose a joint video transcoding and quality adaptation framework for ABR streaming by enabling RAN with computing capability. By modeling the wireless channel as a finite state Markov channel, we formulate the optimization problem as a stochastic optimization problem of joint computational resource assignment and video quality adaptation for maximizing the average reward, which is defined as the tradeoff between user perceived quality of experience and the cost of performing transcoding at edge server. By using deep reinforcement learning (DRL) algorithm, we develop an automatic algorithm to perform the computational resource assignment and video quality adaptation without any prior knowledge of channel statistics. Simulation results using Tensorflow show the effectiveness of the designed MEC-enabled ABR streaming system and DRL algorithm. [ABSTRACT FROM AUTHOR]

This paper presents a novel approach for designing a robotic orthosis controller considering physical human-robot interaction (pHRI). Computer simulation for this human-robot system can be advantageous in terms of time and cost due to the laborious nature of designing a robot controller that effectively assists humans with the appropriate magnitude and phase. Therefore, we propose a two-stage policy training framework based on deep reinforcement learning (deep RL) to design a robot controller using human-robot dynamic simulation. In Stage 1, the optimal policy of generating human gaits is obtained from deep RL-based imitation learning on a healthy subject model using the musculoskeletal simulation in OpenSim-RL. In Stage 2, human models in which the right soleus muscle is weakened to a certain severity are created by modifying the human model obtained from Stage 1. A robotic orthosis is then attached to the right ankle of these models. The orthosis policy that assists walking with optimal torque is then trained on these models. Here, the elastic foundation model is used to predict the pHRI in the coupling part between the human and robotic orthosis. Comparative analysis of kinematic and kinetic simulation results with the experimental data shows that the derived human musculoskeletal model imitates a human walking. It also shows that the robotic orthosis policy obtained from two-stage policy training can assist the weakened soleus muscle. The proposed approach was validated by applying the learned policy to ankle orthosis, conducting a gait experiment, and comparing it with the simulation results. [ABSTRACT FROM AUTHOR]

In the cooperative vehicle infrastructure system, the road side unit (RSU) equipped with a mobile edge computing (MEC) server and sensors could provide vehicle infrastructure cooperation services for vehicles, such as optimization and cooperative driving, enhanced visibility, and so on. In view of this, the MEC server needs to fuse the sensing information from sensors on the vehicles and RSU, respectively. In the case of bad channel conditions, uploading the raw sensing information from the vehicles results in high uplink transmission latency. To deal with it, the vehicles can process the information locally and just deliver the results to the RSU. However, due to the limited computing resources on the vehicles, the processing accuracy of the raw information on the vehicles is lower than that on the MEC server. Besides, processing locally leads to higher vehicle energy consumption. Thus, in this paper, we aim to jointly optimize execution latency, processing accuracy, and energy consumption of the MEC-based cooperative vehicle infrastructure system. Firstly, we design the terminal machine learning task model and the edge machine learning task model on the vehicle side and RSU side, respectively. Then, we formulate a long-term multi-objective optimization problem. Owing to the stochastic traffic and time-varying communication conditions, we reformulate it as a Markov decision process and propose a two-stage deep reinforcement learning-based offloading and resource allocation (TDORA) strategy to determine the task offloading and the transmit power of each vehicle. Simulation results demonstrate the efficacy of the proposed strategy. [ABSTRACT FROM AUTHOR]

The recently proposed dual active protocol stack (DAPS) handover (HO) is one of the mobility enhancements that can effectively reduce the handover interruption time (HIT) in 5G networks. By using a DAPS solution, users can be connected to both the source cell and target cell during the HO execution phase, and thereby 0 ms of HIT becomes possible. However, the DAPS HO procedure may fail in 5G networks due to the channel characteristics of millimeter-wave (mmWave) signals. Since mmWave links are vulnerable to blockages, the received signal quality may degrade suddenly, which gives rise to an abrupt outage before DAPS HO can be completed. In this paper, a novel learning-based DAPS HO technology named intelligent DAPS (I-DAPS) HO is proposed to avoid sudden radio link failure (RLF) while providing a high data rate. The proposed I-DAPS HO uses a double deep Q-network (DDQN) deep reinforcement learning (DRL) framework, where blockage predictions are made based on past received signal data such that RLFs can be actively avoided. The performance evaluation demonstrates that the proposed I-DAPS HO scheme can effectively avoid RLF and improve the throughput performance compared to advanced 5G HO schemes. [ABSTRACT FROM AUTHOR]

In recent years, the growing development of Connected Autonomous Vehicles (CAV), Intelligent Transport Systems (ITS), and 5G communication networks have led to the advent of Autonomous Intersection Management (AIM) systems. AIMs present a new paradigm for CAV control in future cities, taking control of CAVs in scenarios where cooperation is necessary and allowing safe and efficient traffic flows, eliminating traffic signals. So far, the development of AIM algorithms has been based on basic control algorithms, without the ability to adapt or keep learning new situations. To solve this, in this paper we present a new advanced AIM approach based on end-to-end Multi-Agent Deep Reinforcement Learning (MADRL) and trained using Curriculum through Self-Play, called advanced Reinforced AIM (adv.RAIM). adv.RAIM enables the control of CAVs at intersections in a collaborative way, autonomously learning complex real-life traffic dynamics. In addition, adv.RAIM provides a new way to build smarter AIMs capable of proactively controlling CAVs in other highly complex scenarios. Results show remarkable improvements when compared to traffic light control techniques (reducing travel time by 59% or reducing time lost due to congestion by 95%), as well as outperforming other recently proposed AIMs (reducing waiting time by 56%), highlighting the advantages of using MADRL. [ABSTRACT FROM AUTHOR]