Search

With the rapid growth of urban populations,the number of private cars has grown exponentially,which makes overwhelming traffic congestion problem become more and more acute.The traditional traffic signal control technology is difficult to adapt to the complex and changeable traffic conditions,and the data-driven methods bring new research directions for the control-based system.The combination of deep reinforcement learning and traffic control systems plays an important role in adaptive traffic signal control.First,this paper reviews the latest progress in the application of intelligent traffic signal control systems,the methods of intelligent traffic signal control are classified and discussed,and the existing works in this field are summarized.The deep reinforcement learning method can effectively solve the problems of inaccurate state information acquisition,poor algorithm robust and weak regional coordination control ability in intelligent traffic signal control.Then,on the basis of the above,this paper gives an overview of the simulation platforms and experimental setup for intelligent traffic signal control,and analyzes and verifies it through examples.Finally,The challenges and unsolved problems in this field are discussed and future research directions are summarized.

Real-time collision avoidance in dynamic environments is a challenge in trajectory planning of mobile robots. Focusing on environments with variable number of obstacles,this paper proposes a real-time trajectory planning algorithm,Crit-LSTM-DRL,based on long short-term memory(LSTM) and deep reinforcement learning(DRL). First,it predicts the time to the occurrence of a collision between an obstacle and the robot based on their states,and then computes the collision criticality of each obstacle with respect to the robot. Second,it generates the obstacle sequence based on the collision criticality and abstracts a fixed-dimension vector by LSTM to represent the environment. Finally,the robot state and the extracted vector are concatenated as the input of the DRL's value network to compute the value with respect to the system state. At any instant,for each action,it predicts the value of the next state based on the LSTM and DRL models and then the value of the current state; hence,the action generating the maximal value of the current state is selected to control the robot. To evaluate the performance of Crit-LSTM-DRL,it is first trained in three different environments and obtain three models: the model trained in the environment with 5 obstacles,the model trained in the environment with 10 obstacles,and the model trained in the environment with variable number of obstacles(1～10). The models then are tested in various environments containing different number of obstacles. To further investigate the effects of the interaction between an obstacle and the robot,this paper also takes the joint state of an obstacle and the robot as the state of the obstacle and trains another three models in the above training environments. Experimental results show the effectiveness and efficiency of Crit-LSTM-DRL.

Typically, the strategy of central heating for buildings in winter is climate compensator.However, this strategy heavily relies on manual experience with a relatively simple regulation.Therefore, how to optimize the heating control strategy is very important to keep the indoor temperature stable and comfortable.For this task, this paper proposes a heating strategy optimization method based on deep learning and deep reinforcement learning, which can optimize the original control strategy based on real historical data.The paper first develops a deep MTDN (Multiple Time Difference Network) as the simulator to predict the next time slot's room temperature.By learning the thermodynamic law of indoor temperature change, the network has high accuracy and confirms the physical laws.After that, the SAC (Soft Actor-Critic) algorithm based on maximum entropy reinforcement learning is employed as the strategy optimizer to interact with the simulator.Here, we use the evaluation index of the human body's thermal response as the reward to train and optimize the heating control strategy.Based on the real data of a heat exchange station in Tianjin, we evaluate the predictive ability of the simulator and the control ability of the strategy optimizer, respectively.The results verify that, compared with other types of prediction simulators, this simulator not only has high prediction accuracy but also conforms to physical laws.At the same time, compared with the original strategy, the strategy learned by the strategy optimizer can ensure that the indoor temperature is more stable and comfortable in multiple time periods of random sampling.

Recommender systems are devoted to find and automatically recommend valuable information and services for users from massive data,which can effectively solve the information overload problem,and become an important information technology in the era of big data.However,the problems of data sparsity,cold start,and interpretability are still the key technical difficulties that limit the wide application of the recommender systems.Reinforcement learning is an interactive learning technique,which can dynamically model user preferences by interacting with users and obtaining feedback to capture their interest drift in real time,and can better solve the classical key issues faced by traditional recommender systems.Nowadays,reinforcement lear-ning has become a hot research topic in the field of recommendation systems.From the perspective of survey,this paper first analyzes the improvement ideas of reinforcement learning for recommender systems based on a brief review of recommender systems and reinforcement learning.Then,the paper makes a general overview and summary of reinforcement learning based recommender systems in recent years,and further summarizes the research situation of traditional reinforcement learning based recommendation and deep reinforcement learning based recommendation respectively.Furthermore,the paper summarizes the frontiers of reinforcement learning based recommendation research topic in recent years and its application.Finally,the future development trend and application of reinforcement learning in recommender systems are analyzed.

With the development of artificial intelligence,the field of autonomous driving is also growing.The deep reinforcement learning (DRL) method is one of the main research methods in this field.DRL algorithms have been reported to achieve excellent performance in many control tasks.However,the unconstrained exploration in the learning process of DRL usually restricts its application to automatic driving.For example,in common reinforcement learning (RL) algorithms,an agent often has to select an action to execute in each state although this action may result in a crash,deteriorating the performance,or even failing the task.To solve the problem,this paper proposes a new method of action constrained with the soft actor-critic algorithm (CSAC) where the 'NO-OP'(NO-Option) identifies and replaces inappropriate actions,and we test the algorithm in the lane-keeping tasks.The method firstly limits the environmental reward reasonably.When the rotation angle of the driverless car is too large,it will shake,then a penalty term will be added to the reward function to avoid the driverless car falling into a dangerous state as far as possible.The contributions of this paper are as follows:first,we incorporates action constrained function with SAC algorithm,which achieves faster learning speed and higher stability;second,we propose a reward setting framework that overcomes the shaking and instability of driverless cars,achieving a better performance;finally,we trains the model in the unity virtual environment for evaluating the performance and successfully transplant the model to a donkey driverless car.

With the wide application of intelligent unmanned vehicles,intelligent navigation,path planning and obstacle avoidance technology have become important research contents.This paper proposes model-free deep reinforcement learning algorithms DDPG and SAC,which use environmental information to track to the target point,avoid static and dynamic obstacles,and can be generally suitable for different environments.Through the combination of global planning and local obstacle avoidance,it solves the path planning problem with better globality and robustness,solves the obstacle avoidance problem with better dynamicity and generalization,and shortens the iteration time.In the network training stage,PID,A* and other traditional algorithms are combined to improve the convergence speed and stability of the method.Finally,a variety of experimental scenarios such as navigation and obstacle avoidance are designed in the robot operating system ROS and the simulation program gazebo.Simulation results verify the reliability of the proposed approach,which takes the global and dynamic nature of the problem into account and optimizes the generated paths and time efficiency.

Deep reinforcement learning(DRL) has attracted much attention in multi-user intelligent dynamic spectrum access due to its advantages in sensing and decision making.However,the weak interpretability of deep neural networks(DNNs) makes DRL models vulnerable to backdoor attacks.In this paper,a non-invasive backdoor attack method with low-cost is proposed against DSA-oriented DRL models in cognitive wireless networks.The attacker monitors the wireless channels to select backdoor triggers,and generates backdoor samples into the experience pool of a secondary user's DRL model.Then,the trigger can be implanted into the DRL model during the training phase.The attacker actively sends signals to activate the triggers in the DRL model during the inference phase,inducing secondary users to take the actions set by the attacker,thereby reducing their success rate of channel access.A series of simulation show that the proposed backdoor attack method can reduce the attack cost by 20%~30% while achieving an attack success rate over 90%,and is suitable for three different DRL models.

Mobile Edge Computing(MEC) is used to enhance data processing in low power networks,and it has become an efficient computing paradigm.This paper considers an edge-cloud collaborative system composed of multiple MTs and adopts a variety of offloading modes.In order to reduce the total time delay of MTs,a task offloading algorithm based on deep reinforcement learning is proposed.It implements deep neural network(DNN) as a scalable solution,learns the multi-base offloading mode from experience to minimize the total time delay.Simulation results indicate that compared with the deep Q network(DQN) algorithm and the deep deterministic policy gradient(DDPG) algorithm,the proposed algorithm can improve the maximum performance gain significantly.In addition,the proposed algorithm has good convergence,and its result can approach the optimal result obtained by exhaustive search.

With the popularization of smart mobile devices,a new generation of mobile applications such as face recognition and virtual reality have gradually emerged.The limited computing power and battery capacity of mobile devices cannot support applications with high computing requirements and latency-sensitive applications.Therefore,mobile edge computing(MEC) is proposed to solve this problem.However,in the MEC environment,the reliability of the edge server is low,and the possible equipment failure will lead to the existing offloading decision failure,which increases the application response time and reduces the user experience.In view of the possible failure of edge servers,and considering that the deep deterministic policy gradient(DDPG) algorithm can better deal with the problem of high-dimensional action space through the network fitting strategy function,this paper proposes a server-reliability task offloading based on deep deterministic policy gradient(SRTO-DDPG).The main work is as follows.Firstly,the failure rate of application execution is reduced by duplicating subtasks for secondary offload.Secondly,the task offloading and resource allocation problems with server reliability constraints to minimize application delay are modeled as Markov decision process(MDP).Finally,an algorithm based on DDPG is used to solve the problem.Simulation results show that the SRTO-DDPG strategy can effectively interact with the environment to obtain the optimal offloading decision,and its perfor-mance is better than the local execution strategy(LE).Compared with the single location task offloading based on deep determi-nistic policy gradient(SLTO-DDPG),this strategy can achieve a low total delay of about 26.16% under reliability constraints,and can better adapt to the reliability problems of edge servers in multi-server scenarios.

In recent years,the application of deep reinforcement learning in recommendation system has attracted much attention.Based on the existing research,this paper proposes a new recommendation model RP-Dueling,which is based on the deep reinforcement learning Dueling-DQN algorithm,and adds the regret exploration mechanism to make the algorithm adaptively and dynamically adjust the proportion of “exploration-utilization” according to the training degree.The algorithm can capture users’ dynamic interest and fully explore the action space in the recommendation system with large-scale state space.By testing the proposed algorithm model on multiple data sets,the optimal average results of MAE and RMSE are 0.16 and 0.43 respectively,which are 0.48 and 0.56 higher than the current optimal research results.Experimental results show that the proposed model is superior to the existing traditional recommendation model and recommendation model based on deep reinforcement learning.

In order to compensate the performance loss caused by obstacle blocking in mobile edge computing (MEC) system in 6G-enabled “intelligent Internet of Things”,this paper proposes a partial task offloading scheme supported by aerial reconfigurable intelligent surface (RIS).Firstly,we investigate the joint design of the RIS phase shift vector,the proportion of offloading task,time slot allocation,the transmit power of users and the position of UAV,formulating a non-convex problem for minimization of the total energy consumption of users.Then,the original non-convex problem is decomposed into four subproblems,and the proximal policy optimization (PPO) method in deep reinforcement learning (DRL) is utilized to provide time slot allocation.The alternative optimization (AO) is leveraged to decouple the original problem into four subproblems,including the RIS phase shift design,the convex optimization of transmit power and offloading task amount,and the UAV altitude optimization.Simulation results show that the proposed PPO model can be trained quickly,the total energy consumption of users can be reduced by about 23% and 5.3%,compared with the fully-offload strategy and fixed-UAV-height strategy,respectively.

Using deep learning technology to control vehicles at intersections is a research hotspot in the field of intelligent transportation.Previous studies suffer from the inability to adapt to dynamic changes in the number of self-driving vehicles,slow convergence of training,and locally optimal training results.This work focuses on how autonomous vehicles can use distributed deep reinforcement methods to improve the efficiency of intersections at unsignalized intersections.First,an efficient reward function is proposed to apply the distributed reinforcement learning algorithm to the unsignalized intersection scenario,which can effectively improve the efficiency of intersection passage by relying on only local information even if the vehicle cannot obtain the whole intersection state information.Then,to address the problem of inefficient training of reinforcement learning methods in open intersection scenarios,a transfer learning approach is used to improve the training efficiency by using the trained strategy in the closed figure-of-eight scenario as a warm start and continuing the training in the unsignalized intersection scenario.Finally,this paper proposes a strategy that can be adapted to all proportions of autonomous vehicles,and this strategy can improve intersection access efficiency in scenarios with any proportion of autonomous vehicles.The algorithm is validated on the simulation platform Flow,and the experimental results show that the proposed smart body model converges quickly in training,can adapt to dynamic changes in the proportion of self-driving vehicles,and can effectively improve the efficiency of intersections.

The mutual promotion of mobile communication technology and mobile communication industry has achieved unprecedented prosperity in the mobile Internet era.The explosion of mobile devices,expansion of the network scale,improvement of service requirements are driving the next technological revolution in wireless networks.5G meets the requirements for the thousand-fold improvement of service performance through intensive network deployment,but co-channel interference and bursty request problems make the energy consumption of this solution very huge.In order to support 5G network to provide energy-efficient and high-performance services,it is imperative to upgrade and improve the management scheme of mobile networks.In this article,we use a short-cycle management framework with cache queues to achieve agile and smooth management of request burst scenarios to avoid dramatic fluctuations in service quality due to request bursts.We use deep reinforcement learning to learn the user distribution and communication needs,and infer the load change rules of the base station,and then realize the pre-scheduling and pre-allocation of energy,while ensuring the quality of service and improving the energy efficiency.Compared with the classic DQN algorithm,the two-buffer DQN algorithm proposed in this paper can provide nearly 20% acceleration in convergence.In terms of decision performance,it can save 4.8% energy consumption compared to the currently widely used keep on strategy.

In this paper,a flexible video multicast mechanism assisted by the UAV base station is proposed.In combination with SVC encoding,the dynamic deployment and resource allocation of UAV are considered jointly in order to maximize the overall number of enhancement layers received by users.The traditional heuristic algorithm is difficult to deal with the complexity of user movement,considering that the user movement within the range of macro station will change the network topology.To this end,the DDPG algorithm based on deep reinforcement learning is used to train the neural network to decide the optimal location and bandwidth allocation proportion of UAV.After the model converges,the learning agent can find the optimal UAV deployment and bandwidth allocation strategy in a short time.The simulation results show that the proposed scheme achieves the expected goal and is superior to the existing scheme based on Q-learning.