Search

A stable output voltage of a boost converter is vital for the appropriate functioning of connected devices and loads in a DC microgrid. Variations in load demands and source uncertainties can damage equipment and disrupt operations. In this study, a modified twin-delayed deep deterministic policy gradient (TD3) algorithm is proposed to regulate the output voltage of a boost converter in a DC microgrid. TD3 optimizes PI controller gains, which ensure system stability by employing a non-negative, fully connected layer. To achieve optimal gains, multi-deep reinforcement learning agents are trained. The agents utilize the error signal to obtain the desired output voltage. Furthermore, a new reward function used in the TD3 algorithm is introduced. The proposed controller is tested under load variations and input voltage uncertainties. Simulation and experimental results demonstrate that TD3 outperforms PSO, GA, and the conventional PI. TD3 exhibits less steady-state error, reduced overshoots, fast response times, fast recovery times, and a small voltage deviation. These findings confirm TD3's superiority and its potential application in DC microgrid voltage control. It can be used by engineers and researchers to design DC microgrids. [ABSTRACT FROM AUTHOR]

The incorporation of intermittent and stochastic renewable energy into a microgrid creates frequent fluctuations, which provides new challenges in frequency control. This paper deals with the frequency control problem in the islanded AC microgrid (IACMG) via a model-free deep reinforcement learning (DRL) method, which includes offline learning and online control. Twin-delayed deep deterministic policy gradient is involved to improve the performance of the agent to minimise the frequency deviation. The advantage of the proposed method is self-adaptive to the uncertain IACMG model including renewable energy sources. Finally, the effectiveness and robustness of the proposed controller is demonstrated by four simulation scenarios. [ABSTRACT FROM AUTHOR]

The present article enables reinforcement learning (RL)‐based controllers for process control applications. Existing instances of RL‐based solutions have significant challenges for online implementation since the training process of an RL agent (controller) presently requires practically impossible number of online interactions between the agent and the environment (process). To address this challenge, we propose an implementable model‐free RL method developed by leveraging industrially implemented model predictive control (MPC) calculations (often designed using a simple linear model identified via step tests). In the first step, MPC calculations are used to pretrain an RL agent that can mimic the MPC performance. Specifically, the MPC calculations are used to pretrain the actor, and the objective function is used to pretrain the critic(s). The pretrained RL agent is then employed within a model‐free RL framework to control the process in a way that initially imitates MPC behavior (thus not compromising process performance and safety), but also continuously learns and improve its performance over the nominal linear MPC. The effectiveness of the proposed approach is illustrated through simulations on a chemical reactor example. [ABSTRACT FROM AUTHOR]

The need for reliable wireless communication in remote areas has led to the adoption of unmanned aerial vehicles (UAVs) as flying base stations (FlyBSs). FlyBSs hover over a designated area to ensure continuous communication coverage for mobile users on the ground. Moreover, rate-splitting multiple access (RSMA) has emerged as a promising interference management scheme in multi-user communication systems. In this paper, we investigate an RSMA-enhanced FlyBS downlink communication system and formulate an optimization problem to maximize the sum-rate of users, taking into account the three-dimensional FlyBS trajectory and RSMA parameters. To address this continuous non-convex optimization problem, we propose a TD3-RFBS optimization framework based on the twin-delayed deep deterministic policy gradient (TD3). This framework overcomes the limitations associated with the overestimation issue encountered in the deep deterministic policy gradient (DDPG), a well-known deep reinforcement learning method. Our simulation results demonstrate that TD3-RFBS outperforms existing solutions for FlyBS downlink communication systems, indicating its potential as a solution for future wireless networks. [ABSTRACT FROM AUTHOR]

The stability of dual active bridge converter (DAB) is threatened when feeding the constant power loads (CPLs). This article proposes a deep reinforcement learning-based backstepping control strategy to solve this problem. First, a nonlinear disturbance observer is adopted to estimate the large-signal nonlinear disturbance. Then, a backstepping controller is used to stabilize the voltage response of the DAB under the large-signal disturbance. Finally, a compensation method based on deep reinforcement learning is developed to intelligently minimize output voltage tracking error and improve the operating efficiency of the system. The proposed controller can guarantee system stability under the large-signal disturbance of the CPL and achieve a fast dynamic response with accurate voltage tracking; it is more adaptive by using the deep reinforcement learning technique through the learning of its neural networks. The effectiveness of the proposed controller is verified by experiments. [ABSTRACT FROM AUTHOR]

The energy management system (EMS) of hybridization and electrification plays a pivotal role in improving the stability and cost-effectiveness of future vehicles. Existing efforts mainly concentrate on specific optimization targets, like fuel consumption, without sufficiently taking into account the degradation of on-board power sources. In this context, a novel multi-objective energy management strategy based on deep reinforcement learning is proposed for a hybrid electric vehicle (HEV), explicitly conscious of lithium-ion battery (LIB) wear. To be specific, this paper mainly contributes to three points. Firstly, a non-parametric reward function is introduced, for the first time, into the twin-delayed deep deterministic policy gradient (TD3) strategy, to facilitate the optimality and adaptability of the proposed energy management strategy and to mitigate the effort of parameter tuning. Then, to cope with the problem of state redundancy, state space refinement techniques are included in the proposed strategy. Finally, battery health is incorporated into this multi-objective energy management strategy. The efficacy of this framework is validated, in terms of training efficiency, optimality and adaptability, under various standard driving tests. [ABSTRACT FROM AUTHOR]

A series-parallel hybrid banana-harvesting robot was previously developed to pick bananas, with inverse kinematics intractable to an address. This paper investigates a deep reinforcement learning-based inverse kinematics solution to guide the banana-harvesting robot toward a specified target. Because deep reinforcement learning algorithms always struggle to explore huge robot workspaces, a practical technique called automatic goal generation is first developed. This draws random targets from a dynamic uniform distribution with increasing randomness to facilitate deep reinforcement learning algorithms to explore the entire robot workspace. Then, automatic goal generation is applied to a state-of-the-art deep reinforcement learning algorithm, the twin-delayed deep deterministic policy gradient, to learn an effective inverse kinematics solution. Simulation experiments show that with automatic goal generation, the twin-delayed deep deterministic policy gradient solved the inverse kinematics problem with a success rate of 96.1% and an average running time of 23.8 milliseconds; without automatic goal generation, the success rate was just 81.2%. Field experiments show that the proposed method successfully guided the robot to approach all targets. These demonstrate that automatic goal generation enables deep reinforcement learning to effectively explore the robot workspace and to learn a robust and efficient inverse kinematics policy, which can, therefore, be applied to the developed series-parallel hybrid banana-harvesting robot. [ABSTRACT FROM AUTHOR]

Due to the large scale of power systems, latency uncertainty in communications can cause severe problems in wide-area measurement systems. To resolve this issue, a significant amount of past work focuses on using emerging technology, including machine learning methods such as Q-learning, for addressing latency issues in modern controls. Although the method can deal with the stochastic characteristics of communication latency, the Q-values can be overestimated in Q-learning methods, leading to high bias. To address the overestimation bias issue, we redesign the learning structure of the deep deterministic policy gradient (DDPG). Then we develop a damping control twin-delayed deep deterministic policy gradient method to handle the damping control issue under unknown latency in the power network. The purpose is to address the damping control issue under unknown latency in the power network. This paper will create a novel reward algorithm, taking into account the machine speed deviation, the episode termination prevention, and the feedback from action space. In this way, the system optimally damps down frequency oscillations while maintaining the system’s stability and reliable operation within defined limits. The simulation results verify the proposed algorithm in various perspectives, including the latency sensitivity analysis under high renewable energy penetration and the comparison with conventional and machine learning control algorithms. The proposed method shows a fast learning curve and good control performance under varying communication latency.

• Proposing a deep RL-based controller for cancer chemotherapy, which handles states and action in infinite space. • The proposed approach does not need to have a mathematical model of the system, it just needs to have tumor volume. • Providing an adaptive control technique to respond to the special conditions and diagnosis measurements of different categories of patients. • Evaluating and comparing the efficiency and robustness of the proposed method using patients with diverse conditions. Advancement in the treatment of cancer, as a leading cause of death worldwide, has promoted several research activities in various related fields. The development of effective treatment regimens with optimal drug dose administration using a mathematical modeling framework has received extensive research attention during the last decades. However, most of the control techniques presented for cancer chemotherapy are mainly model-based approaches. The available model-free techniques based on Reinforcement Learning (RL), commonly discretize the problem states and variables, which other than demanding expert supervision, cannot model the real-world conditions accurately. The more recent Deep Reinforcement Learning (DRL) methods, which enable modeling the problem in its original continuous space, are rarely applied in cancer chemotherapy. In this paper, we propose an effective and robust DRL-based, model-free method for the closed-loop control of cancer chemotherapy drug dosing. A nonlinear pharmacological cancer model is used for simulating the patient and capturing the cancer dynamics. In contrast to previous work, the state variables and control action are modeled in their original infinite spaces to avoid expert-guided discretization and provide a more realistic solution. The DRL network is trained to automatically adjust the drug dose based on the monitored states of the patient. The proposed method provides an adaptive control technique to respond to the special conditions and diagnosis measurements of different categories of patients. The performance of the proposed DRL-based controller is evaluated by numerical analysis of different diverse simulated patients. Comparison to the state-of-the-art RL-based method, which uses discretized state and action spaces, shows the superiority of the approach in the process and duration of cancer chemotherapy treatment. In the majority of the studied cases, the proposed model decreases the medication period and the total amount of administrated drug, while increasing the rate of reduction in tumor cells. [ABSTRACT FROM AUTHOR]

• Dynamic control framework for joint operations of the integrated offshore wind farm and PV power system. • Generator torque control in the wind farm is employed to mitigate the wake effects. • The TD3 control algorithm for jointly improving energy efficiency and smoothing power oscillations. • Investigating influences of hyper parameters of TD3 method on the convergence/response performances. Integrated offshore wind and photovoltaic (PV) power generation has high potential in significantly improving renewable power utilization, but the complementary operation of the integrated power system is fundamentally challenging, especially under high uncertainties of meteorological conditions. In this paper, the joint regulation of the offshore wind farm and the PV array is considered where the generator torque of each wind turbine and the tilt angles of the PV array are regarded as the control actions to simultaneously maximize the overall power generation and improve power quality. The model of the offshore wind farm including wake models and the turbine drive-train dynamic model is established while model of the PV array using the tilt angle as the regulation variable is also constructed. By considering the control of the integrated power system of the wind farm and the PV array as a typical partially-observable Markov decision process (MDP), a twin-delayed deep deterministic policy gradient (TD3) algorithm is developed to jointly regulate the wind farm and PV array. Design experiments have been conducted to evaluate the potential and effectiveness of the proposed TD3 control method for the integrated offshore wind and PV power system. The test results demonstrate that the TD3 method has good convergence performances in the overall power output and power variation rate regulations. The control objectives of improving the overall power output and smoothing the power variations can be simultaneously achieved by using appropriate parameter settings in the TD3 algorithm. [ABSTRACT FROM AUTHOR]

• The paper benefits from Reinforcement Learning in industrial control applications. • TD3 policy gradient is implemented to learn optimal PI controller dynamics. • The actor-network learns the optimal PI dynamics using the reward mechanism. • The performance of the optimal PI dynamics is compared with the other results. To benefit from the advantages of Reinforcement Learning (RL) in industrial control applications, RL methods can be used for optimal tuning of the classical controllers based on the simulation scenarios of operating conditions. In this study, the Twin Delay Deep Deterministic (TD3) policy gradient method, which is an effective actor-critic RL strategy, is implemented to learn optimal Proportional Integral (PI) controller dynamics from a Direct Current (DC) motor speed control simulation environment. For this purpose, the PI controller dynamics are introduced to the actor-network by using the PI-based observer states from the control simulation environment. A suitable Simulink simulation environment is adapted to perform the training process of the TD3 algorithm. The actor-network learns the optimal PI controller dynamics by using the reward mechanism that implements the minimization of the optimal control objective function. A setpoint filter is used to describe the desired setpoint response, and step disturbance signals with random amplitude are incorporated in the simulation environment to improve disturbance rejection control skills with the help of experience based learning in the designed control simulation environment. When the training task is completed, the optimal PI controller coefficients are obtained from the weight coefficients of the actor-network. The performance of the optimal PI dynamics, which were learned by using the TD3 algorithm and Deep Deterministic Policy Gradient algorithm, are compared. Moreover, control performance improvement of this RL based PI controller tuning method (RL-PI) is demonstrated relative to performances of both integer and fractional order PI controllers that were tuned by using several popular metaheuristic optimization algorithms such as Genetic Algorithm, Particle Swarm Optimization, Grey Wolf Optimization and Differential Evolution. [ABSTRACT FROM AUTHOR]

Due to the large scale of power systems, latency uncertainty in communications can cause severe problems in wide-area measurement systems. To resolve this issue, a significant amount of past work focuses on using emerging technology, including machine learning methods such as Q-learning, for addressing latency issues in modern controls. Although the method can deal with the stochastic characteristics of communication latency, the Q-values can be overestimated in Q-learning methods, leading to high bias. To address the overestimation bias issue, we redesign the learning structure of the deep deterministic policy gradient (DDPG). Then we develop a damping control twin-delayed deep deterministic policy gradient method to handle the damping control issue under unknown latency in the power network. The purpose is to address the damping control issue under unknown latency in the power network. This paper will create a novel reward algorithm, taking into account the machine speed deviation, the episode termination prevention, and the feedback from action space. In this way, the system optimally damps down frequency oscillations while maintaining the system's stability and reliable operation within defined limits. The simulation results verify the proposed algorithm in various perspectives, including the latency sensitivity analysis under high renewable energy penetration and the comparison with conventional and machine learning control algorithms. The proposed method shows a fast learning curve and good control performance under varying communication latency. [ABSTRACT FROM AUTHOR]