Search

In an effort to address the problem of hypersonic morphing vehicles reaching the target while avoiding no-fly zones, an improved predictor–corrector guidance method is proposed. Firstly, the aircraft motion model and the constraint model are established. Then, the basic algorithm is given. The Q-learning method is used to design the attack angle and sweep angle scheme to ensure that the aircraft can fly over low-altitude zones. The B-spline curve is used to determine the locations of flight path points, and the bank angle scheme is designed using the predictor–corrector method, so that the aircraft can avoid high-altitude zones. Next, the Monte Carlo reinforcement learning (MCRL) method is used to improve the predictor–corrector method and a Deep Neural Network (DNN) is used to fit the reward function. The planning method in this paper realizes the use of a variable sweep angle, while the improved method further improves the performance of the trajectory, including the attainment of greater final speed and a smaller turning angle. The simulation results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In recent years, solar power technology and energy storage technology have advanced, leading to the increased use of solar power devices and energy storage systems in residential areas. Carbon management has become an important method to help the community manager guide energy consumption in a timely manner, effectively reduce the carbon emissions of the community, and reduce the substantial harm to the environment. This paper aims to study the issue of carbon management and resource allocation in an intelligent community with combined heat and power (CHP) systems and solar power. The presence of heterogeneous load demands in the power grid was considered. The main objective was to minimize the average system cost over time, which included the costs associated with the power grid and gas. The Lyapunov optimization theory was employed to solve the non-convex optimization problem of carbon management and resource allocation without energy sharing. To solve the energy-sharing problem, we designed an energy-sharing algorithm based on the Q-learning algorithm. Lastly, we conducted extensive simulations using actual trace data to validate the effectiveness of our proposed algorithms. [ABSTRACT FROM AUTHOR]

Mobile ad hoc networks (MANETS) are nodes connected in a peer-to-peer fashion. Because MANETs have challenging characteristics such as mobility and limited energy, traditional existing routing protocols are not very efficient – they suffer several limitations in terms of network stability and lifetime, especially in the emerging era of IoT, crowd-sensing, and smart cities. In this work, we present SDODV, a new smart and dynamic on-demand distance vector routing protocol for mobile ad hoc networks that addresses the shortcomings of existing routing protocols. Our proposed adaptive algorithm effectively increases the built network's lifetime by considering the network topology when establishing a route. It monitors the traffic load, nodes mobility, neighborhood density, and battery power to adjust packets accordingly. This protocol is based on the distributed reinforcement learning approach and on the traditional AODV. SDODV improves the quality of service because it chooses the shortest and most stable path while considering mobility, bandwidth, and power. Experimental results prove that SDODV outperforms the shortest path method and reduces energy consumption. [ABSTRACT FROM AUTHOR]

Copyright of Aeronautical Science Bulletins / Repüléstudományi Közlemények is the property of National University of Public Service and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

This research proposes an efficient algorithm for solving the green vehicle routing problem (GVRP), capable of generating high-quality solutions while considering environmental impact and computational efficiency. We employ an adapted grey wolf optimizer (GWO) algorithm and Q-learning (QL) for parameter optimization, introducing a discrete grey wolf optimizer (DGWO), a discrete variant of the GWO. The DGWO leverages the 2-opt technique and the Hamming distance concept, making it suitable for addressing discrete problems like GVRP. The key novelty of our approach is the use of QL to refine the parameters of the DGWO, specifically the number of iterations and number of wolves. This application of QL significantly enhances the efficiency and effectiveness of the algorithm compared to DGWO with manual parameter tuning, highlighting the significance of QL in parameter optimization. The proposed discrete grey wolf optimizer-Q-learning (DGWO-QL) algorithm is extensively validated on benchmark instances of GVRP, demonstrating promising results. For smaller benchmark instances comprising of 20 customers and 3 stations, our approach outperforms in 12 out of 16 instances. When tested on larger benchmark instances, within a range of 111 to 500 customers and 21 stations, it achieves success in 10 out of 12 instances. Compared to existing methods, our approach demonstrates improved performance in terms of solution quality and computational efficiency. The results show the robust performance of the DGWO-QL, particularly under stochastic route scenarios, which underscores the advantage of the proposed technique. This study represents a significant contribution to the current body of literature by underscoring the potential of the DGWO-QL algorithm in generating high-quality solutions for GVRP. [ABSTRACT FROM AUTHOR]

We investigate the ability of a platform to design its marketplace to promote competition, improve consumer surplus, and increase its own payoff. We consider demand‐steering rules that reward firms that cut prices with additional exposure to consumers. We examine the impact of these rules both in theory and by using simulations with artificial intelligence pricing algorithms (specifically Q‐learning algorithms, which are commonly used in computer science). Our theoretical results indicate that these policies (which require little information to implement) can have strongly beneficial effects, even when sellers are infinitely patient and seek to collude. Similarly, our simulations suggest that platform design can benefit consumers and the platform, but that achieving these gains may require policies that condition on past behavior and treat sellers in a nonneutral fashion. These more sophisticated policies disrupt the ability of algorithms to rotate demand and split industry profits, leading to low prices. [ABSTRACT FROM AUTHOR]

Path planning for Unmanned Aerial Vehicles in environments with obstacles remains a challenging task. Traditional algorithms, such as A* and Dijkstra, have limitations when dealing with dynamic and changing obstacles, as well as unknown environments. In this paper, a QLearning approach for the path planning of UAVs in obstacle-rich and unknown environments is proposed. The impact of alpha, gamma, epsilon, the initial matrix, and reward parameters on the learning process is investigated to achieve safe and cost-effective paths with reduced execution time. The proposed approach is evaluated through simulations by adjusting the alpha, gamma, epsilon, initial matrix, and reward values, and the results demonstrate the effectiveness of the proposed method. The simulation results show that adjusting all the studied parameters can significantly improve the performance of the proposed approach, leading to paths that meet cost and timing objectives while avoiding obstacles. [ABSTRACT FROM AUTHOR]

Many previous studies showed that some nodes should be in a sleeping state while the traffic is high. These nodes can wake up periodically to transmit data while they have data to transmit. The system throughput will be increased while each node can change between wake-up and sleeping states periodically. If a node can estimate the active rate and wake up at the optimal time, the collision probability decreases. This study proposes a Q-learning-based distributed queuing medium access control (QL-based DQMAC) protocol for Internet-of-Things (IoT) networks. In the proposed QL-based DQMAC, we derive the optimal number of contention IoT nodes. Each node calculates the active rate by itself through the Q-learning algorithm. Then, each node determines whether it will be active or in sleeping mode in the next contention period according to the active rate. Finding the optimal IoT nodes in each contention period decreases the probability of collision. The energy consumption due to the contention and delay for MAC contention is reduced owing to the lower number of contentions. Protocol comparison with other DQMAC protocols shows that the proposed QL-based DQMAC protocol achieves higher performance in IoT networks. [ABSTRACT FROM AUTHOR]

Lots of learning tasks require experience learning based on activities performed in real scenarios which are affected by environmental factors. Therefore, real-time systems demand a model to learn from working experience—such as physical object properties-driven system models, trajectory prediction, and Atari games. This experience-driven learning model uses reinforcement learning which is considered as an important research topic and needs problem-specific reasoning model simulation. In this research paper, cart-pole balancing problem is selected as a problem where the system learns using Q-learning and Deep Q network reinforcement learning approaches. Pragmatic foundation of cart-pole problem and its solution with the help of Q learning and DQN reinforcement learning model are validated, and a comparison of achieved outcome in the form of accuracy and fast convergence is presented. An unexperienced Huber loss function is applied on cart-pole balancing problem, and results are in favor of Huber loss function in comparison with mean-squared error loss function. Hence, experimental study suggests the use of DQN with Huber loss reward function for fast learning and convergence of cart pole in balanced condition. [ABSTRACT FROM AUTHOR]

Recent studies and literature reviews have shown promising results for 3GPP system solutions in unlicensed bands when coexisting with Wi-Fi, either by using the duty cycle (DC) approach or licensed-assisted access (LAA). However, it is widely known that general performance in these coexistence scenarios is dependent on traffic and how the duty cycle is adjusted. Most DC solutions configure their parameters statically, which can result in performance losses when the scenario experiences changes on the offered data. In our previous works, we demonstrated that reinforcement learning (RL) techniques can be used to adjust DC parameters. We showed that a Q-learning (QL) solution that adapts the LTE DC ratio to the transmitted data rate can maximize the Wi-Fi/LTE-Unlicensed (LTE-U) aggregated throughput. In this paper, we extend our previous solution by implementing a simpler and more efficient algorithm based on multiarmed bandit (MAB) theory. We evaluate its performance and compare it with the previous one in different traffic scenarios. The results demonstrate that our new solution offers improved balance in throughput, providing similar results for LTE and Wi-Fi, while still showing a substantial system gain. Moreover, in one of the scenarios, our solution outperforms the previous approach by 6% in system throughput. In terms of user throughput, it achieves more than 100% gain for the users at the 10th percentile of performance, while the old solution only achieves a 10% gain. [ABSTRACT FROM AUTHOR]

The explosive growth of users and applications in IoT environments has promoted the development of cloud computing. In the cloud computing environment, task scheduling plays a crucial role in optimizing resource utilization and improving overall performance. However, effective task scheduling remains a key challenge. Traditional task scheduling algorithms often rely on static heuristics or manual configuration, limiting their adaptability and efficiency. To overcome these limitations, there is increasing interest in applying reinforcement learning techniques for dynamic and intelligent task scheduling in cloud computing. How can reinforcement learning be applied to task scheduling in cloud computing? What are the benefits of using reinforcement learning-based methods compared to traditional scheduling mechanisms? How does reinforcement learning optimize resource allocation and improve overall efficiency? Addressing these questions, in this paper, we propose a Q-learning-based Multi-Task Scheduling Framework (QMTSF). This framework consists of two stages: First, tasks are dynamically allocated to suitable servers in the cloud environment based on the type of servers. Second, an improved Q-learning algorithm called UCB-based Q-Reinforcement Learning (UQRL) is used on each server to assign tasks to a Virtual Machine (VM). The agent makes intelligent decisions based on past experiences and interactions with the environment. In addition, the agent learns from rewards and punishments to formulate the optimal task allocation strategy and schedule tasks on different VMs. The goal is to minimize the total makespan and average processing time of tasks while ensuring task deadlines. We conducted simulation experiments to evaluate the performance of the proposed mechanism compared to traditional scheduling methods such as Particle Swarm Optimization (PSO), random, and Round-Robin (RR). The experimental results demonstrate that the proposed QMTSF scheduling framework outperforms other scheduling mechanisms in terms of the makespan and average task processing time. [ABSTRACT FROM AUTHOR]

In recent sequential multiple assignment randomized trials, outcomes were assessed multiple times to evaluate longer-term impacts of the dynamic treatment regimes (DTRs). Q-learning requires a scalar response to identify the optimal DTR. Inverse probability weighting may be used to estimate the optimal outcome trajectory, but it is inefficient, susceptible to model mis-specification, and unable to characterize how treatment effects manifest over time. We propose modified Q-learning with generalized estimating equations to address these limitations and apply it to the M-bridge trial, which evaluates adaptive interventions to prevent problematic drinking among college freshmen. Simulation studies demonstrate our proposed method improves efficiency and robustness. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) is a subfield of machine learning that has shown significant promise in improving the efficiency and quality of additive manufacturing (AM). Additive manufacturing, also known as 3D printing, is a process of building objects layer by layer using digital models. RL can help optimize the manufacturing process by allowing machines to learn from their experiences and make decisions accordingly. In AM, this can be particularly useful for controlling the printing parameters such as temperature, speed, and material flow rate. By using RL algorithms to optimize these parameters, manufacturers can achieve higher levels of precision, accuracy, and speed, while minimizing waste and reducing costs. The goal of the present research work is to find the optimal parameters of an additively manufactured specimens' experimental dataset that contains four input parameters i.e. Infill percentage, Layer height, Print speed and Extrusion temperature and three output parameters i.e. Tensile Strength, Flexural Strength and Impact Strength using Q-learning. Q-learning is a popular reinforcement learning algorithm that is used to find the optimal policy for an agent in a given environment. In the present work, Q-learning algorithm is used to optimize the parameters of the dataset in order to achieve the highest possible performance. The results showed that the obtained Tensile Strength, Flexural Strength and Impact strength have error of 0.66%, 4.37%, and 10.2% in compared to the experimental results. [ABSTRACT FROM AUTHOR]

Edge computing has emerged as a promising solution to reduce communication delays and enhance the performance of real-time applications. However, due to the limited processing power of edge servers, it is challenging to achieve efficient task processing. In this paper, we propose a Q-learning-based load-balancing method that optimizes the distribution of real-time tasks between edge servers and cloud servers to reduce processing time. The proposed method is dynamic and adaptive, taking into account the constantly changing network status and server usage. To evaluate the effectiveness of the proposed load-balancing method, extensive simulations are conducted in an Edge-Cloud network environment. The simulation results demonstrate that the proposed method significantly reduces processing time compared to traditional static load-balancing methods. The Q-learning algorithm enables the load-balancing system to dynamically learn the optimal decision-making strategy to allocate tasks to the most appropriate server. Overall, the proposed Q-learning-based load-balancing method provides a dynamic and efficient solution to balance the workload between edge servers and cloud servers. The proposed method effectively achieves real-time task processing in edge computing environments and can contribute to the development of high-performance edge computing systems. [ABSTRACT FROM AUTHOR]

This paper describes a Q-learning based algorithm to design the linear quadratic tracker (LQT) for linear time invariant (LTI) continuous-time systems with partially unknown dynamics. The proposed approach uses a fixed-point equation in terms of Q-function in order to estimate the unknown optimal gain parameters. The fixedpoint equation, which is derived by applying the Pontryagin's minimum principle in Q-learning, is based on the modified algebraic Riccati equation (ARE) for LQT problem. The online adaptation of the optimal parameters are achieved by using the gradient descent based parameter update laws by minimizing the Bellman's error term which is derived from fixed-point equation mentioned earlier. A persistence of excitation condition has been used to establish the desired optimal convergence of the estimated control parameters. Simulation results have been shown to validate the efficiency of the proposed Q-learning approach. [ABSTRACT FROM AUTHOR]

The technological advances of smart cities have been progressively increasing to improve the quality of life to humans, especially in urban mobility. Parking appears to be a major issue, with residents needing to find a suitable parking space among many parking areas, resulting in time and fuel waste as well as environmental pollution. We propose in this paper a new automated system model that integrates reinforcement learning (RL), Q-learning, and image processing algorithms based on modified internet of spatial things (IoST) architecture to optimize automated parking in smart cities. For demonstrating the efficiency of the proposed model, iFogSim simulation is used to reduce network usage and latency. Moreover, it deploys heterogeneous devices in multi layers and different scenarios. The experimental results show that the suggested system for automated car parking in fog-based placement-IoST network is feasible and effective. It minimizes latency and the total network usage compared to the cloud-based placement of the implemented system. [ABSTRACT FROM AUTHOR]

Fog computing is one of the emerging forms of cloud computing which aims to satisfy the ever-increasing computation demands of the mobile applications. Effective offloading of tasks leads to increased efficiency of the fog network, but at the same time it suffers from various uncertainty issues with respect to task demands, fog node capabilities, information asymmetry, missing information, low trust, transaction failures, and so on. Several machine learning techniques have been proposed for the task offloading in fog environments, but they lack efficiency. In this paper, a novel uncertainty proof Type-2-Soft-Set (T2SS) enabled apprenticeship learning based task offloading framework is proposed which formulates the optimal task offloading policies. The performance of the proposed T2SS based apprenticeship learning is compared and found to be better than Q-learning and State-Action-Reward-State-Action (SARSA) learning techniques with respect to performance parameters such as total execution time, throughput, learning rate, and response time.

The essential part of existing reinforcement learning algorithms that use Gaussian process regression involves a complicated online Gaussian process regression algorithm. Our study proposes online and mini-batch Gaussian process regression algorithms that are easier to implement and faster to estimate for reinforcement learning. In our algorithm, the Gaussian process regression updates the value function through only the computation of two equations, which we then use to construct reinforcement learning algorithms. Our numerical experiments show that the proposed algorithm works as well as those from previous studies.

The trade-off between exploitation and exploration is a dilemma inherent to particle swarm optimization (PSO) algorithms. Therefore, a growing body of PSO variants is devoted to solving the balance between the two. Among them, the method of self-adaptive multi-strategy selection plays a crucial role in improving the performance of PSO algorithms but has yet to be well exploited. In this research, with the aid of the reinforcement learning technique to guide the generation of offspring, a novel self-adaptive multi-strategy selection mechanism is designed, and then a multi-strategy self-learning PSO algorithm based on reinforcement learning (MPSORL) is proposed. First, the fitness value of particles is regarded as a set of states that are divided into several state subsets non-uniformly. Second, the ε-greedy strategy is employed to select the optimal strategy for each particle. The personal best particle and the global best particle are then updated after executing the strategy. Subsequently, the next state is determined. Thus, the value of the Q-table, as a scheme adopted in self-learning, is reshaped by the reward value, the action and the state in a non-stationary environment. Finally, the proposed algorithm is compared with other state-of-the-art algorithms on two well-known benchmark suites and a real-world problem. Extensive experiments indicate that MPSORL has better performance in terms of accuracy, convergence speed and non-parametric tests in most cases. The multi-strategy selection mechanism presented in the manuscript is effective.

Vehicular ad hoc networks (VANETs) use multiple channels to communicate using wireless access in vehicular environment (WAVE) standards to provide a variety of vehicle-related applications. The current IEEE 802.11p WAVE communication channel structure is composed of one control channel (CCH) and several service channels (SCHs). SCHs are used for non-safety data transmission, while the CCH is used for broadcasting beacons, control, and safety. WAVE devices transmit data that alternate between CCHs and SCHs, and each channel is active for a duration called the CCH interval (CCHI) and SCH interval (SCHI), respectively. Currently, both intervals are fixed at 50 ms. However, fixed-length intervals cannot effectively respond to dynamically changing traffic loads. Additionally, when many vehicles are simultaneously using the limited channel resources for data transmission, the network performance significantly degrades due to numerous packet collisions. Herein, we propose an adaptive resource allocation technique for efficient data transmission. The technique dynamically adjusts the SCHI and CCHI to improve network performance. Moreover, to reduce data collisions and optimize the network’s backoff distribution, the proposed scheme applies reinforcement learning (RL) to provide an intelligent channel access algorithm. The simulation results demonstrate that the proposed scheme can ensure high throughputs and low transmission delays.

The transition to Industry 4.0 and 5.0 underscores the need for integrating humans into manufacturing processes, shifting the focus towards customization and personalization rather than traditional mass production. However, human performance during task execution may vary. To ensure high human–robot teaming (HRT) performance, it is crucial to predict performance without negatively affecting task execution. Therefore, to predict performance indirectly, significant factors affecting human performance, such as engagement and task load (i.e., amount of cognitive, physical, and/or sensory resources required to perform a particular task), must be considered. Hence, we propose a framework to predict and maximize the HRT performance. For the prediction of task performance during the development phase, our methodology employs features extracted from physiological data as inputs. The labels for these predictions—categorized as accurate performance or inaccurate performance due to high/low task load—are meticulously crafted using a combination of the NASA TLX questionnaire, records of human performance in quality control tasks, and the application of Q-Learning to derive task-specific weights for the task load indices. This structured approach enables the deployment of our model to exclusively rely on physiological data for predicting performance, thereby achieving an accuracy rate of 95.45% in forecasting HRT performance. To maintain optimized HRT performance, this study further introduces a method of dynamically adjusting the robot’s speed in the case of low performance. This strategic adjustment is designed to effectively balance the task load, thereby enhancing the efficiency of human–robot collaboration.

This research article delves into the development of a reinforcement learning (RL)-based continuous authentication system utilizing behavioral biometrics for user identification on computing devices. Keystroke dynamics are employed to capture unique behavioral biometric signatures, while a reward-driven RL model is deployed to authenticate users throughout their sessions. The proposed system augments conventional authentication mechanisms, fortifying them with an additional layer of security to create a robust continuous authentication framework compatible with static authentication systems. The methodology entails training an RL model to discern atypical user typing patterns and identify potentially suspicious activities. Each user’s historical data are utilized to train an agent, which undergoes preprocessing to generate episodes for learning purposes. The environment involves the retrieval of observations, which are intentionally perturbed to facilitate learning of nonlinear behaviors. The observation vector encompasses both ongoing and summarized features. A binary and minimalist reward function is employed, with principal component analysis (PCA) utilized for encoding ongoing features, and the double deep Q-network (DDQN) algorithm implemented through a fully connected neural network serving as the policy net. Evaluation results showcase training accuracy and equal error rate (EER) ranging from 94.7% to 100% and 0 to 0.0126, respectively, while test accuracy and EER fall within the range of approximately 81.06% to 93.5% and 0.0323 to 0.11, respectively, for all users as encoder features increase in number. These outcomes are achieved through RL’s iterative refinement of rewards via trial and error, leading to enhanced accuracy over time as more data are processed and incorporated into the system.

This research paper presents the Buckley-James Q-learning (BJ-Q) algorithm, a cutting-edge method designed to optimize personalized treatment strategies, especially in the presence of right censoring. We critically assess the algorithm’s effectiveness in improving patient outcomes and its resilience across various scenarios. Central to our approach is the innovative use of the survival time to impute the reward in Q-learning, employing the Buckley-James method for enhanced accuracy and reliability. Our findings highlight the significant potential of personalized treatment regimens and introduce the BJ-Q learning algorithm as a viable and promising approach. This work marks a substantial advancement in our comprehension of treatment dynamics and offers valuable insights for augmenting patient care in the ever-evolving clinical landscape.

This paper investigates the output feedback (OPFB) tracking control problem for discrete-time linear (DTL) systems with unknown dynamics. To solve this problem, we use an augmented system approach, which first transforms the tracking control problem into a regulation problem with a discounted performance function. The solution to this problem is derived using a Bellman equation, based on the Q-function. In order to overcome the challenges of unmeasurable system state variables, we employ a multistep Q-learning algorithm that surpasses the advantages of the policy iteration (PI) and value iteration (VI) techniques and state reconstruction methods for output feedback control. As such, the requirement for an initial stabilizing control policy for the PI method is removed and the convergence speed of the learning algorithm is improved. Finally, we demonstrate the effectiveness of the proposed scheme using a simulation example.

This article proposes an adaptable path tracking control system, based on reinforcement learning (RL), for autonomous cars. A four-parameter controller shapes the behaviour of the vehicle to navigate lane changes and roundabouts. The tuning of the tracker uses an ‘educated’ Q-Learning algorithm to minimize the lateral and steering trajectory errors, this being a key contribution of this article. The CARLA (CAR Learning to Act) simulator was used both for training and testing. The results show the vehicle is able to adapt its behaviour to the different types of reference trajectories, navigating safely with low tracking errors. The use of a robot operating system (ROS) bridge between CARLA and the tracker (i) results in a realistic system, and (ii) simplifies the replacement of CARLA by a real vehicle, as in a hardware-in-the-loop system. Another contribution of this article is the framework for the dependability of the overall architecture based on stability results of non-smooth systems, presented at the end of this article.

Opportunistic network (Oppnet) is consisting of wireless nodes, which follows the Bluetooth transmission range between nodes. These kinds of networks are capable to handle long delay during transmission of messages. Dynamic network topology and unstable connection between nodes makes routing difficult in Oppnet. Overall to provide efficient and reliable transmission of messages are such a crucial task in Oppnet. To resolve these issues; this article introduces reinforcement learning based on Deep Q-learning technique. In this approach, Q-value is used to find best current intermediate node for forwarding the packet from source Oppnet node to destination Oppnet node. This protocol is simulated on ONE tool and it is found that Q-Routing in Oppnet works more efficiently as compared to conventional protocols like Epidemic, Maximum Probability (Max-Prob), and Probability Routing Protocol using History of Encounters and Transitivity (PRoPHET). Proposed protocol utilized; 71% lesser energy consumption, 66% reduced overhead ratio, 61% lesser end-to-end delay, 51% enhanced throughput, and 10% higher delivery ratio as compared to PRoPHET protocol when variation in number of nodes. [ABSTRACT FROM AUTHOR]

There has been tremendous interest in the development of DC microgrid systems which consist of interconnected DC renewable energy sources. However, operating a DC microgrid system optimally by minimizing operational cost and ensures stability remains a problem when the system's model is not available. In this paper, a novel model-free approach to perform operation control of DC microgrids based on reinforcement learning algorithms, specifically Q-learning and Q-network, has been proposed. This approach circumvents the need to know the accurate model of a DC grid by exploiting an interaction with the DC microgrids to learn the best policy, which leads to more optimal operation. The proposed approach has been compared with with mixed-integer quadratic programming (MIQP) as the baseline deterministic model that requires an accurate system model. The result shows that, in a system of three nodes, both Q-learning (74.2707) and Q-network (74.4254) are able to learn to make a control decision that is close to the MIQP (75.0489) solution. With the introduction of both model uncertainty and noisy sensor measurements, the Q-network performs better (72.3714) compared to MIQP (72.1596), whereas Q-learn fails to learn. [ABSTRACT FROM AUTHOR]

In this paper, we propose the notification optimization method by providing multiple alternative times as a reminder for a forecasted activity with and without probabilistic considerations for the activity that needs to be completed and needs notification. It is important to consider various factors when sending notifications to people after obtaining the results of the forecasted activity. We should not send notifications only when we have forecasted results because future daily activities are unpredictable. Therefore, it is important to strike a balance between providing useful reminders and avoiding excessive interruptions, especially for low probabilities of forecasted activity. Our study investigates the impact of the low probability of forecasted activity and optimizes the notification time with reinforcement learning. We also show the gaps between forecasted activities that are useful for self-improvement by people for the balance of important tasks, such as tasks completed as planned and additional tasks to be completed. For evaluation, we utilize two datasets: the existing dataset and data we collected in the field with the technology we have developed. In the data collection, we have 23 activities from six participants. To evaluate the effectiveness of these approaches, we assess the percentage of positive responses, user response rate, and response duration as performance criteria. Our proposed method provides a more effective way to optimize notifications. By incorporating the probability level of activity that needs to be done and needs notification into the state, we achieve a better response rate than the baseline, with the advantage of reaching 27.15%, as well as than the other criteria, which are also improved by using probability. [ABSTRACT FROM AUTHOR]

This paper deals with a robust H∞$$ {H}_{\infty } $$ tracking problem with a discounted factor. A new auxiliary system is established in terms of norm‐bounded time‐varying uncertainties. It is shown that the robust discounted H∞$$ {H}_{\infty } $$ tracking problem for the auxiliary system solves the original problem. Then, the new robust discounted H∞$$ {H}_{\infty } $$ tracking problem is represented as a well‐known zero‐sum game problem. Moreover, the robust tracking Bellman equation and the robust tracking Algebraic Riccati equation (RTARE) are inferred. A lower bound of a discounted factor for stability is obtained to assure the stability of the closed‐loop system. Based on the auxiliary system, the system is reshaped in a new structure that is applicable to Reinforcement Learning methods. Finally, an online Q‐learning algorithm without the knowledge of system matrices is proposed to solve the algebraic Riccati equation associated with the robust discounted H∞$$ {H}_{\infty } $$ tracking problem for the auxiliary system. Simulation results are given to verify the effectiveness and merits of the proposed method. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Guangxi Normal University - Natural Science Edition is the property of Gai Kan Bian Wei Hui and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The dispatching of automated guided vehicles (AGVs) is essential for efficient horizontal transportation at automated container terminals. Effective planning of AGV transportation can reduce equipment energy consumption and shorten task completion time. Multiple AGVs transport containers between storage blocks and vessels, which can be regarded as the supply sides and demand points of containers. To meet the requirements of shipment in terms of timely and high-efficient delivery, multiple AGVs should be dispatched to deliver containers, which includes assigning tasks and selecting paths. A contract net protocol (CNP) is employed for task assignment in a multiagent system, while machine learning provides a logical alternative, such as Q-learning (QL), for complex path planning. In this study, mathematical models for multi-AGV dispatching are established, and a QL-CNP algorithm is proposed to tackle the multi-AGV dispatching problem (MADP). The distribution of traffic load is balanced for multiple AGVs performing tasks in the road network. The proposed model is validated using a Gurobi solver with a small experiment. Then, QL-CNP is used to conduct experiments with different sizes. The other algorithms, including Dijkstra, GA, and PSO, are also compared with the QL-CNP algorithm. The experimental results demonstrate the superiority of the proposed QL-CNP when addressing the MADP. [ABSTRACT FROM AUTHOR]

The dynamic programming principle (DPP) is fundamental for control and optimization, including Markov decision problems (MDPs), reinforcement learning (RL), and, more recently, mean-field controls (MFCs). However, in the learning framework of MFCs, the DPP has not been rigorously established, despite its critical importance for algorithm designs. In this paper, we first present a simple example in MFCs with learning where the DPP fails with a misspecified Q function and then propose the correct form of Q function in an appropriate space for MFCs with learning. This particular form of Q function is different from the classical one and is called the IQ function. In the special case when the transition probability and the reward are independent of the mean-field information, it integrates the classical Q function for single-agent RL over the state-action distribution. In other words, MFCs with learning can be viewed as lifting the classical RLs by replacing the state-action space with its probability distribution space. This identification of the IQ function enables us to establish precisely the DPP in the learning framework of MFCs. Finally, we illustrate through numerical experiments the time consistency of this IQ function. [ABSTRACT FROM AUTHOR]

Copyright of ESPOCH Congresses: The Ecuadorian Journal of S.T.E.A.M. is the property of Knowledge E DMCC and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In this paper, we investigate the optimal control strategies for model‐free zero‐sum games involving the H∞$$ {H}_{\infty } $$ control. The key contribution is the development of a Q‐learning algorithm for linear quadratic games without knowing the system dynamics. The finite‐horizon setting is more practical than the infinite‐horizon setting, but it is difficult to solve the time‐varying Riccati equation associated with the finite‐horizon setting directly. The proposed algorithm is shown to solve the time‐varying Riccati equation iteratively without the use of models, and numerical experiments on aircraft dynamics demonstrate the algorithm's efficiency. [ABSTRACT FROM AUTHOR]

In open pit mines, the long-term scheduling defines how the mine should be developed. Uncertainties in geological attributes makes the search for an optimal scheduling a challenging problem. In this work, we provide a framework to account for uncertainties in the spatial distribution of grades in long-term mine planning using deep Q-Learning. Mining, processing and metallurgical constraints are accounted as restrictions in the reinforcement learning environment. Such environment provides a flexible structure to incorporate geometallurgical properties in production scheduling, as part of the block model. Geometric constraints (block precedence) and operational restrictions have been included as part of the agent-environment interaction. The effectiveness of the method is demonstrated in a controlled study case using a real multivariate drill-hole dataset, maximizing the net-present value of the project. The present framework can be extended and improved, to meet the particular needs and requirements of mining operations. We discuss on the current limitations and potential for further research and applications. [ABSTRACT FROM AUTHOR]

The discrete strategy game, in which agents can only choose cooperation or defection, has received lots of attention. However, this hypothesis seems implausible in the real world, where choices may be continuous or mixed. Furthermore, when applying Q -learning to continuous or mixed strategy games, one of the challenges is that the learning space grows drastically as the number of states and actions rises. So, in this article, we redesign the Q -learning method by considering the spatial reciprocity, in which agents simply interact with their four neighbours to get the reward and learn the action by taking neighbours' strategy into account. As a result, the learning state and action space is transformed into a 5×5 table that stores the state and action of the focal agent and its four neighbours, avoiding the curse of dimensionality caused by a continuous or mixed strategy game. The numerical simulation results reveal the striking differences between the three classes of games. In detail, the discrete strategy game is more vulnerable to the setting of relevant parameters, whereas the other two strategy games are relatively stable. At the same time, in terms of promoting cooperation, a mixed strategy game is always better than a continuous one. [ABSTRACT FROM AUTHOR]

Concurrent access to large-scale video data streams in edge computing is an important application scenario that currently faces a high cost of network access equipment and high data packet loss rate. To solve this problem, a low-cost link aggregation video stream data concurrent transmission method is proposed. Data Plane Development Kit (DPDK) technology supports the concurrent receiving and forwarding function of multiple Network Interface Cards (NICs). The Q-learning data stream scheduling model is proposed to solve the load scheduling of multiple queues of multiple NICs. The Central Processing Unit (CPU) transmission processing unit was dynamically selected by data stream classification, as well as a reward function, to achieve the dynamic load balancing of data stream transmission. The experiments conducted demonstrate that this method expands the bandwidth by 3.6 times over the benchmark scheme for a single network port, and reduces the average CPU load ratio by 18%. Compared to the UDP and DPDK schemes, it lowers the average system latency by 21%, reduces the data transmission packet loss rate by 0.48%, and improves the overall system transmission throughput. This transmission optimization scheme can be applied in data centers and edge computing clusters to improve the communication performance of big data processing. [ABSTRACT FROM AUTHOR]

Aiming at the attitude control problem of hypersonic flight vehicle, a compound control strategy based on the disturbance compensation technique and Q-learning optimization is proposed. Firstly, a three-channel independent control scheme based on the super-twisting extended state observer (STESO) is proposed such that coupling among channels, uncertainties, and external disturbances can be estimated and compensated in real time. Secondly, the Q-learning mechanism is introduced to optimize the control parameters while keeping the control structure unchanged. Lastly, the convergence of the STESO is analyzed by the Lyapunov theory, and some numerical simulation results are carried out to verify the effectiveness of the proposed strategy. Compared with the traditional linear active disturbance rejection control, the proposed method can avoid a lot of manual parameter tuning process and also has better performance. [ABSTRACT FROM AUTHOR]

Achieving sample efficiency in online episodic reinforcement learning (RL) requires optimally balancing exploration and exploitation. When it comes to a finite-horizon episodic Markov decision process with |$S$| states, |$A$| actions and horizon length |$H$|⁠ , substantial progress has been achieved toward characterizing the minimax-optimal regret, which scales on the order of |$\sqrt{H^2SAT}$| (modulo log factors) with |$T$| the total number of samples. While several competing solution paradigms have been proposed to minimize regret, they are either memory-inefficient, or fall short of optimality unless the sample size exceeds an enormous threshold (e.g. |$S^6A^4 \,\mathrm{poly}(H)$| for existing model-free methods). To overcome such a large sample size barrier to efficient RL, we design a novel model-free algorithm, with space complexity |$O(SAH)$|⁠ , that achieves near-optimal regret as soon as the sample size exceeds the order of |$SA\,\mathrm{poly}(H)$|⁠. In terms of this sample size requirement (also referred to the initial burn-in cost), our method improves—by at least a factor of |$S^5A^3$| —upon any prior memory-efficient algorithm that is asymptotically regret-optimal. Leveraging the recently introduced variance reduction strategy (also called reference-advantage decomposition), the proposed algorithm employs an early-settled reference update rule, with the aid of two Q-learning sequences with upper and lower confidence bounds. The design principle of our early-settled variance reduction method might be of independent interest to other RL settings that involve intricate exploration–exploitation trade-offs. [ABSTRACT FROM AUTHOR]