Search

Q-learning has long been one of the most popular reinforcement learning algorithms, and theoretical analysis of Q-learning has been an active research topic for decades. Although researches on asymptotic convergence analysis of Q-learning have a long tradition, non-asymptotic convergence has only recently come under active study. The main goal of this paper is to investigate a new finite-time analysis of asynchronous Q-learning under Markovian observation models via a control system viewpoint. In particular, we introduce a discrete-time time-varying switching system model of Q-learning with diminishing step-sizes for our analysis, which significantly improves recent development of the switching system analysis with constant step-sizes, and leads to $\mathcal {O}(\sqrt {{\log k}/{k}})$ convergence rate that is comparable to or better than most of the state of the art results in the literature. In the meanwhile, we consider the continuous-time Lyapunov equation to avoid the difficulty in the analysis posed by using diminishing step-sizes in discrete-time. The proposed analysis brings in additional insights, covers different scenarios, and provides new simplified templates for analysis to deepen our understanding on Q-learning via its unique connection to discrete-time switching systems.

Current multi-objective reinforcement learning (MORL) research often struggles to balance multiple objectives and manage the stability and performance of learning algorithms, especially in complex environments. To address this, we propose a new multi-objective deep Q-network (MO-DQN) framework that integrates linear scalarization in MORL. This framework has the following features: First, we provide immediate feedback by incorporating linear scalarization into reward processing. Compared with some complex multi-objective optimization methods, this approach is relatively easy to understand and implement, offering greater convenience for practical applications. Additionally, linear scalarization accelerates the learning process and enhances the algorithm’s ability to dynamically adjust strategies. Second, we develop the Linear Scalarized Multi-objective Deep Q-Network (LSMO-DQN) under different reward mechanisms, improving MO-DQN’s ability to balance multiple objectives effectively. Immediate reward strategies accelerate learning and enable rapid adjustments, benefiting dynamic environments. In contrast, delayed reward strategies help understand long-term action impacts and promote strategic decision-making. Experiments are conducted in two different multi-objective environments, where our proposed method slightly outperforms other techniques, indicating its robustness and adaptability. Specifically, LSMO-DQN achieves higher cumulative rewards and demonstrates improved stability across various reward structures. The findings suggest that integrating linear scalarization in reward processing not only enhances learning performance but also provides a more straightforward approach to managing the trade-offs in multi-objective settings. These results highlight the potential of LSMO-DQN to improve learning performance, particularly in scenarios with data imbalances.

Maintaining player engagement in serious management games is a challenge due to the repetitive nature of traditional predetermined difficulty levels. A dynamic difficulty adjustment (DDA) system is introduced in this study to address this issue by integrating fuzzy logic and Q-learning. Player ennui is frequently the consequence of static difficulty adjustments. In order to dynamically adjust game complexity in accordance with player performance and preferences, our DDA system utilizes a diverse array of performance metrics, adaptive narrative elements, and real-time feedback, as well as fuzzy logic and Q-learning algorithms. According to empirical assessments, players were 28% more effective overall, and play sessions lasted an average of 35% longer. Player satisfaction and involvement were also much improved. Customers played the game longer and were less bored because of the higher degree of difficulty and customization. The integration of fuzzy logic and Q-learning in DDA systems greatly enhances the ability to maintain long-term player engagement in essential management games. This approach offers a long-lasting alternative for creating constantly captivating gaming experiences by effectively reducing the repetitiveness of traditional difficulty adjustments.

In distributed Software-defined networking (SDN), control plane functions are partitioned across multiple controller instances to enhance fault tolerance and scalability. However, the dynamic nature of network traffic and rapid network events, such as link failures and controller node failures, can lead to uneven workload distribution among controller nodes. This research aims to adjust switch-to-controller mapping to address load imbalance dynamically. We model flow arrivals at switches and subsequent actions within a Markov decision process (MDP) framework. In MDP, precise knowledge of the arrival rate is required, however, such an assumption is impractical in dynamic environments. Reinforcement learning (RL) learns policies from environment interactions, enabling autonomous decision-making in complex domains by adeptly navigating uncertainties. The proposed scheme uses RL to monitor SDN flow dynamics and maintain system load balance through switch migration. Herein, the proposed scheme generates migration triplets specifying the source controller, the destination controller for migration, and the switch to be migrated. The scheme considers the cost of migrating the flows in terms of the flow arrival rate and hop count between the switch and the controllers. Experimental results confirm that the framework effectively achieves load balancing across different network topologies and diverse traffic load distributions on switches.

This paper investigates the secrecy performance of a multi-relay non-orthogonal multiple access (NOMA) network. Considering the presence of eavesdroppers, an optimal relay selection scheme and a jamming signal transmission scheme for both amplify-and-forward (AF) and decode-and-forward (DF) strategies are proposed. However, the resource allocation problem aimed at maximizing the effective secure throughput (EST) of the system is non-convex. It is difficult to directly solve this optimization problem using conventional methods. As such, a Q-learning approach to solve the resource allocation problem in this system is applied, and an innovative reward function that can maximize the communication quality of edge users while ensuring secure communication of nearby users is designed. According to the analysis of the simulation results, the convergence of the proposed scheme is verified. Under the same conditions, DF relays achieve a higher signal-to-noise ratio (SINR) at the user terminal, and the EST is closely related to the transmission power of the source node and the relays. The numerical results also show that compared to conventional power allocation methods, the proposed method achieves a larger average EST and provides better confidentiality performance.

Trajectory planning for unmanned aerial vehicles (UAVs) presents a formidable challenge, particularly in environments where details about the terrain and objectives are not known beforehand. This uncertainty has propelled the adoption of UAVs across diverse sectors such as logistics, weather forecasting, security surveillance, and autonomous driving, where their ability to navigate autonomously is crucial. Recent advancements have leveraged deep learning for robust object detection and tracking, significantly enhancing data acquisition from aerial imagery and UAV-based applications. Despite numerous innovations, the critical issue of optimizing UAV trajectories in unknown environments remains largely unaddressed. This paper introduces a novel approach, the Instructed Reinforcement Q-Learning Algorithm (IR-QLA), which utilizes Received Signal Strength (RSS) as a dynamic reward metric to refine trajectory planning continuously. By integrating a unique instructed reinforcement principle, this method significantly accelerates the learning process, improving the UAVs’ navigational efficiencies. Our findings demonstrate that IR-QLA not only surpasses traditional algorithms like Ant Colony and Particle Swarm Optimization in terms of convergence speed but also minimizes the path length and iteration counts needed for trajectory optimization. Future studies will explore alternative reward sources and advanced algorithms to further enhance the precision and speed of UAV trajectory planning, aiming to establish new benchmarks in the field.

Multivariate time series (MTS) forecasting is a rapidly expanding field with diverse and futuristic applications. However, traditional statistical learning models need more prediction accuracy when faced with dynamic variability, non-linearity, and non-stationarity, as well as the challenge of selecting MTS data for classification. Moreover, the existing methods for early classification of multivariate time series data suffer from numerous severe challenges, including evaluating the length of testing the MTS data component, which must be equal to the training MTS data component, and the availability of faulty data components in MTS. To address this issue, we propose a novel framework for early MTS forecasting using double Q-learning-based ensemble techniques to improve prediction accuracy. Our framework uses Q-learning agents to select optimal actions, which results in maximum rewards and accurate prediction. We investigate the ensemble behavior of learned agents using double Q-learning and Gaussian Process Classifiers (GPC) for early forecasting of MTS data. We also determine the minimum required time-series length for classifying faulty data components using the probabilistic Auto-Regressive Integrated Moving Average (ARIMA) model, enhancing framework robustness and mitigating miss-classification accuracy. Our proposed framework achieves 99.89% accuracy for early forecasting, surpassing existing methods based on different benchmark settings and publicly available multivariate time-series datasets. The framework provides a promising solution to the challenges of accurate MTS forecasting and offers insights into the early prediction of recent stock market trading data.

The underwater wireless sensor network (UWSNs) is an important communication facility supporting underwater monitoring applications. However, the transmission channel has the characteristics of high bit error rate, strong multipath effect, and many interference factors, and the network node has the characteristics of high energy consumption, difficult energy supply, and the node position vulnerable to change, which makes it extremely difficult for UWSNs to realize the reliable and efficient packet forwarding. To address the problem, we propose the Stackelberg Q-learning based multi-hop cooperative routing algorithm (SQMCR). The SQMCR builds the transmission routes based on the Q-learning algorithm, considering factors such as the delay, the remaining energy, and the network topology, which improves the rationality and adaptability of selecting the next-hop node. By balancing the packet forwarding benefits and the energy consumption costs based on the Stackelberg Q-learning algorithm, the SQMCR establishes the cooperative communication policy to ensure both the reliability and efficiency of underwater communications. It also adopts initializing Q-values and dynamic exploration probabilities optimization methods to further improve the performance of routing algorithms. Experimental results show that the SQMCR can help UWSNs increase the packet forwarding reliability and prolong the network lifetime by 17%. It has a better environment and application adaptability and is more suitable for underwater high-reliability applications.

During the process of waste collection, various unpredicted disturbances might occur. Meanwhile, vehicle transport is an important source of carbon emissions. In this study, an energy-efficient multi-trip dynamic vehicle routing model is established for waste collection, which introduces two types of dynamic events: new collecting requirements of waste sites and vehicle breakdowns. A Q-learning-based hyperheuristic particle swarm optimization (QLHPSO) is proposed as a dynamic rescheduling method to solve the model. A set of low-level heuristics (LLHs) are designed by combining the learning operators in particle swarm optimization and local search operators. A Q-learning-based high-level strategy is developed to find a suitable LLH for each evolutionary state based on historical performance of LLHs. When rescheduling is triggered, a response mechanism is incorporated to construct initial population by utilizing the features of dynamic events and historical elites. Extensive experimental results on one real instance and nine synthetic instances show that QLHPSO can react to the environmental changes rapidly, and reschedule the vehicle routes with lower cost and carbon emissions compared to the state-of-the-art algorithms.

Students are a gold asset for each country. Proper guidance/recommendation to the students regarding their education-related issues can ultimately result in uplifting the economy of a country. Different education models are followed in the world, out of which Outcome Based Education (OBE) is the one. OBE education model comprises three main components, which include Program Educational Objectives (PEOs), Program Learning Outcomes (PLOs), and Course Learning Outcomes (CLOs). CLOs are outcomes that a student achieves after studying a course. A single course may contain one or more CLOs. These CLOs are then mapped to PLOs and PLOs are then mapped to PEOs. Therefore, our objective in this work is to improve deficient/weak CLOs of students by suggesting online resources. Whereas, in the absence of this proposed system, students have to find out these resources by themselves or course teacher recommends relevant online resources. To achieve this objective, we created a dataset for the OBE education model, as to date no standard dataset exists on the OBE education model. From this dataset, we created a Student-to-CLO matrix and performed bi-clustering on this matrix to find groups of students having similar performance in different CLOs. So far, Bi-clustering has been used in the Bio-informatics field to determine similarity in gene expression data. Generated bi-clusters are sorted according to their homogeneity of contained values. These sorted bi-clusters are then mapped to a 2D grid to formulate a reinforcement-learning environment. The start state of the recommendation agent is determined using cosine similarity. If an agent visits a state, deficient CLOs of that state are recommended to the student. The agent can visit only those states that are nearby to its current state and accessible through its legal action space. An optimal sequence of actions to visit different states of a 2D grid, which can improve student’s performance, is determined using Q-learning. Online resources including research articles, YouTube videos, books, and online tutorials are suggested to the student to improve his deficient CLOs using a mobile app.

The burgeoning growth of vehicular traffic, fuelled by rapid urbanization and an ever-expanding population, has resulted in congested road networks. Traditional and sensor-based adaptive traffic light management systems have shown commendable progress in some scenarios, but they suffer from inherent disadvantages that hinder their effectiveness and scalability. To combat these challenges, a dynamic and adaptable traffic control system is imperative to optimize traffic flow. The present paper explores the limitations of sensor-based adaptive traffic light management and advocates for integrating a novel algorithm to overcome the existing drawbacks. It proposes an intelligent traffic management algorithm called IntelliSignal that leverages the map service for fetching real-time traffic information to calculate the optimal green time and a penalty-based road selection to optimize traffic flow. The proposed IntelliSignal is designed to provide equal chances for all roads while prioritizing higher-density roads with more green time, effectively mitigating traffic congestion and improving overall transportation efficiency. It incorporates Q-learning, a reinforcement learning technique that enables the system to adapt and learn from traffic patterns. The proposed IntelliSignal’s performance is assessed through rigorous evaluations conducted on the simulation platform SUMO. The acquired results demonstrate substantial enhancements across various crucial metrics, including average waiting time, vehicle density, travel time, CO2 emissions, and queue length. Furthermore, the simulation results demonstrate that the proposed IntelliSignal algorithm exhibits a remarkable 30.52% increment in system throughput compared to the traditional approach. This significant enhancement underscores the efficacy of the proposed IntelliSignal in optimizing system performance and merits consideration for practical implementation.

An increasing number of users are expected to be served by wireless network with heterogeneous requirements. Unmanned aerial vehicles (UAV) can be deployed to augment the heterogeneous network from aerial area by taking advantages of the characteristics of UAV such as mobility, manoeuvrability, low cost, and Line-of-Sight (LoS) communication. However, the deployment of UAV can also cause problems. For example, in UAV-assisted HetNet, the number of handover (HO) will increase because of the dense distribution of small base stations (SBS) and UAV base stations (UBS). Also, because of the small coverage of SBS and the LoS communication links from neighbour UBSs, the number of unnecessary HO will also rise. Frequent HO and unnecessary HO can result in interruption, increased overhead and energy consumption, which is not desirable for battery powered UAVs. In this paper, to solve the problem, a HO decision making algorithm adopting TOPSIS and Q-learning (QL) is proposed with the aim to reduce HO number and improve energy efficiency. Q-learning can be applied to address decision-making challenges in communication systems widely. However, a large volume of training data can pose challenges and complexities, therefore, the TOPSIS is utilised to reduce the size of the action space in Q-learning. The proposed hybrid TOPSIS-Q-learning method enhances both the handover performance and the scalability. In the method, signal to interference and noise ratio (SINR), time of stay (ToS) and average energy efficiency (EE) are taken into account. The simulation results show that the number of HO and unnecessary HO is remarkably reduced and the average EE is notably improved in comparison with other existing methods.

Exploration is an integral part of learning dynamics which allows algorithms to search a space of solutions. When many algorithms simultaneously explore, this can lead to counter-intuitive effects. This paper contributes an analysis of the influence that exploration has on a multi-agent system of $Q$ -learners in a famous congestion dilemma, the Braess paradox. I find ranges of the exploration rate for which $\epsilon $ -greedy $Q$ -learners show chaotic and oscillatory dynamics which do not converge, and yield better than Nash equilibrium results. I decouple the dynamics endogenous to $Q$ -learning from the exogenous exploration rate $\epsilon $ , and find that $Q$ -learners implicitly coordinate with low exploration rates $\epsilon \in (0, 0.1)$ , but are disrupted in their coordination for larger exploration rates $\epsilon > 0.1$ . The best implicit coordination leads to a 20% reduction in average travel times which approaches the social optimum. I discuss how our results may inform multi-agent algorithm design, fit within a cognitive science perspective of cognitive noise during learning, and provide a mechanistic hypothesis for the lack of empirical evidence of the Braess Paradox in traffic systems.

Various methods are employed in computer vision applications to identify individuals, including using face recognition as a human visual feature helpful in tracking or searching for a person. However, tracking systems that rely solely on facial information encounter limitations, particularly when faced with occlusions, blurred images, or faces oriented away from the camera. Under these conditions, the system struggles to achieve accurate tracking-based face recognition. Therefore, this research addresses this issue by fusing descriptions of the face visual with body visual features. When the system cannot find the target face, the CNN+LSTM hybrid method assists in multi-feature body visual recognition, narrowing the search space and speeding up the search process. The results indicate that the combination of the CNN+LSTM method yields higher accuracy, recall, precision, and F1 scores (reaching 89.20%, 87.36%, 91.02%, and 88.43%, respectively) compared to the single CNN method (reaching 88.84%, 74.00%, 67.00%, and 69.00% respectively). However, the combination of these two visual features requires high computation. Thus, it is necessary to add a tracking system to reduce the computational load and predict the location. Furthermore, this research utilizes the Q-Learning algorithm to make optimal decisions in automatically tracking objects in dynamic environments. The system considers factors such as face and body visual features, object location, and environmental conditions to make the best decisions, aiming to enhance tracking efficiency and accuracy. Based on the conducted experiments, it is concluded that the system can adjust its actions in response to environmental changes with better outcomes. It achieves an accuracy rate of 91.5% and an average of 50 fps in five different videos, as well as a video benchmark dataset with an accuracy of 84% and an average error of 11.15 pixels. Utilizing the proposed method speeds up the search process and optimizes tracking decisions, saving time and computational resources.

Traffic congestion has become an increasingly severe problem and needs to be solved urgently. Reinforcement learning (RL) is an advanced data-driven approach for large-scale adaptive traffic signal control (ATSC) in the complex urban traffic network. The decentralized multi-agent RL (MARL) framework is feasible for multi-intersection scenarios. However, the environment becomes partially observable and nonstationary without effective communication among agents from the perspective of each local agent. Previous studies solve this problem by improving the observation of each local agent and using a spatial discount factor. In this paper, we are motivated by achieving coordination among intersections in the RL settings and better performance in ATSC and solve the problem of traffic signal control at multiple intersections in large-scale networks based on RL techniques. Pressure is used as an indicator to represent the traffic conditions of an intersection considering its neighbor intersections. A new reward function with pressure and vehicle waiting time is designed to achieve coordination among intersections. The above three control policies are innovatively applied to a multi-agent Q-learning method for ATSC at multiple intersections. Experiments are carried out on a large synthetic traffic grid under simulated peak-hour traffic dynamics. Results demonstrate that our approach outperforms the state-of-the-art decentralized MARL algorithms in average queue length and intersection delay.

Unmanned aerial vehicles (UAVs) are receiving increasing attention due to their wide range of applications. Flying ad-hoc networks (FANETs) enable inter-communication between UAVs, but their dynamics and fast changes in mobility and topology make it challenging to achieve stable routing. Clustering based FANETs provide a self-organizing approach for routing, but clustering is a decentralized process based on different utilities that can affect performance. In this work, we propose a centralized agent placed in a remote ground station to adjust the role of each utility in the cluster-head arbitration. Our approach uses six utilities, including node centrality, residual energy, link holding time, velocity similarity, buffer occupancy, and diversity. The developed reinforcement learning (RL)-clustering approach enables routing over multiple planes, with each plane representing a network cell with a different coverage radius of nodes, including Femto, Pico, and Micro planes. We propose a novel reward formulation for effective learning of agents that includes indicators of stability based on role change, energy consumption, and confirmation message through the selected node. For evaluation, the developed RL based agent was evaluated on three types of agents, namely, Q-learning, DQN, and DDPG and it was compared with random agent. Results have shown the superiority of our developed RL based clustering in terms of stability, energy consumption and most network metrics.

The current application of control theory is commonly carried out in systems with a model or known system dynamics. However, in practice this is a formidable task to achieve as not all state information can be known. The use of the Output Feedback (OPFB) scheme in the field of control systems also possesses a weakness because it requires the use of an observer. This appears rather contradictory as the use of an observer requires system dynamics information. This research proposes an optimal control scheme using Deep Recurrent Q-Networks (DRQN) to generate an optimal control signal trajectory based on a collection of input and output data from the system itself. The approach proposed in this study is based on the Q-Learning method from the Reinforcement Learning (RL) scheme. The Long-Short Term Memory (LSTM) is used to approximate the Q-function and determine the control signals for a system without a known model. The method that we proposed in this study has been tested on four case studies. The control signal trajectory generated from our proposed algorithm, is much smaller than the control signal that generated from classical Q-Learning scheme. The results of this research are certainly relevant to the aim of OPFB, namely that the controller is designed to be able to regulate (bring the state trajectory to zero) and minimize control signal energy. It is empirically discovered that the same result is proven by the norm values resulting from the Q-function trajectory. The norm of Q-function trajectory for our proposed algorithm on the 1st, 2nd, 3rd, and 4th case studies are 2.11E-08, 3.15E-06, 3.79E-09, and 1.59E-13, respectively.

Fluid resuscitation (therapy) is used to maintain tissue perfusion and restore cardiac functions in critical care. Automated fluid therapy can result in faster care, fewer dosing errors, and less cognitive burden on healthcare providers, ultimately improving patient outcomes. Despite a few attempts at automating this process, fluid management is an open research area for which optimal, personalized strategies are yet to be developed. This work presents a novel, model-free, subject-specific dose adjustment tool for fluid resuscitation. The proposed approach is based on reinforcement learning (RL) where a Q-learning algorithm automatically recommends subject-specific fluid infusion dosages in different hemorrhaging scenarios without having the knowledge of dose-response models. Comparison studies against two model-free fluid resuscitation controllers, i.e., fuzzy and proportional-integral-derivative (PID), within a verified simulated environment demonstrated the superior performance of the proposed approach in the closed-loop control of fluid resuscitation. Statistical analyses of performance measures indicated that the RL approach, with lower average resuscitation rates, can achieve more desired mean arterial pressure (MAP) responses than the fuzzy and PID controller for all virtual subjects. Additionally, simulation results demonstrated the higher robustness of our approach than the other two methods against external disturbances in resuscitation scenarios. These results confirm the potential of RL in the closed-loop control of hemodynamic responses in fluid therapy.

Multi-agent reinforcement learning (MARL) for cooperative tasks has been extensively researched over the past decade. The prevalent framework for MARL algorithms is centralized training and decentralized execution. Q-learning is often employed as a centralized learner. However, it requires finding the maximum value by comparing the Q-value of each joint action a’ in the next state s’ to update the Q-value of the last visited state-action pair (s,a). When the joint action space is extensive, the maximization operation involving comparisons becomes time-consuming and becomes the dominant computational burden of the algorithm. To tackle this issue, we propose an algorithm to reduce the number of comparisons by saving the joint actions with the top 2 Q-values (T2Q). Updating the top 2 Q-values involves seven cases, and the T2Q algorithm can avoid traversing the Q-table to update the Q-value in five of these seven cases, thus alleviating the computational burden. Theoretical analysis demonstrates that the upper bound of the expected ratio of comparisons between T2Q and Q-learning decreases as the number of agents increases. Simulation results from two-stage stochastic games are consistent with the theoretical analysis. Furthermore, the effectiveness of the T2Q algorithm is validated through the distributed sensor network task and the target transportation task. The T2Q algorithm successfully completes both tasks with a 100% success rate and minimal computational overhead.

Value-based Reinforcement Learning algorithms achieve superior performance by utilizing experiences gathered in the past to update their so-called value-function. In most cases, it is accomplished by applying a sampling strategy to an experience buffer, in which state transitions are stored during the training process. However, the design of such methods is not so intuitive. General theoretic approaches tend to determine the expected learning progress from each experience, based on which the update of neural networks can be carried out efficiently. Proper choice of these methods can not only accelerate, but also stabilize the training significantly by increasing sampling efficiency, which indirectly leads to a reduction in time and computing capacity requirements. As one of the most critical aspects of using Machine Learning (ML) based techniques originates from the lack of decent computing power, thus endeavour to find optimal solutions has long been a researched topic in the field of Reinforcement Learning. Therefore the main focus of this research has been to develop an experience prioritization method acquiring competitive performance, besides having the overall cost of training considerably lowered. In this paper, we propose a novel priority value assignment concept for experience prioritization in Reinforcement Learning, based on the Upper Confidence Bound algorithm. Furthermore, we present empirical findings of our solution, that it outperforms current state-of-the-art in terms of sampling efficiency, while enabling faster and more cost-efficient training processes.

The SAS-CBRS framework is being tested to share the federally held spectrum with licensed users and opportunistic users to maximize the underutilized spectrum’s utility and overcome spectrum scarcity. In the SAS-CBRS framework, radio resources are assigned to the incumbent access (IA), primary access licensees (PAL), and general authorized access (GAA) users according to the given priority. The SAS-CBRS three-tier framework is different from the conventional cognitive radio networks (CRN) as it involves a central entity that acts as a server called a spectrum access system (SAS). The methods to assign the resources using the SAS are still in the research phase. Yet, no standard method is defined by the FCC for resource allocation. The current CRN methods cannot be directly applied because of the addition of the third tier and a central server. Moreover, strict rules are defined for using the 3.5 GHz spectrum band for communication. In this paper, a novel DDRQ-SAS algorithm integrated with the double auction (DA) algorithm is proposed that uses deep recurrent double Q-learning. The DDRQ-SAS is used by the SAS to hold a spectrum auction and create a spectrum pool to get information on PAL channels. PAL operators use the DA algorithm to generate the asking prices intelligently for their available idle channels and the GAA users will use the DA algorithm to intelligently bid for their preferred channels. The DDRQ-SAS-DA algorithm allows the GAA users to get the guaranteed QoS offered by the PAL operators in an auction. GAA users maintain the preference list of the PAL reserved idle channels and bid intelligently based on the available QoS. SAS completes the transaction by allocating the channels to the winning GAAs. The defined problem is also modeled using the double auction multi-winner multi-channel technique and the TDSA-PS algorithm. Numerical results show that the proposed DDRQ-SAS-DA algorithm provides up to 20% better QoS at higher loads for GAA users, generates 24% more revenue for PAL operators, and is 1.6 times more efficient in assigning 500 GAA users.

As the adoption of cryptocurrencies, especially Bitcoin (BTC) continues to rise in today’s digital economy, understanding their unpredictable nature becomes increasingly critical. This research paper addresses this need by investigating the volatile nature of the cryptocurrency market, mainly focusing on Bitcoin trend prediction utilizing on-chain data and whale-alert tweets. By employing a Q-learning algorithm, a type of reinforcement learning, we analyze variables such as transaction volume, network activity, and significant Bitcoin transactions highlighted in whale-alert tweets. Our findings indicate that the algorithm effectively predicts Bitcoin trends when integrating on-chain and Twitter data. Consequently, this study offers valuable insights that could potentially guide investors in informed Bitcoin investment decisions, thereby playing a pivotal role in the realm of cryptocurrency risk management.

Dynamic pricing for preemptible cloud services (DPPCS) is highly demanded to effectively utilize the excess capacity in cloud computing. However, the dynamic nature of excess capacity exhibits high non-stationarity, which is characterized by multi-temporal stochastic patterns with time-varying statistical properties. The non-stationarity results in the DPPCS problem being a Non-Stationary Markov Decision Process (NSMDP) with unknown transition probabilities. Moreover, DPPCS is constrained by a certain maximum preemption rate, further complicating the DPPCS problem as a Constrained NSMDP (CNSMDP). We transform the CNSMDP into a piecewise Lagrangian dual model, which converts the CNSMDP into an unconstrained optimization problem. To solve the above problem, we propose a novel Q-Learning approach for DPPCS. We first present estimation methods for the unknown environment parameters, including a detection method for identifying temporal pattern changes, and a diffusion approximation method for estimating the actual preemption rate. Then, we introduce a Lagrange multiplier updating method, which can strike a balance between revenue and the preemption rate in the reward function. Building upon the above methods, we develop a Constrained Non-Stationary Q-Learning (CNSQL) algorithm for DPPCS, which dynamically adjusts its learning process to adapt to the multi-temporal patterns. Through simulated experiments, we demonstrate the effectiveness of our proposed approach compared to state-of-the-art algorithms. It performs well in improving revenue generated from excess capacity while maintaining the actual preemption rate within the specified constraint.

Learning activities are considerably supported and improved by the rapid advancement of e-learning systems. This gives students a tremendous chance to participate in learning activities worldwide. The Massive Open Online Courses (MOOCs) platform has emerged as one of the most significant platforms for e-learning as a result of the rapid growth of network information technologies. Due to the increased number of online courses available, it is harder for individual learners to choose the appropriate courses, activities, and learning paths for the actual necessities they want, which reduces their learning performances. Moreover, a sequential Recommender System (RS) can identify the learner’s future interest and suggest the subsequent item or learning content given a sequence of past interactions. This is in contrast to interactive recommendation methods that can create recommendations based on the learner’s feedback via constant interactions. To address these challenges, the goal of this paper is to propose a Reinforcement Learning (RL) based smart e-learning framework with Markov Decision Process (MDP) that has the potential to enhance the learning experience for each student by providing them with a personalized and effective learning path. Applying the MDP and RL-based techniques such as Q-learning for Sequential Path Recommendation (SPR) and learning development is more achievable. This is because the MDP allows for adjusting the recommendations method to find new activities and learning paths based on the learners’ feedback on recommendations results. Experimental findings reveal that the suggested model obtains significant improvements and provides viable performance under different parameters optimization. Furthermore, we also show that the proposed method outperforms a long session (long-term rewards such that they maximize learning progress while minimizing frustration and disengagement). This demonstrates the model’s improvements in simulating the learner’s sequential behavior, learning activities, various learning materials, and learning paths simultaneously. These promising initial results provide a possible solution to assess this challenge further in future work.

The ever-increasing and diverse user demands as well as the need for uninterrupted access to the medium with minimum latency in dense machine type communication networks, are the key driving forces to a holistic network redesign. In this context, fifth-generation and beyond (5G/B5G) networks, incorporate various advanced physical layer techniques, such as relaying-assisted transmission, aiming to improve network performance hl and extend the coverage area of multicellular orientations. However, the deployment of such techniques in a cellular environment characterized by high interference levels and multi-variate channel representations, leads to increased computational complexity for radio resource management (RRM) tasks. Machine learning (ML), and especially Deep Learning (DL), is proposed as an efficient way to support end-to-end user applications in highly complex environments, since ML/DL models can relax the R hl RM-associated computational burden. In this paper, we consider the joint problem of relay node (RN) placement and selection subject to subcarrier allocation and power management constraints in 5G/B5G hl networks. Various DL-based methods are examined and combined to solve both sub-problems. The performance of these schemes is evaluated for various relaying-assisted transmission approaches, hl either considering known channel state information (CSI) or not. According to the derived results, total system energy efficiency (EE) and spectral efficiency (SE) can be improved by up to 30%, when considering only the DL-based RN placement scheme compared to state-of-the-art non-ML schemes. The deployment of the reinforcement learning (RL) model for RN selection, can improve EE up to 80%, while SE can be improved up to 75%, compared to a system with only DL-enabled RN placement.

The combination of model predictive control (MPC) and learning methods has been gaining increasing attention as a tool to control systems that may be difficult to model. Using MPC as a function approximator in reinforcement learning (RL) is one approach to reduce the reliance on accurate models. RL is dependent on exploration to learn, and currently, simple heuristics based on random perturbations are most common. This paper considers variance-based exploration in RL geared towards using MPC as function approximator. We propose to use a non-probabilistic measure of uncertainty of the value function approximator in value-based RL methods. Uncertainty is measured by a variance estimate based on inverse distance weighting (IDW). The IDW framework is computationally cheap to evaluate and therefore well-suited in an online setting, using already sampled state transitions and rewards. The gradient of the variance estimate is then used to perturb the policy parameters in a direction where the variance of the value function estimate is increasing. The proposed method is verified on two simulation examples, considering both linear and nonlinear system dynamics, and compared to standard exploration methods using random perturbations.

Ultra-dense heterogeneous networks (UDHN) based on small cells are a requisite part of the future cellular networks as they are proposed as one of the enabling technologies to handle coverage and capacity problems. But co-tier and cross-tier interferences in UDHN severely degrade the quality of service due to K-tiered architecture. Machine learning based radio resource management either through independent learning or cooperative learning is a proven efficient scheme for interference mitigation and quality of service provision in UDHN in a both distributive and cooperative manner. However, an optimal learning paradigm selection, i.e., either independent or cooperative learning and optimal cooperative cluster size in cooperative learning for efficient radio resource management in UDHN is still an open research problem. In this article, a Q-learning based radio resource management scheme is proposed and evaluated for both distributive and cooperative schemes using independent and cooperative learning. The proposed Q-learning solution follows the $\epsilon -$ greedy policy for optimal convergence. The simulation results for the UDHN in an urban setup show that in comparison to the independent learning paradigm, cooperative learning has no significant impact on macro cell user capacity. However, there is a significant improvement in small cell user capacity and the sum capacity of the cooperating small cells in the cluster. A significant increase of 48.57% and 37.9% is observed in the small cell user capacity, and sum capacity of the cooperating small cells, respectively, using cooperative learning as compared to independent learning which sets cooperative learning as an optimal learning strategy in UDHN. The improvement in small cell user capacity is at cost of increased computational time which is directly proportional to the number of cooperating small cells. To solve the issue of computational time in cooperative learning, an optimal clustering algorithm is proposed. The proposed optimal clustering reduced the computational time by four times in cooperative Q-learning.

The 6TiSCH (IPv6 over IEEE802.15.4e time-slotted channel hopping mode) wireless sensor network architecture utilizes control packets to construct network formation. These control packets are essential for establishing communication links between nodes and configuring network settings. The trickle timer algorithm is utilized to broadcast the DIO control packet. DIO carries information about the available parent nodes, which is then used to form the routing tree. Sensors transmit control packets in one cell on each TSCH slotframe, called the minimal cell. This leads to the problem that RPL trickle timer algorithm encounters congestion in DIO control packet transmission with other control messages, particularly in dense networks. Moreover, high traffic transmission also leads to high queue usage, which then drops the DIO control packet. Failed DIO transmission can increase network formation time and energy consumption. To address this issue, we propose Q-Trickle, an adaptive trickle timer algorithm based on Q-learning that determines the optimal policy for transmitting or suppressing DIO based on minimal cell and transmission queue conditions. Q-Trickle adaptively selects a redundancy constant value and transmission interval that promotes fair transmission distribution and considers network condition. Additionally, a control scheme over minimal cell transmission is formulated to lower transmission congestion and faster synchronization. The proposed methods were assessed using simulation and actual testing on the FIT IoT-LAB testbed. The results indicated that Q-Trickle performed better than the benchmark methods. Q-Trickle decreases joining time, energy consumption, and number of failed DIO compared with the original algorithm by −13%, −11%, and −43%, respectively.

Global path planning techniques have been widely employed in solving path planning problems, however they have been found to be unsuitable for unknown environments. Contrarily, the traditional Q-learning method, which is a common reinforcement learning approach for local path planning, is unable to complete the task for multiple targets. To address these limitations, this paper proposes a modified Q-learning method, called Vector Field Histogram based Q-learning (VFH-QL) utilized the VFH information in state space representation and reward function, based on a 2D LiDAR sensor. We compared the performance of our proposed method with the classical Q-learning method (CQL) through training experiments that were conducted in a simulated environment with a size of 400 square pixels, representing a 20-meter square map. The environment contained static obstacles and a single mobile robot. Two experiments were conducted: experiment A involved path planning for a single target, while experiment B involved path planning for multiple targets. The results of experiment A showed that VFH-QL method had 87.06% less training time and 99.98% better obstacle avoidance compared to CQL. In experiment B, VFH-QL method was found to have an average training time that was 95.69% less than that of the CQL method and 83.99% better path quality. The VFH-QL method was then evaluated using a benchmark dataset. The results indicated that the VFH-QL exhibited superior path quality, with efficiency of 94.89% and improvements of 96.91% and 96.69% over CQL and SARSA in the task of path planning for multiple targets in unknown environments.

Planning precise manipulation in robotics to perform grasp and release-related operations, while interacting with humans is a challenging problem. Reinforcement learning (RL) has the potential to make robots attain this capability. In this paper, we propose an affordance-based human-robot interaction (HRI) framework, aiming to reduce the action space size that would considerably impede the exploration efficiency of the agent. The framework is based on a new algorithm called Contextual Q-learning (CQL). We first show that the proposed algorithm trains in a reduced amount of time (2.7 seconds) and reaches an 84% of success rate. This suits the robot’s learning efficiency to observe the current scenario configuration and learn to solve it. Then, we empirically validate the framework for implementation in HRI real-world scenarios. During the HRI, the robot uses semantic information from the state and the optimal policy of the last training step to search for relevant changes in the environment that may trigger the generation of a new policy.

In recent years, unmanned aerial vehicles (UAVs) have gained significant popularity and are used for many applications, from entertainment to surveillance and the modern battlefield. As regulation demands arose worldwide, controling and reacting to unauthorized flights of UAVs became a pressing issue. In this work, we present an algorithm to accelerate the training of a reinforcement learning drone agent for a counter unmanned aerial system (C-UAS). The main objective of this C-UAS is to guide an invading drone to a safe-killing zone (SZ) using a hunter quadrotor drone. The hunter quadrotor launches a spoofing, or meaconing, attack on the GNSS receiver of the invading drone. The proposed algorithms employ an abstraction of the C-UAS problem to accelerate the training step and enable training during the mission. Results for different SZ radii are discussed using a software-in-the-loop simulation for ground truth based on a detailed model of the UAV embedded system and flight dynamics, including error metrics and action time. We show that a 99% probability of successful target steering to the SZ can be achieved considering a SZ radius of 75 meters and a Q-table trained with the proposed accelerated training model.

This paper proposes a FAst paRAllel and pipeliNE Q-learning accelerator (FARANE-Q) for a configurable Reinforcement Learning (RL) algorithm implemented in a System on Chip (SoC). The proposed work offers flexibility, configurability, and scalability while maintaining computation speed and accuracy to overcome the challenges of a dynamic environment and increasing complexity. The proposed method includes a Hardware/Software (HW/SW) design methodology for the SoC architecture to achieve flexibility. We also propose joint optimizations on the algorithm, architecture, and implementation to obtain optimum (high efficiency) performance, specifically in energy and area efficiency. Furthermore, we implemented the proposed design in a real-time Zynq Ultra96-V2 FPGA platform to evaluate the functionality with an actual use case of smart navigation. Experimental results confirm that the proposed accelerator FARANE-Q outperforms state-of-the-art works by achieving a throughput of up to 148.55 MSps. It corresponds to the energy efficiency of 1747.64 MSps/W per agent for 32-bit and 2424.33 MSps/W per agent for 16-bit FARANE-Q. Moreover, the proposed 16-bit FARANE-Q outperforms other related works by an improvement of at least $1.23\times $ in energy efficiency. The designed system also maintains an error accuracy of less than 0.4% with optimized bit precision for more than eight fraction bits. The proposed FARANE-Q also offers a speed up of processing time up to $1795\times $ compared to embedded SW computation executed on ARM Zynq processor and $280\times $ of computation of full software executed on i7 processor. Hence, the proposed work has the potential to be used for smart navigation, robotic control, and predictive maintenance.

Investigation of value based Reinforcement Learning algorithms exhibited a resurgence into mainstream research in 2015 following demonstration of super-human performance when applied to Atari 2600 games. Since then, significant media attention and hype have accompanied this area, and the field of Artificial Intelligence generally, spread across distinct categories. This review paper is focused exclusively on the progression of value based Reinforcement Learning in the last five years. We aim to distill the incremental improvements to stability and performance in this period, highlighting the minimal changes to the base algorithm over this time. This holds true with all but the one exception of the Recurrent Experience Replay in Distributed Reinforcement Learning algorithm, representing a fundamental shift and increase in agent performance through an advanced memory representation. We suggest a new focus area for value based Reinforcement Learning research.

Autonomous trajectory planning is a hot topic in the UAV mission planning area of research. Autonomous UAVs have major use case applications which involve navigation in complex environments such as aerial photography, package delivery and relief operations. Many existing trajectory planning solutions rely on the GPS system. However, such GPS-based solutions do not provide a reliable real-time navigation solution, particularly in dense urban environments. Opportunely, cellular networks can be utilized as an attractive alternative for UAV navigation applications. We therefore propose to utilize existing 5G infrastructure to enable the UAV to navigate complex environments, independent of the GPS and other detectable signals transmitted by the UAV. Our objective is to propose an efficient solution to enable the UAVs to autonomously execute such tasks while meeting the real-time operational requirements, without the need to actually interact with the cellular network. For this purpose, we formulate the UAV trajectory planning problem as a joint objective optimization problem to minimize a composite cost metric that we introduce. The computational complexity involved in exact optimization techniques hinders obtaining the real-time calculation requirement that is needed due to the dynamic nature of the environment. To overcome this complexity, we utilize machine learning based techniques to solve the formulated trajectory planning problem. Specifically, we propose two machine learning-based techniques, namely, the reinforcement learning and the deep supervised learning-based approaches. We then analyze the performance of each of the proposed techniques as compared to the optimization-based approaches and other solutions from the literature. Our simulation results show that the proposed reinforcement and deep supervised learning-based solutions provide near optimal solutions to the formulated trajectory planning problem, with comparable accuracy of 99% and 98%, respectively, as compared to the optimal bound while meeting the real-time calculation requirement.

In this paper, we address the chance-constrained safe Reinforcement Learning (RL) problem using the function approximators based on Stochastic Model Predictive Control (SMPC) and Distributionally Robust Model Predictive Control (DRMPC). We use Conditional Value at Risk (CVaR) to measure the probability of constraint violation and safety. In order to provide a safe policy by construction, we first propose using parameterized nonlinear DRMPC at each time step. DRMPC optimizes a finite-horizon cost function subject to the worst-case constraint violation in an ambiguity set. We use a statistical ball around the empirical distribution with a radius measured by the Wasserstein metric as the ambiguity set. Unlike the sample average approximation SMPC, DRMPC provides a probabilistic guarantee of the out-of-sample risk and requires lower samples from the disturbance. Then the Q-learning method is used to optimize the parameters in the DRMPC to achieve the best closed-loop performance. Wheeled Mobile Robot (WMR) path planning with obstacle avoidance will be considered to illustrate the efficiency of the proposed method.

In recent years, several studies have been conducted on the dynamic control of traffic signal durations using deep reinforcement learning with the aim of reducing traffic congestion. The unique advantages of independent control of traffic signals include reduction in the cost of information transmission and stable control without being affected by the failure of other traffic signals. However, conventional deep reinforcement learning methods such as Deep Q-Network may degrade the learning performance in a multi-agent environment where there are multiple traffic signals in the environment. Therefore, we propose a traffic light control system based on the dual targeting algorithm, which incorporates reinforcement of successful experiences in multi-agent environments, with the aim of realizing a better traffic light control system. The experimental results in a multi-agent environment using a traffic flow simulator based on simulation of urban mobility (SUMO) show that the proposed traffic light control system reduces the waiting time at traffic lights by 33% compared to a conventional traffic light control system using deep reinforcement learning. In the future works, we aim to apply this research to traffic light control systems in real environments.

The concept of the “connected car” offers the potential for safer, more enjoyable and more efficient driving and eventually autonomous driving. However, in urban Vehicular Networks (VNs), the high mobility of vehicles along roads poses major challenges to the routing protocols needed for a reliable and flexible vehicular communications system. Thus, urban VNs rely on static Road-Side-Units (RSUs) to forward data and to extend coverage across the network. In this paper, we first propose a new Q-learning-based routing algorithm, namely Infrastructure-aided Traffic-Aware Routing (I-TAR), which leverages the static wired RSU infrastructure for packet forwarding. Then, we focus on the multi-source, multi-destination problem and the effect this imposes on node availability, as nodes also participate in other communications paths. This motivates our new hybrid approach, namely Hybrid Infrastructure-aided Traffic Aware Routing (HI-TAR) that aims to select the best Vehicle-to-Vehicle/Infrastructure (V2V/I) route. Our findings demonstrate that I-TAR can achieve up to 19% higher average packet-delivery-ratio (APDR) compared to the state-of-the-art. Under a more realistic scenario, where node availability is considered, a decline of up to 51% in APDR performance is observed, whereas the proposed HI-TAR in turn can increase the APDR performance by up to 50% compared to both I-TAR and the state-of-the-art. Finally, when multiple source-destination vehicle pairs are considered, all the schemes that model and consider node availability, i.e. limited-availability, achieve from 72.2% to 82.3% lower APDR, when compared to those that do not, i.e. assuming full-availability. However, HI-TAR still provides 34.6% better APDR performance than I-TAR, and ~40% more than the state-of-the-art.

Unmanned Aerial Vehicles (UAVs) have emerged as a flexible and cost-effective solution for remote monitoring of the vital signs of patients in large-scale Internet of Medical Things (IoMT) Wireless Body Area Networks (WBANs). This paper deals with the problem of using UAVs for real-time scheduling of the transmission of vital signs in delay-sensitive IoMT WBANs. The main challenge for such a network is to timely and reliably transmit the vital signs of patients to the remote monitoring center without interrupting their daily lifestyles. To achieve this goal, we propose a $Q$ -learning-based algorithm to optimize the trajectory of each UAV, as the mobile Base Station (BS), to harvest vital signs of patients in outdoor applications, especially in unreachable areas. In this algorithm, UAVs learn to reach the best 3D position by discovering the network environment step-by-step. It stands for the position in which the covered patients by each UAV have the highest transmission rate, the least delay and energy consumption. Moreover, we employ the Non-Orthogonal Multiple Access (NOMA) technique to simultaneously schedule multiple transmissions by accepting a degree of interference between them in order to enhance the spectrum efficiency of the network. Eventually, the performance of our proposed scheme is evaluated via extensive simulations in terms of throughput, energy consumption, and delay. The simulation results show that our proposed scheme iteratively converges to the benchmark value of the mentioned factors by increasing the information of cluster environment through episodes.

Bitcoin price prediction based on people’s opinions on Twitter usually requires millions of tweets, using different text mining techniques, and developing a machine learning model to perform the prediction. These attempts lead to the employment of a significant amount of computer power, central processing unit (CPU) utilization, random-access memory (RAM) usage, and time. To address this issue, in this paper, we consider a classification of tweet attributes that effects on price changes and computer resource usage levels while obtaining an accurate price prediction. To classify tweet attributes having a high effect on price movement, we collect all Bitcoin-related tweets posted in a certain period and divide them into four categories based on the following tweet attributes: $(i)$ the number of followers of the tweet poster, $(ii)$ the number of comments on the tweet, $(iii)$ the number of likes, and $(iv)$ the number of retweets. We separately train and test by using the Q-learning model with the above four categorized sets of tweets and find the best accurate prediction among them. We compare our approach with a classic approach where all Bitcoin-related tweets are used as input data for the model, by analyzing the CPU workloads, RAM usage, memory, time, and prediction accuracy. The results show that tweets posted by users with the most followers have the most influence on a future price, and their utilization leads to spending 80% less time, 88.8% less CPU consumption, and 12.5% more accurate predictions compared with the classic approach.

This paper presents the computation of feasible paths for mobile robots in known and unknown environments using a QAPF learning algorithm. Q-learning is a reinforcement learning algorithm that has increased in popularity in mobile robot path planning in recent times, due to its self-learning capability without requiring a priori model of the environment. However, Q-learning shows slow convergence to the optimal solution, notwithstanding such an advantage. To address this limitation, the concept of partially guided Q-learning is employed wherein, the artificial potential field (APF) method is utilized to improve the classical Q-learning approach. Therefore, the proposed QAPF learning algorithm for path planning can enhance learning speed and improve final performance using the combination of Q-learning and the APF method. Criteria used to measure planning effectiveness include path length, path smoothness, and learning time. Experiments demonstrate that the QAPF algorithm successfully achieves better learning values that outperform the classical Q-learning approach in all the test environments presented in terms of the criteria mentioned above in offline and online path planning modes. The QAPF learning algorithm reached an improvement of 18.83% in path length for the online mode, an improvement of 169.75% in path smoothness for the offline mode, and an improvement of 74.84% in training time over the classical approach.

Wireless mesh networks are popular due to their adaptability, easy-setup, flexibility, cost, and transmission time-reductions. The routing algorithm plays a vital role in transferring the data between the nodes. The network’s performance is significantly impacted by the route opted by the algorithm. The router takes the decision to send the packet to the next router as per the policy of that algorithm. So even though that decision does not favor the right path selection, the router tends to follow its policy. This can be avoided by having intelligent routers that can make routing decisions on the fly. This paper presents the QL-Feed Forward routing algorithm (QFFR), a new generation of routing algorithms that combines reinforcement learning based on the Q-learning algorithm with a Feed Forward neural network. This algorithm (QFFR) can learn from the network environment and make routing decisions based on the algorithm’s learnings. The AI agent’s ability to select the fastest path, which enhances the efficiency of the routing operation, is demonstrated by the working of the suggested QFFR algorithm. This paper also evaluates the performance of traditional algorithms, namely, Ad-hoc On-Demand Distance-Vector, Optimized-Link-State-routing, Destination-Sequenced Distance-Vector and Distance Source routing. The evaluation parameters include throughput, packet delivery ratio, and delay. The parameters are the outcomes of the time the information takes to reach from source to destination. This analysis highlights the improvement in the routing decision ability of a router. As per analysis, Ad hoc On-Demand Distance Vector Algorithm outperforms with throughput 723.13 Kbps, delay 343.73 ns. Q-learning agent identifies the route and reaches the destination in average of 3.7s in non-grid architecture. The Q-learning agent takes 0.49sec with a grid size ten by ten and 0.53sec in three by four grid size. The suggested QFFR takes 7.62s score-over time with stable, consistent performance.

Workload balancing in cloud computing is still challenging problem, especially in Infrastructure as a Service (IaaS) in the cloud model. A problem that should not occur during cloud access is a host or server being overloaded or underloaded, which may affect the processing time or may result in a system crash. Therefore, to prevent these problems, an appropriate schedule of access should be considered so that the system can distribute tasks across all available resources, which is called load balancing. The load balancing technique should ensure that all Virtual Machines (VMs) are used appropriately. In this paper, an independent task scheduling approach in cloud computing is proposed using a Multi-objective task scheduling optimization based on the Artificial Bee Colony Algorithm (ABC) with a Q-learning algorithm,which is a reinforcement learning technique that helps the ABC algorithm work faster, called the MOABCQ method. The proposed method aims to optimize scheduling and resource utilization, maximize VM throughput, and create load balancing between VMs based on makespan, cost, and resource utilization, which are limitations of concurrent considerations. Performance analysis of the proposed method was compared using CloudSim with the existing load balancing and scheduling algorithms: Max-Min, FCFS, HABC_LJF, Q-learning, MOPSO, and MOCS algorithms in three datasets: Random, Google Cloud Jobs (GoCJ), and Synthetic workload. The experimental results indicated that the algorithms used MOABCQ approach outperformed the other algorithms in terms of reducing makespan, reducing cost, reducing degree of imbalance, increasing throughput and average resource utilization.

The network-assisted full-duplex (NAFD) system realizes flexible duplex in the spatial domain within the same time-frequency resource. With the explosive growth of the number of users and remote antenna units (RAUs) under 6G scenario, the resource utilization of the system is lower. When the resource of users is selected by the RAUs to send or receive, collisions or congestion may occur due to mechanisms such as grant-free. Aiming at making better use of system resources, a load-aware dynamic mode selection scheme with NAFD scheme is proposed to improve the access efficiency and resource utility of the system. This paper first propose a centralized Q-learning algorithm which determines a clever strategy to approach the ultimate goal by itself and excels in environment dynamics. However, the size of the Q-table used in the centralized Q-learning algorithm for storage is huge. Further, a distributed multi-agent Q-learning algorithm is proposed which has a smaller size of Q-table and lower complexity to suit for actual scenarios. The simulation results showed that the proposed load-aware dynamic mode selection scheme can significantly improve resource utility and throughput performance than other traditional schemes.

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

Spectrum access system (SAS) is a three-tier layered spectrum sharing architecture proposed by the Federal Communications Commission (FCC) for Citizens Broadband Radio Service (CBRS) 3.5 GHz band. The available 150 MHz spectrum is dynamically shared among Incumbent Access (IA), Primary Access Licensees (PAL) and General Authorized Access (GAA) users. IA users are the highest priority federal military users, PAL users are the licensed users and the GAA users are the least priority unlicensed users. In this scenario, PAL operators are willing to give access to their idle spectrum to GAA users to generate extra revenue. SAS will ensure to protect IA users and PAL users from interference caused by lower-tier users. It is the responsibility of SAS to allocate resources to GAA users but the method to do so is left open. In this article, a novel auction algorithm based on Q-learning for dynamic spectrum access (SAS-QLA) is proposed. In SAS-QLA, multiple GAA users dynamically and intelligently bid using Q-learning to access PAL reserved idle channels. SAS will decide to allocate the channels to GAA users with maximum bidding offers. GAA users have their own quality of service (QoS) demands i.e., transmission rate, packet loss, bidding efficiency, and maintain the preference of available PAL reserved idle channels based on Q-learning considering the available QoS. The proposed scenario is also modeled as a knapsack NP-hard problem and solved using dynamic programming and distributed relaxation method. Numerical results demonstrate the effectiveness of the SAS-QLA algorithm in improving the bidding efficiency, maximizing the data rate per unit cost and spectrum utilization.

A major amount of the energy of battery-powered sensors is spent during packet transmissions. This issue has led to the development of power-control-based multiple-access collision avoidance (MACA) protocols that can reduce the packet transmission power and conserve energy. However, the reduction in transmission power renders the packets susceptible to collisions. To reduce these collisions while maintaining high energy efficiency, we propose a power control protocol that utilizes reinforcement learning to choose the optimal transmission power. The total reward is determined by the occurrence of a collision, amount of transmission power used, frequency of DATA packet retransmissions, and update of the interference range. A key feature of the proposed protocol is that it enables sensors to prevent collisions without any prior knowledge of interferences, thus eliminating the need for additional signaling. Simulation results under varying average traffic loads indicate that the proposed protocol can improve network throughput by up to 20% compared to benchmark protocols, while minimizing network energy consumption with a similar gain and reducing collisions per packet by more than 35%. These results demonstrate that the proposed protocol is effective.

This paper suggests and implements how to apply the reinforcement learning on delayed reward system which is known to be complex to apply the machine learning technology such as Q-learning. Such games as Tetris game is known to be a delayed reward system because of its characteristics of generating sparse reward in learning process. Tetris game requires the actor’s quick judgment ability and speed of response because the blocks must be stacked in an optimal location quickly, considering the random shape and rotation of appearing blocks. Also, since the number of cases is very large due to the various block types and order, if a human-being is playing the game, the performance is limited by simply relying on human memorization capability. Therefore, we applied a reinforcement learning implemented in this study for this delayed reward system. We find that the general legacy reinforcement learning method shows its limitation in improving the performance. Hence, we apply the heuristic to increase the decision accuracy as the weighting method of reward. As a result, we were able to obtain high scores in games. Although it is not yet possible to say that this heuristic(rule-based) approach has completely conquered the game. In several experiments, this hybrid reinforcement learning shows better playability than human in terms of learning speed, as well as high scores. In this paper, it is shown that general Q-learning is not suitable for delayed reward system. And a hybrid learning that adds prioritized experience replay tactics, and the related techniques and algorithms are introduced to increase the reinforcement learning performance.

Machine-type communications (MTC) should account for half the connections to the internet by 2030. The use case massive MTC (mMTC) allows for applications to connect a massive number of low-power and low-complexity devices, leading to challenges in resource allocation. Not only that, mMTC networks suffer under rigid random access schemes due to mMTC ultra-dense nature resulting in poor performance. In this sense, this paper proposes a $Q$ -Learning-based random access method for massive machine-type communications, with device clustering and non-orthogonal multiple access (NOMA). The traditional NOMA implementation increases spectral efficiency, but at the same time, demands a larger $Q$ -Table, thus slowing down convergence, which is known to be a highly detrimental effect on massive networks. We use pre-clustering through short-range device-to-device technology to mitigate this drawback, allowing devices to operate with a smaller $Q$ -Table. Furthermore, the previous selection of partner devices allows us to implement a full-feedback-based reward mechanism so that clusters avoid time slots already successfully allocated. Additionally, to cope with the negative impact of system overload, we propose an adaptive frame size algorithm to run in the base station (BS). It allows adjusting the frame size to the network load, preventing idle slots in an underloaded scenario, and providing extra slots when the network is overloaded. The results show the great benefits in terms of throughput of the proposed method. In addition, the impact of the use of clustering and the size of the clusters, as well as the frame size adaptation, are analyzed.

This paper addresses the problem of navigating decentralized multi-agent systems in partially cluttered environments and proposes a new machine-learning-based approach to solve it. On the basis of this approach, a new robust and flexible Q-learning-based model is proposed to handle a continuous space problem. As in reinforcement learning (RL) algorithms, Q-learning does not require a model of the environment. Additionally, Q-Learning (QL) has the advantages of being fast and easy to design. However, one disadvantage of QL is that it needs a massive amount of memory, and it grows exponentially with each extra feature introduced to the state space. In this research, we introduce an agent-level decentralized collision avoidance low-cost model for solving a continuous space problem in partially cluttered environments, followed by introducing a method to merge non-overlapping QL features in order to reduce its size significantly by about 70% and make it possible to solve more complicated scenarios with the same memory size. Additionally, another method is proposed for minimizing the sensory data that is used by the controller. A combination of these methods is able to handle swarm navigation low memory cost with at least18 number of robots. These methods can also be adapted for deep q-learning architectures so as to increase their approximation performance and also decrease their learning time process. Experiments reveal that the proposed method also achieves a high degree of accuracy for multi-agent systems in complex scenarios.

The energy consumption of the routing protocol can affect the lifetime of a wireless sensor network (WSN) because tiny sensor nodes are usually difficult to recharge after they are deployed. Generally, to save energy, data aggregation is used to minimize and/or eliminate data redundancy at each node and reduce the amount of the overall data transmitted in a WSN. Furthermore, energy-efficient routing is widely used to determine the optimal path from the source to the destination, while avoiding the energy-short nodes, to save energy for relaying the sensed data. In most conventional approaches, data aggregation and routing path selection are considered separately. In this study, we consider the degrees of the possible data aggregation of neighbor nodes when a node needs to determine the routing path. We propose a novel Q-learning-based data-aggregation-aware energy-efficient routing algorithm. The proposed algorithm uses reinforcement learning to maximize the rewards, defined in terms of the efficiency of the sensor-type-dependent data aggregation, communication energy and node residual energy, at each sensor node to obtain an optimal path. We used sensor-type-dependent aggregation rewards. Finally, we performed simulations to evaluate the performance of the proposed routing method and compared it with that of the conventional energy-aware routing algorithms. Our results indicate that the proposed protocol can successfully reduce the amount of data and extend the lifetime of the WSN.