Search

This paper presents a least-mean-square-based (LMS-based) background calibration algorithm with reinforcement learning optimization to calibrate the capacitor mismatch in successive approximation-register (SAR) analog-to-digital converters (ADCs). When calibrating capacitor mismatch, the convergence speed and stability of the conventional LMS-based algorithm cannot be balanced due to the fixed iteration step size. To improve the calibration effect, nine different iteration step sizes are proposed for calibration. An iteration step size selection strategy is obtained by training the Q-learning algorithm with the conventional LMS-based calibration iteration process. Leading by this strategy, the proposed calibration algorithm performs better convergence speed and stability than the conventional LMS-based calibration algorithm. A 12-bit sub-radix-2 redundant SAR ADC is used to test our proposed calibration algorithm for better evaluation. Behavioral simulations show that the proposed calibration algorithm converges at 0.065 M sampling points, and the standard deviation of the effective number of bits (ENOB) in the convergence state is only 0.034-bit. [ABSTRACT FROM AUTHOR]

Summary: In this article, we propose an adaptive Kalman filtering based on Q‐learning for partial model‐free dynamic systems. First, a cost function is defined to iteratively update the prior state value when the model parameters are unknown. Then, the observations in a period of time are utilized to improve the accuracy and updating speed of the prior state estimation by means of the multi‐innovation least squares. Next, considering that the weight matrix in the cost function will change due to external noise noise and model mismatch, the innovation‐based adaptive estimation algorithm is presented to adjust the weight matrix by using the covariance of the information sequence. Finally, the proposed algorithms are applied to estimate the water level of a quadruple water tank system. [ABSTRACT FROM AUTHOR]

I characterize the outcomes of a class of model-free reinforcement learning algorithms, such as stateless Q-learning, in a prisoner's dilemma. The behavior is studied in the limit as players stop experimenting after sufficiently exploring their options. A closed form relationship between the learning rate and game payoffs reveals whether the players will learn to cooperate or defect. The findings have implications for algorithmic collusion and also apply to asymmetric learners with different experimentation rules. [ABSTRACT FROM AUTHOR]

Unmanned Aerial Vehicle (UAV) networks, also known as drone networks, have gained significant attention for their potential in various applications, including communication. UAV networks for communication involve using a fleet of drones to establish wireless connectivity and provide communication services in areas where traditional infrastructure is lacking or disrupted. UAV communication networks need to be highly secured to ensure the technology's security and the users' safety. The proposed survey provides a comprehensive overview of the current state-of-the-art UAV network security solutions. In this paper, we analyze the existing literature on UAV security and identify the various types of attacks and the underlying vulnerabilities they exploit. Detailed mitigation techniques and countermeasures for the protection of UAVs are described in this paper. The survey focuses on the implementation of novel technologies like Q-learning, blockchain, IRS, and mmWave. This paper discusses network simulation tools that range in complexity, features, and programming capabilities. Finally, future research directions and challenges are highlighted. [ABSTRACT FROM AUTHOR]

Intelligent traffic control requires accurate estimation of the road states and incorporation of adaptive or dynamically adjusted intelligent algorithms for making the decision. In this article, these issues are handled by proposing a novel framework for traffic control using vehicular communications and Internet of Things data. The framework integrates Kalman filtering and Q-learning. Unlike smoothing Kalman filtering, our data fusion Kalman filter incorporates a process-aware model which makes it superior in terms of the prediction error. Unlike traditional Q-learning, our Q-learning algorithm enables adaptive state quantization by changing the threshold of separating low traffic from high traffic on the road according to the maximum number of vehicles in the junction roads. For evaluation, the model has been simulated on a single intersection consisting of four roads: east, west, north, and south. A comparison of the developed adaptive quantized Q-learning (AQQL) framework with state-of-the-art and greedy approaches shows the superiority of AQQL with an improvement percentage in terms of the released number of vehicles of AQQL is 5% over the greedy approach and 340% over the state-of-the-art approach. Hence, AQQL provides an effective traffic control that can be applied in today's intelligent traffic system. [ABSTRACT FROM AUTHOR]

In this study, a traffic queue model is employed to simulate traffic parameters for optimization purposes. The primary objective is to enhance the average queue length, cumulative delay, and cumulative reward for varying numbers of vehicles generated using an innovative approach. The proposed methodology integrates deep Reinforcement Q-Learning techniques implemented through a realistic simulator, Simulation of Urban MObility, based on OpenStreetMap. This simulator is used to balance the flow of traffic by monitoring traffic conditions and adjusting signals (Red/Yellow/Green) accordingly. The research extensively evaluates simulation outcomes across different scenarios involving 50, 100, 500, and 1000 vehicles, with episodes 10, 50 and 100. The results provide valuable insights into training times, simulation durations, and total rewards obtained, indicating improvements in mitigating traffic congestion. The introduction of a traffic control strategy enhances model effectiveness through the utilization of four-layer deep neural network architectures with 400 neurons in hidden layers. However, the study also highlights persistent challenges related to real-time implementation and complexities in state-action-agent relationships. The progress highlights improvements in reducing congestion and enhancing efficiency at junctions by dynamically adjusting signal phases while leveraging historical data. The results highlight the potential of dynamic traffic management strategies in effectively addressing urban traffic challenges. [ABSTRACT FROM AUTHOR]

This paper investigates the output feedback (OPFB) tracking control problem for discrete-time linear (DTL) systems with unknown dynamics. To solve this problem, we use an augmented system approach, which first transforms the tracking control problem into a regulation problem with a discounted performance function. The solution to this problem is derived using a Bellman equation, based on the Q-function. In order to overcome the challenges of unmeasurable system state variables, we employ a multistep Q-learning algorithm that surpasses the advantages of the policy iteration (PI) and value iteration (VI) techniques and state reconstruction methods for output feedback control. As such, the requirement for an initial stabilizing control policy for the PI method is removed and the convergence speed of the learning algorithm is improved. Finally, we demonstrate the effectiveness of the proposed scheme using a simulation example. [ABSTRACT FROM AUTHOR]

Research on dynamic treatment regimes has enticed extensive interest. Many methods have been proposed in the literature, which, however, are vulnerable to the presence of misclassification in covariates. In particular, although Q‐learning has received considerable attention, its applicability to data with misclassified covariates is unclear. In this article, we investigate how ignoring misclassification in binary covariates can impact the determination of optimal decision rules in randomized treatment settings, and demonstrate its deleterious effects on Q‐learning through empirical studies. We present two correction methods to address misclassification effects on Q‐learning. Numerical studies reveal that misclassification in covariates induces non‐negligible estimation bias and that the correction methods successfully ameliorate bias in parameter estimation. [ABSTRACT FROM AUTHOR]

StarCraft is a popular real-time strategy game that has been widely used as a research platform for artificial intelligence. Micromanagement refers to the process of making each unit perform appropriate actions separately, depending on the current state in the the multi-agent system comprising all of the units, i.e., the fine-grained control of individual units for common benefit. Therefore, cooperation between different units is crucially important to improve the joint strategy. We have selected multi-agent deep reinforcement learning to tackle the problem of micromanagement. In this paper, we propose a method for learning cooperative strategies in StarCraft based on role-based montonic value function factorization (RoMIX). RoMIX learns roles based on the potential impact of each agent on the multi-agent task; it then represents the action value of a role in a mixed way based on monotonic value function factorization. The final value is calculated by accumulating the action value of all roles. The role-based learning improves the cooperation between agents on the team, allowing them to learn the joint strategy more quickly and efficiently. In addition, RoMIX can also reduce storage resources to a certain extent. Experiments show that RoMIX can not only solve easy tasks, but it can also learn better cooperation strategies for more complex and difficult tasks. [ABSTRACT FROM AUTHOR]

Vulnerabilities such as SQL injection represent a serious challenge to security. While tools with a pre-defined logic are commonly used in the field of penetration testing, the continually evolving nature of the security challenge calls for models able to learn autonomously from experience. In this paper we build on previous results on the development of reinforcement learning models devised to exploit specific forms of SQL injection, and we design agents that are able to tackle a varied range of SQL injection vulnerabilities, virtually comprising all the archetypes normally considered by experts. We show that our agents, trained on a synthetic environment, perform a transfer of learning among the different SQL injections challenges; in particular, they learn to use their queries to efficiently gain knowledge about multiple vulnerabilities at once. We also introduce a novel and more versatile way to interpret server messages that reduces reliance on expert inputs. Our simulations show the feasibility of our approach which easily deals with a number of homogeneous challenges, as well as some of its limitations when presented with problems having higher degrees of uncertainty. [ABSTRACT FROM AUTHOR]

In accordance with the actual production circumstances of enterprises, a scheduling problem model is designed for open-shop environments, considering AGV transport time. A Q-learning-based method is proposed for the resolution of such problems. Based on the characteristics of the problem, a hybrid encoding approach combining process encoding and AGV encoding is applied. Three pairs of actions are constituted to form the action space. Decay factors and a greedy strategy are utilized to perturb the decision-making of the intelligent agent, preventing it from falling into local optima while simultaneously facilitating extensive exploration of the solution space. Finally, the proposed method proved to be effective in solving the open-shop scheduling problem considering AGV transport time through multiple comparative experiments. [ABSTRACT FROM AUTHOR]

Some protocols operated in the MAC layer and the open-source interconnections model to share the packet delivery and the network channel to deliver the packet is done. Transmitting two or more messages through one channel causes problems in packing and information delivery. This causes the lagging of the data to reach the destinations. So, the method of multiple hubs and the host must be created for the packet and the information delivery. This makes the fast transmission of data and messages. Method: The Tunnel monitoring system and the FANET are used for analysis in this study for the research, enabling the mobile radio networks for the examination. The FANET-based flying network system allows the paradigm for accessing human analysis. Then the drone-related data can be helped. The tunnel monitoring system navigates the information purpose and routing of the drone flying and gathering the data. Then based on human analysis, 40% of the data can be analyzed using the system's formation. Result: These results support the coverage and the tunnel monitoring process for detecting the navigation system. Conclusion: Also, some tunnel-based detection using drones is found in this study based on the wireless muometric navigation system that can be enabled using Q-learning. [ABSTRACT FROM AUTHOR]

Intelligent load scheduling for the smart city is essential due to the growth of industrialization, which directly consumes energy from the grid. The energy management system in industrialization is enhanced using the Internet of things in the same domain. The general need in a smart city is to maintain the energy utilization and cost that the industrial Internet of things could achieve for industry application. Researchers have also found a solution to this problem, but every time the generator cannot sell energy, the industry purchases the same. There is a chance for the industry to supply its energy to other industries or the grid. By keeping this constraint, an intelligent approach to load scheduling based on optimized Q-Net is proposed in this paper, comprising the supplies as renewable energy, battery, and industrial energy. Also, the reactive power flow of smart cities during scheduling is mitigated by classifying the energy available in the bus network. The proposed method scheduled demand power with 99% accuracy with the lowest cost of 13.7041$/KW for peak hr. scheduling with a demand of 6000KW/hr. [ABSTRACT FROM AUTHOR]

Copyright of Telecommunications Science is the property of Beijing Xintong Media Co., Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The internet of things (IoT) underscores pivotal real-world applications ranging from security systems to smart infrastructure and traffic management. However, contemporary IoT devices grapple with significant challenges pertaining to battery longevity and energy efficiency, constraining the assurance of prolonged network lifetimes and expansive sensor coverage. Many existing solutions, although promising on paper, are intricate and often impractical for real-world implementations. Addressing this gap, we introduce an energy-efficient routing protocol leveraging reinforcement learning (RL) tailored for wireless sensor networks (WSNs). This protocol harnesses RL to discern the optimal transmission route from the source to the sink node, factoring in the energy profile of each intermediary node. Training of the RL algorithm is facilitated through a reward function that includes energy outflow and data transmission efficacy. The model was compared against two prevalent routing protocols, LEACH and fuzzy C-means (FCM), for a comprehensive assessment. Simulation results highlight our protocol's superiority with respect to the active node count, energy conservation, network longevity, and data delivery efficiency. [ABSTRACT FROM AUTHOR]

Autonomous optical navigation is one of the most promising techniques to estimate the position and velocity of a spacecraft in Earth orbit without the supports of Earth-based tracking stations. To improve the navigation performance, this paper presents a novel autonomous optical navigation method, where a star camera on the spacecraft is utilized to measure the line-of-sight (LOS) directions of a number of non-cooperative space targets, whose position vectors are supposed to be not precisely known in advance, and an improved Q-learning based extended Kalman filter (IQEKF) is developed to obtain the accurate motion state estimate of both the spacecraft and the space targets based on the LOS direction measurements. The main advantage of the presented method is that the LOS directions of the space targets can be acquired with high-accuracy by using the state-of-the-art star camera, such that the superior navigation accuracy is achievable. In addition, the whole motion state of the spacecraft, such as position, velocity, and attitude, can be obtained with the star camera, in the case that the space targets and the stars are observed simultaneously. The high performance of the presented autonomous navigation method is illustrated through a representative simulation of a medium Earth orbit (MEO) satellite. Furthermore, the simulation results indicate that the IQEKF yields more accurate solutions than the traditional navigation filtering algorithms. [ABSTRACT FROM AUTHOR]

Blackjack is a classic casino game in which the player attempts to outsmart the dealer by drawing a combination of cards with face values that add up to just under or equal to 21 but are more incredible than the hand of the dealer he manages to come up with. This study considers a simplified variation of blackjack, which has a dealer and plays no active role after the first two draws. A different game regime will be modeled for everyone to ten multiples of the conventional 52-card deck. Irrespective of the number of standard decks utilized, the game is played as a randomized discrete-time process. For determining the optimum course of action in terms of policy, we teach an agent-a decision maker-to optimize across the decision space of the game, considering the procedure as a finite Markov decision chain. To choose the most effective course of action, we mainly research Monte Carlobased reinforcement learning approaches and compare them with q-learning, dynamic programming, and temporal difference. The performance of the distinct model-free policy iteration techniques is presented in this study, framing the game as a reinforcement learning problem. [ABSTRACT FROM AUTHOR]

The Internet of Things (IoT) is a network of various interconnected objects capable of collecting and exchanging data without human interaction. These objects have limited processing power, storage space, memory, bandwidth and energy. Therefore, due to these limitations, data transmission and routing are challenging issues where data collection and analysis methods are essential. The Routing Protocol for Low-power and Lossy Networks (RPL) is one of the best alternatives to ensure routing in LoWPAN6 networks. However, RPL lacks scalability and basically designed for non-dynamic devices. Another drawback of the RPL protocol is the lack of load balancing support, leading to unfair distribution of traffic in the network that may decrease network efficiency. This study proposes a novel RPL-based routing protocol, QFS-RPL, using Q-learning algorithm policy and ideation from the Fisheye State Routing protocol. We have developed an algorithm for ease of data transfer in the IoT, which provides better performance than existing protocols, especially when dealing with a mobile network. To evaluate the performance of the proposed method, the Contiki operating system and Cooja simulator have been used in scenarios with mobile and stationary nodes and random network topologies. The results have been compared with RPL and mRPL. We have developed an algorithm for ease of data transfer in the IoT, which provides better performance than existing protocols, especially when dealing with a mobile network. The simulation outputs revealed that our scheme performs more efficiently in load balancing, number of table entries, Packet Delivery Ratio (PDR), End-to-End (E2E) latency, network throughput, convergence speed, control packet overhead and Remaining Useful Lifetime in designed scenarios compared to other methods. Moreover, the simulation results show an out-performance of rival schemes in terms of remaining energy and network lifetime. [ABSTRACT FROM AUTHOR]

Copyright of Tunnel Construction / Suidao Jianshe (Zhong-Yingwen Ban) is the property of Tunnel Construction Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Social network analysis is a rapidly evolving research area having several real-life application areas, e.g. digital marketing, epidemiology, spread of misinformation. Influence maximization aims to select a subset of nodes in such manner that the information propagated over the network is maximized. Competitive influence maximization, which describes the phenomena of multiple actors competing for resources within the same infrastructure, can be solved with a greedy approach selecting the seed nodes utilizing the influence strength between nodes. Recently, deep reinforcement learning methods were applied for estimating the influence strength. We train a controller with reinforcement learning for selecting a node list of given length as the initial seed set for the information spread. Our experiments show that deep Q-learning methods are suitable to analyze the competitive influence maximization on trust and distrust based social networks. Received by the editors: 1 March 2023. 2010 Mathematics Subject Classification. 68T05. 1998 CR Categories and Descriptors. G.2.2 Discrete Mathematics: Graph Theory – Network problems; G.3. Probability and Statistics: Markov Processes; I.2.6 Artificial Intelligence: Learning – Connectionism and neural nets.

Genetic markers for acne are being studied to create personalized treatments based on an individual’s genes, and the field is benefiting from the application of artificial intelligence (AI) techniques. One such AI tool, the Q-learning algorithm, is increasingly being utilized by medical researchers to delve into the genetics of acne. In contrast to previous methods, our research introduces a Q-learning model that is adaptable to diverse sample groups. This innovative approach involves preprocessing data by identifying differentially expressed genes and constructing gene-gene connectivity networks. The key advantage of using the Q-learning model lies in its ability to transform acne gene data into Markovian domains, which are essential for selecting relevant genetic markers. Performance evaluations of our Q-learning model have shown high accuracy and specificity, although there may be some sensitivity variations. Notably, this research has identified specific genes, such as CD86, AGPAT3, TMPRSS11D, DSG3, TNFRSF1B, PI3, C5AR1, and KRT16, as being acne-related through biological verification and text data mining. These findings underscore the potential of AI-driven Q-learning models to revolutionize the study of acne genetics. In conclusion, our Q-learning model offers a promising approach for the selection of acne-related genetic markers, despite minor sensitivity fluctuations. This research highlights the transformative potential of Q-learning in advancing our understanding of the genetics underlying acne, paving the way for more personalized and effective treatments in the future.

The countries have experienced the tremendous potential of unmanned aerial vehicles and their military counterparts in recent years. For further improving the task performances of these autonomous vehicles, their flight paths should be determined or calculated optimally by taking into account enemy weapon systems, fuel or battery usage and some limitations about the turning, climbing or diving angles. Immune Plasma algorithm (IP algorithm or IPA) is the first intelligent optimization technique modeling the details of an infection treatment method called convalescent or immune plasma gained popularity again with the coronavirus disease and showed its promising performance for various engineering problems. In this study, Q-learning that is a reinforcement learning algorithm was integrated into the workflow of the IPA for managing some pandemic measures including lockdown, partial opening and full opening. Moreover, the treatment schema was completely changed in order to improve the search efficiency and remove the requirement of algorithm specific control parameters. The newly introduced IPA variant also named Q-learning IPA (Q-LIPA) was tested with the purpose of planning paths and a set of detailed experiments was carried out over twelve test cases of three different battlefield scenarios. The paths found by Q-LIPA were compared with the paths of well-known intelligent optimization techniques and their modifications. Comparative studies indicated that both Q-learning based pandemic measure management and specialized treatment schema positively contribute to the solving performance and help Q-LIPA to outperform its rivals for the majority of the test cases.

Q-learning has long been one of the most popular reinforcement learning algorithms, and theoretical analysis of Q-learning has been an active research topic for decades. Although researches on asymptotic convergence analysis of Q-learning have a long tradition, non-asymptotic convergence has only recently come under active study. The main goal of this paper is to investigate a new finite-time analysis of asynchronous Q-learning under Markovian observation models via a control system viewpoint. In particular, we introduce a discrete-time time-varying switching system model of Q-learning with diminishing step-sizes for our analysis, which significantly improves recent development of the switching system analysis with constant step-sizes, and leads to $\mathcal {O}(\sqrt {{\log k}/{k}})$ convergence rate that is comparable to or better than most of the state of the art results in the literature. In the meanwhile, we consider the continuous-time Lyapunov equation to avoid the difficulty in the analysis posed by using diminishing step-sizes in discrete-time. The proposed analysis brings in additional insights, covers different scenarios, and provides new simplified templates for analysis to deepen our understanding on Q-learning via its unique connection to discrete-time switching systems.

Current multi-objective reinforcement learning (MORL) research often struggles to balance multiple objectives and manage the stability and performance of learning algorithms, especially in complex environments. To address this, we propose a new multi-objective deep Q-network (MO-DQN) framework that integrates linear scalarization in MORL. This framework has the following features: First, we provide immediate feedback by incorporating linear scalarization into reward processing. Compared with some complex multi-objective optimization methods, this approach is relatively easy to understand and implement, offering greater convenience for practical applications. Additionally, linear scalarization accelerates the learning process and enhances the algorithm’s ability to dynamically adjust strategies. Second, we develop the Linear Scalarized Multi-objective Deep Q-Network (LSMO-DQN) under different reward mechanisms, improving MO-DQN’s ability to balance multiple objectives effectively. Immediate reward strategies accelerate learning and enable rapid adjustments, benefiting dynamic environments. In contrast, delayed reward strategies help understand long-term action impacts and promote strategic decision-making. Experiments are conducted in two different multi-objective environments, where our proposed method slightly outperforms other techniques, indicating its robustness and adaptability. Specifically, LSMO-DQN achieves higher cumulative rewards and demonstrates improved stability across various reward structures. The findings suggest that integrating linear scalarization in reward processing not only enhances learning performance but also provides a more straightforward approach to managing the trade-offs in multi-objective settings. These results highlight the potential of LSMO-DQN to improve learning performance, particularly in scenarios with data imbalances.

Maintaining player engagement in serious management games is a challenge due to the repetitive nature of traditional predetermined difficulty levels. A dynamic difficulty adjustment (DDA) system is introduced in this study to address this issue by integrating fuzzy logic and Q-learning. Player ennui is frequently the consequence of static difficulty adjustments. In order to dynamically adjust game complexity in accordance with player performance and preferences, our DDA system utilizes a diverse array of performance metrics, adaptive narrative elements, and real-time feedback, as well as fuzzy logic and Q-learning algorithms. According to empirical assessments, players were 28% more effective overall, and play sessions lasted an average of 35% longer. Player satisfaction and involvement were also much improved. Customers played the game longer and were less bored because of the higher degree of difficulty and customization. The integration of fuzzy logic and Q-learning in DDA systems greatly enhances the ability to maintain long-term player engagement in essential management games. This approach offers a long-lasting alternative for creating constantly captivating gaming experiences by effectively reducing the repetitiveness of traditional difficulty adjustments.

In distributed Software-defined networking (SDN), control plane functions are partitioned across multiple controller instances to enhance fault tolerance and scalability. However, the dynamic nature of network traffic and rapid network events, such as link failures and controller node failures, can lead to uneven workload distribution among controller nodes. This research aims to adjust switch-to-controller mapping to address load imbalance dynamically. We model flow arrivals at switches and subsequent actions within a Markov decision process (MDP) framework. In MDP, precise knowledge of the arrival rate is required, however, such an assumption is impractical in dynamic environments. Reinforcement learning (RL) learns policies from environment interactions, enabling autonomous decision-making in complex domains by adeptly navigating uncertainties. The proposed scheme uses RL to monitor SDN flow dynamics and maintain system load balance through switch migration. Herein, the proposed scheme generates migration triplets specifying the source controller, the destination controller for migration, and the switch to be migrated. The scheme considers the cost of migrating the flows in terms of the flow arrival rate and hop count between the switch and the controllers. Experimental results confirm that the framework effectively achieves load balancing across different network topologies and diverse traffic load distributions on switches.

This paper investigates the secrecy performance of a multi-relay non-orthogonal multiple access (NOMA) network. Considering the presence of eavesdroppers, an optimal relay selection scheme and a jamming signal transmission scheme for both amplify-and-forward (AF) and decode-and-forward (DF) strategies are proposed. However, the resource allocation problem aimed at maximizing the effective secure throughput (EST) of the system is non-convex. It is difficult to directly solve this optimization problem using conventional methods. As such, a Q-learning approach to solve the resource allocation problem in this system is applied, and an innovative reward function that can maximize the communication quality of edge users while ensuring secure communication of nearby users is designed. According to the analysis of the simulation results, the convergence of the proposed scheme is verified. Under the same conditions, DF relays achieve a higher signal-to-noise ratio (SINR) at the user terminal, and the EST is closely related to the transmission power of the source node and the relays. The numerical results also show that compared to conventional power allocation methods, the proposed method achieves a larger average EST and provides better confidentiality performance.

Trajectory planning for unmanned aerial vehicles (UAVs) presents a formidable challenge, particularly in environments where details about the terrain and objectives are not known beforehand. This uncertainty has propelled the adoption of UAVs across diverse sectors such as logistics, weather forecasting, security surveillance, and autonomous driving, where their ability to navigate autonomously is crucial. Recent advancements have leveraged deep learning for robust object detection and tracking, significantly enhancing data acquisition from aerial imagery and UAV-based applications. Despite numerous innovations, the critical issue of optimizing UAV trajectories in unknown environments remains largely unaddressed. This paper introduces a novel approach, the Instructed Reinforcement Q-Learning Algorithm (IR-QLA), which utilizes Received Signal Strength (RSS) as a dynamic reward metric to refine trajectory planning continuously. By integrating a unique instructed reinforcement principle, this method significantly accelerates the learning process, improving the UAVs’ navigational efficiencies. Our findings demonstrate that IR-QLA not only surpasses traditional algorithms like Ant Colony and Particle Swarm Optimization in terms of convergence speed but also minimizes the path length and iteration counts needed for trajectory optimization. Future studies will explore alternative reward sources and advanced algorithms to further enhance the precision and speed of UAV trajectory planning, aiming to establish new benchmarks in the field.

Multivariate time series (MTS) forecasting is a rapidly expanding field with diverse and futuristic applications. However, traditional statistical learning models need more prediction accuracy when faced with dynamic variability, non-linearity, and non-stationarity, as well as the challenge of selecting MTS data for classification. Moreover, the existing methods for early classification of multivariate time series data suffer from numerous severe challenges, including evaluating the length of testing the MTS data component, which must be equal to the training MTS data component, and the availability of faulty data components in MTS. To address this issue, we propose a novel framework for early MTS forecasting using double Q-learning-based ensemble techniques to improve prediction accuracy. Our framework uses Q-learning agents to select optimal actions, which results in maximum rewards and accurate prediction. We investigate the ensemble behavior of learned agents using double Q-learning and Gaussian Process Classifiers (GPC) for early forecasting of MTS data. We also determine the minimum required time-series length for classifying faulty data components using the probabilistic Auto-Regressive Integrated Moving Average (ARIMA) model, enhancing framework robustness and mitigating miss-classification accuracy. Our proposed framework achieves 99.89% accuracy for early forecasting, surpassing existing methods based on different benchmark settings and publicly available multivariate time-series datasets. The framework provides a promising solution to the challenges of accurate MTS forecasting and offers insights into the early prediction of recent stock market trading data.

The underwater wireless sensor network (UWSNs) is an important communication facility supporting underwater monitoring applications. However, the transmission channel has the characteristics of high bit error rate, strong multipath effect, and many interference factors, and the network node has the characteristics of high energy consumption, difficult energy supply, and the node position vulnerable to change, which makes it extremely difficult for UWSNs to realize the reliable and efficient packet forwarding. To address the problem, we propose the Stackelberg Q-learning based multi-hop cooperative routing algorithm (SQMCR). The SQMCR builds the transmission routes based on the Q-learning algorithm, considering factors such as the delay, the remaining energy, and the network topology, which improves the rationality and adaptability of selecting the next-hop node. By balancing the packet forwarding benefits and the energy consumption costs based on the Stackelberg Q-learning algorithm, the SQMCR establishes the cooperative communication policy to ensure both the reliability and efficiency of underwater communications. It also adopts initializing Q-values and dynamic exploration probabilities optimization methods to further improve the performance of routing algorithms. Experimental results show that the SQMCR can help UWSNs increase the packet forwarding reliability and prolong the network lifetime by 17%. It has a better environment and application adaptability and is more suitable for underwater high-reliability applications.

During the process of waste collection, various unpredicted disturbances might occur. Meanwhile, vehicle transport is an important source of carbon emissions. In this study, an energy-efficient multi-trip dynamic vehicle routing model is established for waste collection, which introduces two types of dynamic events: new collecting requirements of waste sites and vehicle breakdowns. A Q-learning-based hyperheuristic particle swarm optimization (QLHPSO) is proposed as a dynamic rescheduling method to solve the model. A set of low-level heuristics (LLHs) are designed by combining the learning operators in particle swarm optimization and local search operators. A Q-learning-based high-level strategy is developed to find a suitable LLH for each evolutionary state based on historical performance of LLHs. When rescheduling is triggered, a response mechanism is incorporated to construct initial population by utilizing the features of dynamic events and historical elites. Extensive experimental results on one real instance and nine synthetic instances show that QLHPSO can react to the environmental changes rapidly, and reschedule the vehicle routes with lower cost and carbon emissions compared to the state-of-the-art algorithms.

StarCraft is a popular real-time strategy game that has been widely used as a research platform for artificial intelligence. Micromanagement refers to the process of making each unit perform appropriate actions separately, depending on the current state in the the multi-agent system comprising all of the units, i.e., the fine-grained control of individual units for common benefit. Therefore, cooperation between different units is crucially important to improve the joint strategy. We have selected multi-agent deep reinforcement learning to tackle the problem of micromanagement. In this paper, we propose a method for learning cooperative strategies in StarCraft based on role-based montonic value function factorization (RoMIX). RoMIX learns roles based on the potential impact of each agent on the multi-agent task; it then represents the action value of a role in a mixed way based on monotonic value function factorization. The final value is calculated by accumulating the action value of all roles. The role-based learning improves the cooperation between agents on the team, allowing them to learn the joint strategy more quickly and efficiently. In addition, RoMIX can also reduce storage resources to a certain extent. Experiments show that RoMIX can not only solve easy tasks, but it can also learn better cooperation strategies for more complex and difficult tasks.

In the era of fifth-generation (5G) cellular networks and beyond, network sharing has emerged as a promising approach to address the escalating demand for spectrum and infrastructure resources. Intelligent Neutral Host (INH) is an advanced network-sharing method facilitated by Open Radio Access Network (O-RAN) capabilities. This paper addresses the challenge of Radio Resource Management (RRM) in a multi-operator, multi-slice scenario. We propose an algorithm based on Q-learning and deep Q-learning, particularly concerning different Physical Resource Block (PRB) types to cater to diverse operator requirements. Implemented as an xApp on the Colosseum platform, our algorithm introduces a dynamic resource allocation strategy that adheres to Service Level Agreement (SLA) constraints and optimizes real-time Key Performance metrics (KPMs), including throughput, buffer occupancy, and PRB utilization. We assess the performance and efficacy of our algorithm in a complex traffic scenario to demonstrate how it effectively allocates resources among operators’ slices to satisfy their respective SLA while ensuring optimal resource utilization. The experimental results show that our proposed algorithm can efficiently allocate resources to individual slices while satisfying the SLA. Compared to traditional algorithms, our approach significantly minimizes SLA violations, reducing them to 2.5% for enhanced Mobile Broadband (eMBB) slices and eliminating them entirely for Ultra-Reliable Low-Latency Communications (URLLC) slices.

Students are a gold asset for each country. Proper guidance/recommendation to the students regarding their education-related issues can ultimately result in uplifting the economy of a country. Different education models are followed in the world, out of which Outcome Based Education (OBE) is the one. OBE education model comprises three main components, which include Program Educational Objectives (PEOs), Program Learning Outcomes (PLOs), and Course Learning Outcomes (CLOs). CLOs are outcomes that a student achieves after studying a course. A single course may contain one or more CLOs. These CLOs are then mapped to PLOs and PLOs are then mapped to PEOs. Therefore, our objective in this work is to improve deficient/weak CLOs of students by suggesting online resources. Whereas, in the absence of this proposed system, students have to find out these resources by themselves or course teacher recommends relevant online resources. To achieve this objective, we created a dataset for the OBE education model, as to date no standard dataset exists on the OBE education model. From this dataset, we created a Student-to-CLO matrix and performed bi-clustering on this matrix to find groups of students having similar performance in different CLOs. So far, Bi-clustering has been used in the Bio-informatics field to determine similarity in gene expression data. Generated bi-clusters are sorted according to their homogeneity of contained values. These sorted bi-clusters are then mapped to a 2D grid to formulate a reinforcement-learning environment. The start state of the recommendation agent is determined using cosine similarity. If an agent visits a state, deficient CLOs of that state are recommended to the student. The agent can visit only those states that are nearby to its current state and accessible through its legal action space. An optimal sequence of actions to visit different states of a 2D grid, which can improve student’s performance, is determined using Q-learning. Online resources including research articles, YouTube videos, books, and online tutorials are suggested to the student to improve his deficient CLOs using a mobile app.

The burgeoning growth of vehicular traffic, fuelled by rapid urbanization and an ever-expanding population, has resulted in congested road networks. Traditional and sensor-based adaptive traffic light management systems have shown commendable progress in some scenarios, but they suffer from inherent disadvantages that hinder their effectiveness and scalability. To combat these challenges, a dynamic and adaptable traffic control system is imperative to optimize traffic flow. The present paper explores the limitations of sensor-based adaptive traffic light management and advocates for integrating a novel algorithm to overcome the existing drawbacks. It proposes an intelligent traffic management algorithm called IntelliSignal that leverages the map service for fetching real-time traffic information to calculate the optimal green time and a penalty-based road selection to optimize traffic flow. The proposed IntelliSignal is designed to provide equal chances for all roads while prioritizing higher-density roads with more green time, effectively mitigating traffic congestion and improving overall transportation efficiency. It incorporates Q-learning, a reinforcement learning technique that enables the system to adapt and learn from traffic patterns. The proposed IntelliSignal’s performance is assessed through rigorous evaluations conducted on the simulation platform SUMO. The acquired results demonstrate substantial enhancements across various crucial metrics, including average waiting time, vehicle density, travel time, CO2 emissions, and queue length. Furthermore, the simulation results demonstrate that the proposed IntelliSignal algorithm exhibits a remarkable 30.52% increment in system throughput compared to the traditional approach. This significant enhancement underscores the efficacy of the proposed IntelliSignal in optimizing system performance and merits consideration for practical implementation.

Abstract Software-Defined Networking (SDN) has emerged as an innovative networking method that offers effective management and remarkable flexibility. However, current SDN-based solutions primarily focus on static networks or concentrate on backbone networks, where network dynamics have minimal impact. The existing methods for placing flow entries in Software-Defined Networking (SDN)-based Internet of Things (IoT) systems exhibit shortcomings in accurately predicting outcomes and efficiently reducing table misses and associated performance metrics. This research introduces a new approach, specifically the mobility-aware adaptive flow entry placement scheme for SDN-based Internet of Things (IoT) environments, to address the mobility aspect of networks. The proposed scheme utilizes the Q-learning algorithm to predict the next possible location of end devices, while the cost-sensitive AdaBoost algorithm is employed to select heavy and active flows. As a result, efficient flow rules for incoming flows can be dynamically implemented without controller intervention. Extensive computer simulations demonstrate that this approach significantly enhances match probability and prediction accuracy while concurrently reducing the number of table misses and resource expenditure compared to existing schemes.

An increasing number of users are expected to be served by wireless network with heterogeneous requirements. Unmanned aerial vehicles (UAV) can be deployed to augment the heterogeneous network from aerial area by taking advantages of the characteristics of UAV such as mobility, manoeuvrability, low cost, and Line-of-Sight (LoS) communication. However, the deployment of UAV can also cause problems. For example, in UAV-assisted HetNet, the number of handover (HO) will increase because of the dense distribution of small base stations (SBS) and UAV base stations (UBS). Also, because of the small coverage of SBS and the LoS communication links from neighbour UBSs, the number of unnecessary HO will also rise. Frequent HO and unnecessary HO can result in interruption, increased overhead and energy consumption, which is not desirable for battery powered UAVs. In this paper, to solve the problem, a HO decision making algorithm adopting TOPSIS and Q-learning (QL) is proposed with the aim to reduce HO number and improve energy efficiency. Q-learning can be applied to address decision-making challenges in communication systems widely. However, a large volume of training data can pose challenges and complexities, therefore, the TOPSIS is utilised to reduce the size of the action space in Q-learning. The proposed hybrid TOPSIS-Q-learning method enhances both the handover performance and the scalability. In the method, signal to interference and noise ratio (SINR), time of stay (ToS) and average energy efficiency (EE) are taken into account. The simulation results show that the number of HO and unnecessary HO is remarkably reduced and the average EE is notably improved in comparison with other existing methods.

Exploration is an integral part of learning dynamics which allows algorithms to search a space of solutions. When many algorithms simultaneously explore, this can lead to counter-intuitive effects. This paper contributes an analysis of the influence that exploration has on a multi-agent system of $Q$ -learners in a famous congestion dilemma, the Braess paradox. I find ranges of the exploration rate for which $\epsilon $ -greedy $Q$ -learners show chaotic and oscillatory dynamics which do not converge, and yield better than Nash equilibrium results. I decouple the dynamics endogenous to $Q$ -learning from the exogenous exploration rate $\epsilon $ , and find that $Q$ -learners implicitly coordinate with low exploration rates $\epsilon \in (0, 0.1)$ , but are disrupted in their coordination for larger exploration rates $\epsilon > 0.1$ . The best implicit coordination leads to a 20% reduction in average travel times which approaches the social optimum. I discuss how our results may inform multi-agent algorithm design, fit within a cognitive science perspective of cognitive noise during learning, and provide a mechanistic hypothesis for the lack of empirical evidence of the Braess Paradox in traffic systems.

Various methods are employed in computer vision applications to identify individuals, including using face recognition as a human visual feature helpful in tracking or searching for a person. However, tracking systems that rely solely on facial information encounter limitations, particularly when faced with occlusions, blurred images, or faces oriented away from the camera. Under these conditions, the system struggles to achieve accurate tracking-based face recognition. Therefore, this research addresses this issue by fusing descriptions of the face visual with body visual features. When the system cannot find the target face, the CNN+LSTM hybrid method assists in multi-feature body visual recognition, narrowing the search space and speeding up the search process. The results indicate that the combination of the CNN+LSTM method yields higher accuracy, recall, precision, and F1 scores (reaching 89.20%, 87.36%, 91.02%, and 88.43%, respectively) compared to the single CNN method (reaching 88.84%, 74.00%, 67.00%, and 69.00% respectively). However, the combination of these two visual features requires high computation. Thus, it is necessary to add a tracking system to reduce the computational load and predict the location. Furthermore, this research utilizes the Q-Learning algorithm to make optimal decisions in automatically tracking objects in dynamic environments. The system considers factors such as face and body visual features, object location, and environmental conditions to make the best decisions, aiming to enhance tracking efficiency and accuracy. Based on the conducted experiments, it is concluded that the system can adjust its actions in response to environmental changes with better outcomes. It achieves an accuracy rate of 91.5% and an average of 50 fps in five different videos, as well as a video benchmark dataset with an accuracy of 84% and an average error of 11.15 pixels. Utilizing the proposed method speeds up the search process and optimizes tracking decisions, saving time and computational resources.

Vegetation evolution (VEGE) is a newly proposed meta-heuristic algorithm (MA) with excellent exploitation but relatively weak exploration capacity. We thus focus on further balancing the exploitation and the exploration of VEGE well to improve the overall optimization performance. This paper proposes an improved Q-learning based VEGE, and we design an exploitation archive and an exploration archive to provide a variety of search strategies, each archive contains four efficient and easy-implemented search strategies. In addition, online Q-Learning, as well as ε-greedy scheme, are employed as the decision-maker role to learn the knowledge from the past optimization process and determine the search strategy for each individual automatically and intelligently. In numerical experiments, we compare our proposed QVEGE with eight state-of-the-art MAs including the original VEGE on CEC2020 benchmark functions, twelve engineering optimization problems, and wireless sensor networks (WSN) coverage optimization problems. Experimental and statistical results confirm that the proposed QVEGE demonstrates significant enhancements and stands as a strong competitor among existing algorithms. The source code of QVEGE is publicly available at https://github.com/RuiZhong961230/QVEGE.

Traffic congestion has become an increasingly severe problem and needs to be solved urgently. Reinforcement learning (RL) is an advanced data-driven approach for large-scale adaptive traffic signal control (ATSC) in the complex urban traffic network. The decentralized multi-agent RL (MARL) framework is feasible for multi-intersection scenarios. However, the environment becomes partially observable and nonstationary without effective communication among agents from the perspective of each local agent. Previous studies solve this problem by improving the observation of each local agent and using a spatial discount factor. In this paper, we are motivated by achieving coordination among intersections in the RL settings and better performance in ATSC and solve the problem of traffic signal control at multiple intersections in large-scale networks based on RL techniques. Pressure is used as an indicator to represent the traffic conditions of an intersection considering its neighbor intersections. A new reward function with pressure and vehicle waiting time is designed to achieve coordination among intersections. The above three control policies are innovatively applied to a multi-agent Q-learning method for ATSC at multiple intersections. Experiments are carried out on a large synthetic traffic grid under simulated peak-hour traffic dynamics. Results demonstrate that our approach outperforms the state-of-the-art decentralized MARL algorithms in average queue length and intersection delay.

Abstract With the continuous advancement of deep space exploration missions, the solar system boundary exploration mission is established as one of the China's most important deep space scientific exploration missions. The mission of the solar system boundary exploration has many challenges such as ultra-remote detection distance, ultra-long operation time, and ultra-long communication delay. Therefore, the problem of high-precision autonomous navigation needs to be solved urgently. This paper designs an autonomous intelligent navigation method based on X-ray pulsars in the cruise phase, which estimate the motion state of the probe in real time. The proposed navigation method employs the Q-learning Extended Kalman filter (QLEKF) to improve navigation accuracy during long periods of self-determining running. The QLEKF selects automatically the error covariance matrix parameter of the process noise and the measurement noise by the reward mechanism of reinforcement learning. Compared to the traditional EKF and AEKF, the QLEKF improves the estimation accuracy of position and velocity. Finally, the simulation result demonstrates the effectiveness and the superiority of the intelligent navigation algorithm based on QLEKF, which can satisfy the high-precision navigation requirements in the cruise phase of the solar system boundary exploration.

This study investigates energy efficiency issues of device-to-device (D2D) communications in heterogeneous networks. To minimize the total transmitted power, an approach based on Q-learning together with adaptive ɛ-greedy is proposed to optimize the connection of user equipment (UE) with base station (BS) or access point (AP). The proposed adaptive ɛ-greedy can conduct the adequate exploration and exploitation operations for effective optimization. Simulation results indicate that in the single-cell scenario, the proposed method can attain performance close to the best solution.

Wireless sensor networks are susceptible to a variety of network attacks. Due to the limited energy of nodes and selfish nodes in the network, the packet delivery rate is lower. To address these issues, we innovatively propose an energy-harvesting Q-learning secure routing algorithm with authenticated-encryption. The algorithm uses physical unclonable functions and optimized Q-learning to ensure that the transmission path is reliable. Meanwhile, we combine the LSTM-based prediction model to predict the energy value that the nodes replenish. In addition, simulations are performed to compare the performances of the proposed algorithm with other algorithms under different attacks. The proposed algorithm has greater improvements in the packet delivery rate, filtering selfish nodes, and reducing node energy consumption.

Abstract In urban areas, utilizing traffic lights to prioritize vehicles at the intersection is a solution to control traffic. Among the smart traffic light methods, the methods based on machine learning are particularly important due to their simplicity and performance. In this paper, Q-learning with deep neural network are combined and used in two different intersection models. The first one is an individual intersection, and the second one is two intersections that are connected and shared their actions. By using this method, the traffic light can make an intelligent decision at the intersection to reduce vehicle consumption time by managing the allocation of phases. The proposed smart traffic light is studied and simulated via SUMO. The results illustrated that compared to fixed-time traffic lights, the average queue time of each vehicle in different traffic scenarios has been reduced by 34% in the individual intersection. In the case of two intersections, awareness and communication between agents led to a 24% reduction in the queue time of all cars in the heavy traffic scenario.

Multimedia networks utilize low-power scalar nodes to modify wakeup cycles of high-performance multimedia nodes, which assists in optimizing the power-toperformance ratios. A wide variety of machine learning models are proposed by researchers to perform this task, and most of them are either highly complex, or showcase low-levels of efficiency when applied to large-scale networks. To overcome these issues, this text proposes design of a Q-learning based iterative sleep-scheduling and fuses these schedules with an efficient hybrid bioinspired multipath routing model for largescale multimedia network sets. The proposed model initially uses an iterative Q-Learning technique that analyzes energy consumption patterns of nodes, and incrementally modifies their sleep schedules. These sleep schedules are used by scalar nodes to efficiently wakeup multimedia nodes during adhoc communication requests. These communication requests are processed by a combination of Grey Wolf Optimizer (GWO) & Genetic Algorithm (GA) models, which assist in the identification of optimal paths. These paths are estimated via combined analysis of temporal throughput & packet delivery performance, with node-to-node distance & residual energy metrics. The GWO Model uses instantaneous node & network parameters, while the GA Model analyzes temporal metrics in order to identify optimal routing paths. Both these path sets are fused together via the Q-Learning mechanism, which assists in Iterative Adhoc Path Correction (IAPC), thereby improving the energy efficiency, while reducing communication delay via multipath analysis. Due to a fusion of these models, the proposed Q-Learning based Iterative sleep-scheduling & hybrid Bioinspired Multipath Routing model for Multimedia Networks (QIBMRMN) is able to reduce communication delay by 2.6%, reduce energy consumed during these communications by 14.0%, while improving throughput by 19.6% & packet delivery performance by 8.3% when compared with standard multimedia routing techniques.

The multi-stage, multi-level assembly job shop scheduling problem (MsMlAJSP) is commonly encountered in the manufacturing of complex customized products. Ensuring production efficiency while effectively improving energy utilization is a key focus in the industry. For the energy-efficient MsMlAJSP (EEMsMlAJSP), an improved imperialist competitive algorithm based on Q-learning (IICA-QL) is proposed to minimize the maximum completion time and total energy consumption. In IICA-QL, a decoding strategy with energy-efficient triggers based on problem characteristics is designed to ensure solution quality while effectively enhancing search efficiency. Additionally, an assimilation operation with operator parameter self-adaptation based on Q-learning is devised to overcome the challenge of balancing exploration and exploitation with fixed parameters; thus, the convergence and diversity of the algorithmic search are enhanced. Finally, the effectiveness of the energy-efficient strategy decoding trigger mechanism and the operator parameter self-adaptation operation based on Q-learning is demonstrated through experimental results, and the effectiveness of IICA-QL for solving the EEMsMlAJSP is verified by comparing it with other algorithms.

For the relativistic navigation system where the position and velocity of the spacecraft are determined through the observation of the relativistic perturbations including stellar aberration and starlight gravitational deflection, a novel parallel Q-learning extended Kalman filter (PQEKF) is presented to implement the measurement bias calibration. The relativistic perturbations are extracted from the inter-star angle measurement achieved with a group of high-accuracy star sensors on the spacecraft. Inter-star angle measurement bias caused by the misalignment of the star sensors is one of the main error sources in the relativistic navigation system. In order to suppress the unfavorable effect of measurement bias on navigation performance, the PQEKF is developed to estimate the position and velocity, together with the calibration parameters, where the Q-learning approach is adopted to fine tune the process noise covariance matrix of the filter automatically. The high performance of the presented method is illustrated via numerical simulations in the scenario of medium Earth orbit (MEO) satellite navigation. The simulation results show that, for the considered MEO satellite and the presented PQEKF algorithm, in the case that the inter-star angle measurement accuracy is about 1 mas, after calibration, the positioning accuracy of the relativistic navigation system is less than 300 m.

With the rapid development of mobile edge computing (MEC) and wireless power transfer (WPT) technologies, the MEC-WPT system makes it possible to provide high-quality data processing services for end users. However, in a real-world WPT-MEC system, the channel gain decreases with the transmission distance, leading to “double near and far effect” in the joint transmission of wireless energy and data, which affects the quality of the data processing service for end users. Consequently, it is essential to design a reasonable system model to overcome the “double near and far effect” and make reasonable scheduling of multi-dimensional resources such as energy, communication and computing to guarantee high-quality data processing services. First, this paper designs a relay collaboration WPT-MEC resource scheduling model to improve wireless energy utilization efficiency. The optimization goal is to minimize the normalization of the total communication delay and total energy consumption while meeting multiple resource constraints. Second, this paper imports a BK-means algorithm to complete the end terminals cluster to guarantee effective energy reception and adapts the whale optimization algorithm with adaptive mechanism (AWOA) for mobile vehicle path-planning to reduce energy waste. Third, this paper proposes an immune differential enhanced deep deterministic policy gradient (IDDPG) algorithm to realize efficient resource scheduling of multiple resources and minimize the optimization goal. Finally, simulation experiments are carried out on different data, and the simulation results prove the validity of the designed scheduling model and proposed IDDPG.

With the advancement of radar technology toward multifunctionality and cognitive capabilities, traditional radar countermeasures are no longer sufficient to meet the demands of countering the advanced multifunctional radar (MFR) systems. Rapid and accurate generation of the optimal jamming strategy is one of the key technologies for efficiently completing radar countermeasures. To enhance the efficiency and accuracy of jamming policy generation, an efficient jamming policy generation method based on multi-timescale ensemble Q-learning (MTEQL) is proposed in this paper. First, the task of generating jamming strategies is framed as a Markov decision process (MDP) by constructing a countermeasure scenario between the jammer and radar, while analyzing the principle radar operation mode transitions. Then, multiple structure-dependent Markov environments are created based on the real-world adversarial interactions between jammers and radars. Q-learning algorithms are executed concurrently in these environments, and their results are merged through an adaptive weighting mechanism that utilizes the Jensen–Shannon divergence (JSD). Ultimately, a low-complexity and near-optimal jamming policy is derived. Simulation results indicate that the proposed method has superior jamming policy generation performance compared with the Q-learning algorithm, in terms of the short jamming decision-making time and low average strategy error rate.