Search

Introduction: The increasing penetration of distributed generation (e.g., solar power and wind power) in the energy market has caused unpredictable disturbances in power systems and accelerated the application of intelligent control, such as reinforcement learning (RL), in automatic generation control (AGC). However, traditional RL cannot ensure constraint safety during training and frequently violates the constraints (e.g., frequency limitations), further threatening the safety and stability of grid operation. Methods: To address the safety issue, we propose a novel safe RL framework that combines expert experiences with the RL controller to achieve imitation-based RL. This method allows an initialized safe policy by imitating expert experiences to prevent random explorations at the beginning. Specifically, we first formulate the AGC problem mathematically as a Markov decision process. Then, the imitation mechanism is developed atop a soft actor-critic RL algorithm. Results and discussion: Finally, numerical studies are conducted with an IEEE 39-bus network, which show that the proposed method satisfies the frequency control performance standard better and improves the RL training efficiency. [ABSTRACT FROM AUTHOR]

Dialogue policy is responsible to select the next appropriate action from the current dialogue state to accomplish the user goal efficiently. Present commercial task-oriented dialogue systems are mostly rule-based; thus, they are not easily scalable to adapt multiple domains. To design an adaptive dialogue policy, user feedback is an essential parameter. Recently, deep reinforcement learning algorithms have been popularly applied to such problems. However, managing large state-action space is time consuming and computationally expensive. Additionally, it requires good quality and a reliable user simulator to train the dialogue policy which takes additional design efforts. In this paper, we propose a novel approach to improve the performance of dialogue policy by accelerating the training process by using imitation learning for deep reinforcement learning. We utilized proximal policy optimization (PPO) algorithm to model dialogue policy using a large-scale multi-domain tourist dataset MultiWOZ2.1. We observed a remarkable performance of dialogue policy with 91.8% task success rate, and an approximate 50% decrease in the average number of turns required to complete tasks without using user simulator in the early phase of training cycles. This approach is expected to help researchers to design computationally efficient and scalable dialogue agents by avoiding training from scratch. [ABSTRACT FROM AUTHOR]

Imitation learning enables synthesis of controllers for systems with complex objectives and uncertain plant models. However, ensuring an imitation learned controller is stable requires copious amounts of data and/or a known plant model. In this paper, we explore an input–output (IO) stability approach to imitation learning, which achieves stability with sparse data sets while only requiring coarse knowledge of the energy characteristics of the plant. A constrained optimization problem is developed, in which the controller learns to mimic expert data while maintaining stabilizing energy characteristics induced by the plant. While the learning objective is nonconvex, iterative convex overbounding (ICO) and projected gradient descent (PGD) are explored as methods to learn the controller. In numerical examples, it is shown that with little knowledge of the plant model and a small data set, the dissipativity constrained learned controller achieves closed loop stability and successfully mimics the behavior of the expert controller, while other methods often fail to maintain stability and achieve good performance. [ABSTRACT FROM AUTHOR]

Humans can leverage physical interaction to teach robot arms. This physical interaction takes multiple forms depending on the task, the user, and what the robot has learned so far. State-of-the-art approaches focus on learning from a single modality, or combine some interaction types. Some methods do so by assuming that the robot has prior information about the features of the task and the reward structure. By contrast, in this article, we introduce an algorithmic formalism that unites learning from demonstrations, corrections, and preferences. Our approach makes no assumptions about the tasks the human wants to teach the robot; instead, we learn a reward model from scratch by comparing the human's input to nearby alternatives, i.e., trajectories close to the human's feedback. We first derive a loss function that trains an ensemble of reward models to match the human's demonstrations, corrections, and preferences. The type and order of feedback is up to the human teacher: We enable the robot to collect this feedback passively or actively. We then apply constrained optimization to convert our learned reward into a desired robot trajectory. Through simulations and a user study, we demonstrate that our proposed approach more accurately learns manipulation tasks from physical human interaction than existing baselines, particularly when the robot is faced with new or unexpected objectives. Videos of our user study are available at https://youtu.be/FSUJsTYvEKU. [ABSTRACT FROM AUTHOR]

This paper fuses ideas from reinforcement learning (RL), Learning from Demonstration (LfD), and Ensemble Learning into a single paradigm. Knowledge from a mixture of control algorithms (experts) are used to constrain the action space of the agent, enabling faster RL refining of a control policy, by avoiding unnecessary explorative actions. Domain‐specific knowledge of each expert is exploited. However, the resulting policy is robust against errors of individual experts, since it is refined by a RL reward function without copying any particular demonstration. Our method has the potential to supplement existing RLfD methods when multiple algorithmic approaches are available to function as experts, specifically in tasks involving continuous action spaces. We illustrate our method in the context of a visual servoing (VS) task, in which a 7‐DoF robot arm is controlled to maintain a desired pose relative to a target object. We explore four methods for bounding the actions of the RL agent during training. These methods include using a hypercube and convex hull with modified loss functions, ignoring actions outside the convex hull, and projecting actions onto the convex hull. We compare the training progress of each method using expert demonstrators, employing one expert demonstrator with the DAgger algorithm, and without using any demonstrators. Our experiments show that using the convex hull with a modified loss function not only accelerates learning but also provides the most optimal solution compared with other approaches. Furthermore, we demonstrate faster VS error convergence while maintaining higher manipulability of the arm, compared with classical image‐based VS, position‐based VS, and hybrid‐decoupled VS. [ABSTRACT FROM AUTHOR]

Most of the existing MAPF methods based on communication learning have poor scalability or aggregate too much redundant information, resulting in inefficient communication. To solve these problems, this paper proposed disruptor identifiable communication (DIC), which learned concise communication excluding non-disruptors by judging whether the agent in the center of the field of view would change its decision-making due to the presence of neighbors, and successfully filtered out redundant information. At the same time, this paper further instantiated DIC and developed a new highly scalable distributed MAPF solver: disruptor identifiable communication based on reinforcement and imitation learning algorithm (DICRIA). Firstly, the disruptor discriminator and the policy output layer of DICRIA identified the disruptor. Secondly, the algorithm updated the information of the disruptor and the communication wish sender in two rounds of communication respectively. Finally, DICRIA output the final policy according to the coding results of each module. Experimental results show that DICRIA S performance is better than other similar solvers in almost all environment settings, and the algorithm increases the success rate by 5.2% on average compared to the baseline solver. Especially in dense problem instances with large-size maps, the algorithm even increases the success rate of DICRIA by 44.5% compared to the baseline solver. [ABSTRACT FROM AUTHOR]

This work proposes a modular learning framework (MIRA) for rehabilitation robots based on a new deep recurrent neural network (RNN) that achieves adaptive multi-joint motion imitation. The RNN is fed with the fundamental frequencies as well as the ranges of the joint trajectories, in order to predict the future joint trajectories of the robot. The proposed framework also uses a Segment Online Dynamic Time Warping (SODTW) algorithm to quantify the closeness between the robot and patient motion. The SODTW cost decides the amount of modification needed in the inputs to our deep RNN network, which in turn adapts the robot movements. By keeping the prediction mechanism (RNN) and adaptation mechanism (SODTW) separate, the framework achieves modularity, flexibility, and scalability. We tried both Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) RNN architectures within our proposed framework. Experiments involved a group of 15 human subjects performing a range of motion tasks in conjunction with our social robot, Zeno. Comparative analysis of the results demonstrated the superior performance of the LSTM RNN across multiple task variations, highlighting its enhanced capability for adaptive motion imitation. [ABSTRACT FROM AUTHOR]

The construction industry has long been plagued by low productivity and high injury and fatality rates. Robots have been envisioned to automate the construction process, thereby substantially improving construction productivity and safety. Despite the enormous potential, teaching robots to perform complex construction tasks is challenging. We present a generalizable framework to harness human teleoperation data to train construction robots to perform repetitive construction tasks. First, we develop a teleoperation method and interface to control robots on construction sites, serving as an intermediate solution toward full automation. Teleoperation data from human operators, along with context information from the job site, can be collected for robot learning. Second, we propose a new method for extracting keyframes from human operation data to reduce noise and redundancy in the training data, thereby improving robot learning efficacy. We propose a hierarchical imitation learning method that incorporates the keyframes to train the robot to generate appropriate trajectories for construction tasks. Third, we model the robot's visual observations of the working space in a compact latent space to improve learning performance and reduce computational load. To validate the proposed framework, we conduct experiments teaching a robot to generate appropriate trajectories for excavation tasks from human operators' teleoperations. The results suggest that the proposed method outperforms state-of-the-art approaches, demonstrating its significant potential for application. [ABSTRACT FROM AUTHOR]

Reinforcement learning is now being applied more and more in a variety of scenarios, the majority of which are based on the deep Q network (DQN) technology. However, the algorithm is heavily influenced by multiple factors. In this paper, we take the lunar lander as a case to study how various hyper-parameters affect the performance of the DQN algorithm, based on which we tune to get a model with better performance. At present, it is known that the DQN model has an average reward of 280+ on 100 test episodes, and the reward value of the model in this article can reach 290+. Meanwhile, its robustness is tested and verified by introducing additional uncertainty tests into the original problem. In addition, to speed up the training process, imitation learning is incorporated in our model, using heuristic function model guidance method to obtain demonstration data, which accelerates training speed and improves performance. Simulation results have proven the effectiveness of this method.

Reinforcement learning (RL) and imitation learning (IL) are quite two useful machine learning techniques that were shown to be potential in enhancing navigation performance. Basically, both of these methods try to find a policy decision function in a reinforcement learning fashion or through imitation. In this paper, we propose a novel algorithm named Reinforcement Imitation Learning (RIL) that naturally combines RL and IL together in accelerating more reliable and efficient navigation in dynamic environments. RIL is a hybrid approach that utilizes RL for policy optimization and IL as some kind of learning from expert demonstrations with the inclusion of guidance. We present the comparison of the convergence of RIL with conventional RL and IL to provide the support for our algorithm's performance in a dynamic environment with moving obstacles. The results of the testing indicate that the RIL algorithm has better collision avoidance and navigation efficiency than traditional methods. The proposed RIL algorithm has broad application prospects in many specific areas such as an autonomous driving, unmanned aerial vehicles, and robots. [ABSTRACT FROM AUTHOR]

Imitation learning which uses only expert demonstrations is suitable for safety-crucial tasks, such as autonomous driving. However, causal confusion is a problem in imitation learning where, with more features offered, an agent may perform even worse. Hence, we aim to augment agents' imitation ability in driving scenarios under sequential setting, using a novel method we proposed: sequential masking imitation learning (SEMI). Inspired by the idea of Granger causality, we improve the imitator's performance through a random masking operation on the encoded features in a sequential setting. With this design, the imitator is forced to focus on critical features, leading to a robust model. We demonstrated that this method can alleviate causal confusion in driving simulations by deploying it the CARLA simulator and comparing it with other methods. The experimental results showed that SEMI can effectively reduce confusion during autonomous driving. [ABSTRACT FROM AUTHOR]

The ability of robots to tackle complex non-repetitive tasks will be key in bringing a new level of automation in agricultural applications still involving labor-intensive, menial, and physically demanding activities due to high cognitive requirements. Harvesting is one such example as it requires a combination of motions which can generally be broken down into a visual servoing and a manipulation phase, with the latter often being straightforward to pre-program. In this work, we focus on the task of fresh mushroom harvesting which is still conducted manually by human pickers due to its high complexity. A key challenge is to enable harvesting with low-cost hardware and mechanical systems, such as soft grippers which present additional challenges compared to their rigid counterparts. We devise an Imitation Learning model pipeline utilizing Vector Quantization to learn quantized embeddings directly from visual inputs. We test this approach in a realistic environment designed based on recordings of human experts harvesting real mushrooms. Our models can control a cartesian robot with a soft, pneumatically actuated gripper to successfully replicate the mushroom outrooting sequence. We achieve 100% success in picking mushrooms among distractors with less than 20 min of data collection comprising a single expert demonstration and auxiliary, non-expert, trajectories. The entire model pipeline requires less than 40 min of training on a single A4000 GPU and approx. 20 ms for inference on a standard laptop GPU. [ABSTRACT FROM AUTHOR]

Equipping autonomous agents for dynamic interaction and navigation is a significant challenge in intelligent transportation systems. This study aims to address this by implementing a brain-inspired model for decision making in autonomous vehicles. We employ active inference, a Bayesian approach that models decision-making processes similar to the human brain, focusing on the agent's preferences and the principle of free energy. This approach is combined with imitation learning to enhance the vehicle's ability to adapt to new observations and make human-like decisions. The research involved developing a multi-modal self-awareness architecture for autonomous driving systems and testing this model in driving scenarios, including abnormal observations. The results demonstrated the model's effectiveness in enabling the vehicle to make safe decisions, particularly in unobserved or dynamic environments. The study concludes that the integration of active inference with imitation learning significantly improves the performance of autonomous vehicles, offering a promising direction for future developments in intelligent transportation systems. [ABSTRACT FROM AUTHOR]

As the apparent intelligence of artificial neural networks (ANNs) advances, they are increasingly likened to the functional networks and information processing capabilities of the human brain. Such comparisons have typically focused on particular modalities, such as vision or language. The next frontier is to use the latest advances in ANNs to design and investigate scalable models of higher-level cognitive processes, such as conscious information access, which have historically lacked concrete and specific hypotheses for scientific evaluation. In this work, we propose and then empirically assess an embodied agent with a structure based on global workspace theory (GWT) as specified in the recently proposed "indicator properties" of consciousness. In contrast to prior works on GWT which utilized single modalities, our agent is trained to navigate 3D environments based on realistic audiovisual inputs. We find that the global workspace architecture performs better and more robustly at smaller working memory sizes, as compared to a standard recurrent architecture. Beyond performance, we perform a series of analyses on the learned representations of our architecture and share findings that point to task complexity and regularization being essential for feature learning and the development of meaningful attentional patterns within the workspace. [ABSTRACT FROM AUTHOR]

Swarm robotics is the study of how a large number of relatively simple physically embodied robots can be designed such that a desired collective behavior emerges from local interactions. Furthermore, reinforcement learning (RL) is a promising approach for training robotic swarm controllers. However, the conventional RL approach suffers from the sparse reward problem in some complex tasks, such as key-to-door tasks. In this study, we applied hierarchical imitation learning to train a robotic swarm to address a key-to-door transport task with sparse rewards. The results demonstrate that the proposed approach outperforms the conventional RL method. Moreover, the proposed method outperforms the conventional hierarchical RL method in its ability to adapt to changes in the training environment. [ABSTRACT FROM AUTHOR]

Assistive robot arms try to help their users perform everyday tasks. One way robots can provide this assistance is shared autonomy. Within shared autonomy, both the human and robot maintain control over the robot's motion: as the robot becomes confident it understands what the human wants, it intervenes to automate the task. But how does the robot know these tasks in the first place? State-of-the-art approaches to shared autonomy often rely on prior knowledge. For instance, the robot may need to know the human's potential goals beforehand. During long-term interaction these methods will inevitably break down—sooner or later the human will attempt to perform a task that the robot does not expect. Accordingly, in this article we formulate an alternate approach to shared autonomy that learns assistance from scratch. Our insight is that operators repeat important tasks on a daily basis (e.g., opening the fridge, making coffee). Instead of relying on prior knowledge, we therefore take advantage of these repeated interactions to learn assistive policies. We introduce SARI, an algorithm that recognizes the human's task, replicates similar demonstrations, and returns control when unsure. We then combine learning with control to demonstrate that the error of our approach is uniformly ultimately bounded. We perform simulations to support this error bound, compare our approach to imitation learning baselines, and explore its capacity to assist for an increasing number of tasks. Finally, we conduct three user studies with industry-standard methods and shared autonomy baselines, including a pilot test with a disabled user. Our results indicate that learning shared autonomy across repeated interactions matches existing approaches for known tasks and outperforms baselines on new tasks. See videos of our user studies here: https://youtu.be/3vE4omSvLvc. [ABSTRACT FROM AUTHOR]

Generating good and human-like locomotion or other legged motions for bipedal robots has always been challenging. One of the emerging solutions to this challenge is to use imitation learning. The sources for imitation are mostly state-only demonstrations, so using state-of-the-art Generative Adversarial Imitation Learning (GAIL) with Imitation from Observation (IfO) ability will be an ideal framework to use in solving this problem. However, it is often difficult to allow new or complicated movements as the common sources for these frameworks are either expensive to set up or hard to produce satisfactory results without computationally expensive preprocessing, due to accuracy problems. Inspired by how people learn advanced knowledge after acquiring a basic understanding of specific subjects, this paper proposes a Motion capture-aided Video Imitation (MoVI) learning framework based on Adversarial Motion Priors (AMP) by combining motion capture data of primary actions like walking with video clips of target motion like running, aiming to create smooth and natural imitation results of the target motion. This framework is able to produce various human-like locomotion by taking the most common and abundant motion capture data with any video clips of motion without the need for expensive datasets or sophisticated preprocessing. [ABSTRACT FROM AUTHOR]

Shared autonomy holds promise for assistive robotics, whereby physically- impaired people can direct robots to perform various tasks for them. However, a robot that is capable of many tasks also introduces many choices for the user, such as which object or location should be the target of interaction. In the context of non-invasive brain-computer interfaces for shared autonomy -- most commonly electroencephalography-based -- the two most common choices are to provide either auditory or visual stimuli to the user -- each with their respective pros and cons. Using the oddball paradigm, we designed comparable auditory and visual interfaces to speak/display the choices to the user, and had users complete a multi-stage robotic manipulation task involving location and object selection. Users displayed differing competencies -- and preferences -- for the different interfaces, highlighting the importance of considering modalities outside of vision when constructing human-robot interfaces. [ABSTRACT FROM AUTHOR]

Policy learning enables agents to learn how to map states to actions, thus enabling adaptive and flexible behavioral generation in complex environments. Policy learning methods are fundamental to reinforcement learning techniques. However, as problem complexity and the requirement for motion flexibility increase, traditional methods that rely on manual design have revealed their limitations. Conversely, data-driven policy learning focuses on extracting strategies from biological behavioral data and aims to replicate these behaviors in real-world environments. This approach enhances the adaptability of agents to dynamic substrates. Furthermore, this approach has been extensively applied in autonomous driving, robot control, and interpretation of biological behavior. In this review, we survey developments in data-driven policy-learning algorithms over the past decade. We categorized them into the following three types according to the purpose of the method: (1) imitation learning (IL), (2) inverse reinforcement learning (IRL), and (3) causal policy learning (CPL). We describe the classification principles, methodologies, progress, and applications of each category in detail. In addition, we discuss the distinct features and practical applications of these methods. Finally, we explore the challenges these methods face and prospective directions for future research. [ABSTRACT FROM AUTHOR]

Introduction: Robotic assembly tasks require precise manipulation and coordination, often necessitating advanced learning techniques to achieve efficient and effective performance. While residual reinforcement learning with a base policy has shown promise in this domain, existing base policy approaches often rely on hand-designed full-state features and policies or extensive demonstrations, limiting their applicability in semi-structured environments. Methods: In this study, we propose an innovative Object-Embodiment-Centric Imitation and Residual Reinforcement Learning (OEC-IRRL) approach that leverages an object-embodiment-centric (OEC) task representation to integrate vision models with imitation and residual learning. By utilizing a single demonstration and minimizing interactions with the environment, our method aims to enhance learning efficiency and effectiveness. The proposed method involves three key steps: creating an object-embodiment-centric task representation, employing imitation learning for a base policy using via-point movement primitives for generalization to different settings, and utilizing residual RL for uncertainty-aware policy refinement during the assembly phase. Results: Through a series of comprehensive experiments, we investigate the impact of the OEC task representation on base and residual policy learning and demonstrate the effectiveness of the method in semi-structured environments. Our results indicate that the approach, requiring only a single demonstration and less than 1.2 h of interaction, improves success rates by 46% and reduces assembly time by 25%. Discussion: This research presents a promising avenue for robotic assembly tasks, providing a viable solution without the need for specialized expertise or custom fixtures [ABSTRACT FROM AUTHOR]

Recently, reinforcement learning sequence recommender systems have received widespread attention because they can better model the internal dynamics and external tendencies of user interests. However, existing methods face two major challenges: low utilization of same-strategy evaluation data causes the model to rely on a large amount of expert annotation data and heuristic value incentive functions rely on costly repeated manual debugging. This paper proposed a new hetero-strategic imitation-reinforcement learning method to improve data utilization efficiency and achieve a learnable value function. Firstly, it updated the distribution matching objective function through different strategies to avoid the intensive online interaction limitations of same-strategy updates. Secondly, COG4Rec adopted a learnable value function design and imitated the value incentive function of outdoor tendencies through the logarithmic decay state distribution ratio. Finally, in order to avoid the problem of imitation learning distribution drift, COG4Rec strengthened the recommendation strategy for recombining high-value trajectory segments in user behavior records through the cumulative attenuation distribution ratio. The results of performance comparison experiments and ablation experiments on a series of benchmark data sets show that COG4Rec is 17.60% better than the autoregressive model and 3.25% better than the heuristic reinforcement learning method. This proves the effectiveness of the proposed COG4Rec model structure and optimization algorithm. This also proves that the design of a learnable value function is feasible, and the heterogeneous strategy approach can effectively improve data utilization efficiency. [ABSTRACT FROM AUTHOR]

Wind turbines (WTs) have highly nonlinear and uncertain dynamics due to aerodynamic complexity, mechanical factors, and fluctuations in wind conditions. Turbulence and wind shear add complexity to modelling, especially in constant power region (region 3). Thus, an effective control design demands a deep understanding of the nonlinearities and uncertainties. This paper suggests a novel model-free reinforcement learning (RL) collective pitch angle controller to operate efficiently in region 3. The proposed controller stabilizes generator speed, maximizes power output, and minimizes fluctuations while accommodating system uncertainties, nonlinearity, and pitch limits. The disparity between WT dynamics due to wind speed perturbations and uncertainties is measured using a gap-metric criterion. The controller design adopts a deep deterministic policy gradient (DDPG) algorithm to train six agents in a medium-fidelity WT environment at different mean wind speeds to ensure the controller's robustness. Initially, imitation learning is used for efficient sample collection to fasten training convergence. Afterwards, the agent learns by interacting with the environment. After the training, the pitch control outputs from multi-trained agents are processed by a fuzzy system to have smooth transitions under different operating conditions. The resulting fuzzy DDPG (F-DDPG) controller is deployed to obtain the optimal pitch control action. The performance of the proposed F-DDPG controller is compared to the gain-scheduled PI (GSPI), Linear-Quadratic-Regulator (LQR), and single-DDPG-agent controllers. The controllers are simulated in high-fidelity onshore and offshore 5-MW WT environments using the OpenFAST/MATLAB simulation tools. The results reveal the superiority of the proposed controller in generalizing its optimal performance in different operating conditions. • Proposing novel collective pitch control system using deep reinforcement learning for wind turbines (WTs) to operate efficiently in constant power region (Region 3). • Building a modified version of the DDPG algorithm to fastening the training of multiple agents at various wind speed operating points. • Deployement of a fuzzy logic system to smooth the transition between the pitch control signals from the trained agents to overcome the stochastic time-varying dynamics of the wind turbine. • Enhancing the performance of the high-fidelity NREL Open-FAST WT model in technical and economic terms compared to traditional model-based controllers and single DDPG model-free controller. • Achieving stabilized generator speed, minimum fluctuations, robustness of operation, and maximized power output at all operating points of Region 3. [ABSTRACT FROM AUTHOR]

Offline reinforcement learning (RL) aims to create policies for sequential decision-making using exclusively offline datasets. This presents a significant challenge, especially when attempting to accomplish multiple distinct goals or outcomes within a given scenario while receiving sparse rewards. Prior methods using advantage weighting for offline goal-conditioned learning improve policies monotonically. However, they still face challenges from distribution shift and multi-modality that arise due to conflicting ways to reach a goal. This issue is especially challenging in long-horizon tasks, where the presence of multiple, often conflicting, solutions makes it hard to identify a single optimal policy for transitioning from a state to a desired goal. To address these challenges, we introduce a complementary advantage-based weighting scheme that incorporates an additional source of inductive bias. Given a value-based partitioning of the state space, the contribution of actions expected to lead to target regions that are easier to reach, compared to the final goal, is further increased. Our proposed approach, Dual-Advantage Weighted Offline Goal-conditioned RL, outperforms several competing offline algorithms in widely used benchmarks. Furthermore, we provide a theoretical guarantee that the learned policy will not be inferior to the underlying behavior policy. [ABSTRACT FROM AUTHOR]

Model predictive control (MPC) is promising for optimizing building's operation but high hardware, software and know-how requirements impede its commercialization. Therefore, rule-based controllers (RBC) are state-of-the-art. Approximate MPC (AMPC) can help bridge this gap by replacing the optimization with an explicit functional relation called approximator. Literature lacks reproducible use cases and benchmarks and a comparison of sophisticated and traditional approximators. This study aims to close this gap by applying AMPC to BOPTEST's two-zone heat pump testcase. The BOPTEST testcase includes predefined RBCs and KPIs promoting repeatability. Then, a comparison was made between artificial neural networks (ANNs), random forest (RF), linear, and logistic regression. The results show that feature selection significantly affects the performance. After adapting the features, the ANNs and RF outperform the RBC with cost savings of up to 33% and discomfort reductions of 70%, while requiring 15% of the MPC's computation time. The traditional approximators fail to outperform the RBC. [ABSTRACT FROM AUTHOR]

Purpose: This paper aims to study the agent learning from expert demonstration data while incorporating reinforcement learning (RL), which enables the agent to break through the limitations of expert demonstration data and reduces the dimensionality of the agent's exploration space to speed up the training convergence rate. Design/methodology/approach: Firstly, the decay weight function is set in the objective function of the agent's training to combine both types of methods, and both RL and imitation learning (IL) are considered to guide the agent's behavior when updating the policy. Second, this study designs a coupling utilization method between the demonstration trajectory and the training experience, so that samples from both aspects can be combined during the agent's learning process, and the utilization rate of the data and the agent's learning speed can be improved. Findings: The method is superior to other algorithms in terms of convergence speed and decision stability, avoiding training from scratch for reward values, and breaking through the restrictions brought by demonstration data. Originality/value: The agent can adapt to dynamic scenes through exploration and trial-and-error mechanisms based on the experience of demonstrating trajectories. The demonstration data set used in IL and the experience samples obtained in the process of RL are coupled and used to improve the data utilization efficiency and the generalization ability of the agent. [ABSTRACT FROM AUTHOR]

Real-time bidding has become a major means for online advertisement exchange. The goal of a real-time bidding strategy is to maximize the benefits for stakeholders, e.g., click-through rates or conversion rates. However, in practise, the optimal bidding strategy for real-time bidding is constrained by at least three aspects: cost-effectiveness, the dynamic nature of market prices, and the issue of missing bidding values. To address these challenges, we propose Imagine and Imitate Bidding (IIBidder), which includes Strategy Imitation and Imagination modules, to generate cost-effective bidding strategies under partially observable price landscapes. Experimental results on the iPinYou and YOYI datasets demonstrate that IIBidder reduces investment costs, optimizes bidding strategies, and improves future market price predictions. [ABSTRACT FROM AUTHOR]

Artificial intelligence (AI) research on video games primarily focused on the imitation of human-like behavior during the past few years. Moreover, to increase the perceived worth of amusement and gratification, there is an enormous rise in the demand for intelligent agents that can imitate human players and video game characters. However, the agents developed using the majority of current approaches are perceived as rather more mechanical, which leads to frustration, and more importantly, failure in engagement. On that account, this study proposes an imitation learning framework to generate human-like behavior for more precise and accurate reproduction. To build a computational model, two learning paradigms are explored, artificial neural networks (ANN) and adaptive neuro-fuzzy inference systems (ANFIS). This study utilized several variations of ANN, including feed-forward, recurrent, extreme learning machines, and regressions, to simulate human player behavior. Furthermore, to find the ideal ANFIS, grid partitioning, subtractive clustering, and fuzzy c-means clustering are used for training. The results demonstrate that ANFIS hybrid intelligence systems trained with subtractive clustering are overall best with an average accuracy of 95%, followed by fuzzy c-means with an average accuracy of 87%. Also, the believability of the obtained AI agents is tested using two statistical methods, i.e., the Mann–Whitney U test and the cosine similarity analysis. Both methods validate that the observed behavior has been reproduced with high accuracy.

With large-scale grid integration of renewable energy sources (RES), power grid operations gradually exhibit the new characteristics of high-order uncertainty, leading to significant challenges for system operational security. Traditional model-driven generation dispatch methods require large computational resources, whereas the widely concerned Reinforcement Learning (RL)-based methods lead to issues such as slow training speed due to the high complexity and dimension of processed grid state information. For this reason, this paper proposes a novel Grid Expert Strategy Imitation Learning (GESIL)-based real-time (5 min intervals in this paper) dispatch method. Firstly, a grid model is established based on the graph theory. Secondly, a pure rule-based grid expert strategy (GES) considering detailed power grid operations is proposed. Then, the GES is combined with the established model to obtain a GESIL agent using imitation learning by offline–online training, which can produce specific grid dispatch decisions for real-time. By designing a graph theory-based grid model, a model-driven purely rule-based GES, and embedding a penalty factor-based loss function into IL offline–online training, GESIL ultimately achieves high training speed, high solution speed, and strong generalization capability. A modified IEEE 118-node system is employed to compare the proposed GESIL to traditional dispatch method and RL method. Results show that GESIL has significantly improved computational efficiency by approximately 17 times and training speed by 14.5 times. GESIL can more stably and efficiently compute real-time dispatch decisions of grid operations, enhancing the optimization effect in terms of transmission overloading mitigation, transmission loading optimization, and power balancing control.

Organizations are vulnerable to cyber attacks as they rely on computer networks and the internet for communication and data storage. While Reinforcement Learning (RL) is a widely used strategy to simulate and learn from these attacks, RL-guided offensives against unknown scenarios often lead to early exposure due to low stealth resulting from mistakes during the training phase. To address this issue, this work evaluates if the use of Knowledge Transfer Techniques (KTT), such as Transfer Learning and Imitation Learning, reduces the probability of early exposure by smoothing mistakes during training. This study developed a laboratory platform and a method to compare RL-based cyber attacks using KTT for unknown scenarios. The experiments simulated 2 unknown scenarios using 4 traditional RL algorithms and 4 KTT. In the results, although some algorithms using KTT obtained superior results, they were not so significant for stealth during the initial epochs of training. Nevertheless, experiments also revealed that throughout the entire learning cycle, Trust Region Policy Optimization (TRPO) is a promising algorithm for conducting cyber offensives based on Reinforcement Learning. [ABSTRACT FROM AUTHOR]

Real-time and efficient path planning is critical for all robotic systems. In particular, it is of greater importance for industrial robots since the overall planning and execution time directly impact the cycle time and automation economics in production lines. While the problem may not be complex in static environments, classical approaches are inefficient in high-dimensional environments in terms of planning time and optimality. Collision checking poses another challenge in obtaining a real-time solution for path planning in complex environments. To address these issues, we propose an end-to-end learning-based framework viz., Path Planning and Collision checking Network (PPCNet). The PPCNet generates the path by computing waypoints sequentially using two networks: the first network generates a waypoint, and the second one determines whether the waypoint is on a collision-free segment of the path. The end-to-end training process is based on imitation learning that uses data aggregation from the experience of an expert planner to train the two networks, simultaneously. We utilize two approaches for training a network that efficiently approximates the exact geometrical collision checking function. Finally, the PPCNet is evaluated in two different simulation environments and a practical implementation on a robotic arm for a bin-picking application. Compared to the state-of-the-art path-planning methods, our results show significant improvement in performance by greatly reducing the planning time with comparable success rates and path lengths. [ABSTRACT FROM AUTHOR]

Precision control of multiple robotic fish visual navigation in complex underwater environments has long been a challenging issue in the field of underwater robotics. To address this problem, this paper proposes a multi-robot fish obstacle traversal technique based on the combination of cross-modal variational autoencoder (CM-VAE) and imitation learning. Firstly, the overall framework of the robotic fish control system is introduced, where the first-person view of the robotic fish is encoded into a low-dimensional latent space using CM-VAE, and then different latent features in the space are mapped to the velocity commands of the robotic fish through imitation learning. Finally, to validate the effectiveness of the proposed method, experiments are conducted on linear, S-shaped, and circular gate frame trajectories with both single and multiple robotic fish. Analysis reveals that the visual navigation method proposed in this paper can stably traverse various types of gate frame trajectories. Compared to end-to-end learning and purely unsupervised image reconstruction, the proposed control strategy demonstrates superior performance, offering a new solution for the intelligent navigation of robotic fish in complex environments. [ABSTRACT FROM AUTHOR]

Although robots have been widely used in a variety of fields, the idea of enabling them to perform multiple tasks in the same way that humans do remains a difficulty. To solve this, we investigate the learning from demonstration (LFD) system with our independently designed symmetrical humanoid dual-arm robot. We present a novel action feature matching algorithm. This algorithm accurately transforms human demonstration data into task models that robots can directly execute, considerably improving LFD's generalization capabilities. In our studies, we used motion capture cameras to capture human demonstration actions, which included combinations of simple actions (the action layer) and a succession of complicated operational tasks (the task layer). For the action layer data, we employed Gaussian mixture models (GMM) for processing and constructing an action primitive library. As for the task layer data, we created a "keyframe" segmentation method to transform this data into a series of action primitives and build another action primitive library. Guided by our algorithm, the robot successfully imitated complex human tasks. Results show its excellent task learning and execution, providing an effective solution for robots to learn from human demonstrations and significantly advancing robot technology. [ABSTRACT FROM AUTHOR]

The objective of this study is to achieve high efficiency in autonomous hydraulic excavators by imitating the bucket trajectory operated by an expert. For this purpose, bucket trajectories of experts were collected, and a trajectory was planned using machine learning of a model that relates measured soil shapes to the bucket trajectories of the experts. In this study, we proposed a hierarchical model consisting of a model for estimating movement and a trajectory, with a focus on the fact that different trajectories are generated for the same soil shape as a result of the analysis of the skilled persons' movements. The trajectory output from the model was replanned to have a smooth trajectory using numerical optimization. For the backfilling task, the error from the target shape and the amount of soil transported per movement were compared with those of an expert. The proposed method increased the error from the target shape by approximately 66%, while the amount of soil transported was approximately 58% of that of the experts. [ABSTRACT FROM AUTHOR]

Collaborative robots face barriers to widespread adoption due to the complexity of programming them to achieve human-like movement. Learning from demonstration (LfD) has emerged as a crucial solution, allowing robots to learn tasks directly from expert demonstrations, offering versatility and an intuitive programming approach. However, many existing LfD methods encounter issues such as convergence failure and lack of generalization ability. In this paper, we propose: (1) a generative adversarial network (GAN)-based model with multilayer perceptron (MLP) architecture, coupled with a novel loss function designed to mitigate convergence issues; (2) an affine transformation-based generalization method aimed at enhancing LfD tasks by improving their generalization performance; (3) a data preprocessing method tailored to facilitate deployment on robotics platforms. We conduct experiments on a UR5 robotic platform tasked with handwritten digit recognition. Our results demonstrate that our proposed method significantly accelerates generation speed, achieving a remarkable processing time of 23 ms, which is five times faster than movement primitives (MPs), while preserving key features from demonstrations. This leads to outstanding convergence and generalization performance. [ABSTRACT FROM AUTHOR]

Traditional trajectory learningmethods based on Imitation Learning (IL) only learn the existing trajectory knowledge from human demonstration. In this way, it can not adapt the trajectory knowledge to the task environment by interacting with the environment and fine-tuning the policy. To address this problem, a global trajectory learning method which combinines IL with Reinforcement Learning (RL) to adapt the knowledge policy to the environment is proposed. In this paper, IL is proposed to acquire basic trajectory skills, and then learns the agent will explore and exploit more policy which is applicable to the current environment by RL. The basic trajectory skills include the knowledge policy and the time stage information in the whole task space to help learn the time series of the trajectory, and are used to guide the subsequent RL process. Notably, neural networks are not used to model the action policy and the Q value of RL during the RL process. Instead, they are sampled and updated in the whole task space and then transferred to the networks after the RL process through Behavior Cloning (BC) to get continuous and smooth global trajectory policy. The feasibility and the effectiveness of the method was validated in a custom Gym environment of a flower drawing task. And then, we executed the learned policy in the real-world robot drawing experiment. [ABSTRACT FROM AUTHOR]

Handwriting robots as an application of Imitation Learning (IL). However, most methods have poor accuracy of trajectory generation under task constraints, and models are less robust to changes in demonstration data. This paper proposes an IL algorithm named Bagging in Hidden Semi-Markov Model (BHSMM). The demonstration data is first divided into several sub-datasets, and each sub-dataset is encoded into several basic learning models by Hidden Semi-Markov Models (HSMM). Then the relationship between the task constraint points and the basic learning models is used to derive the weights. Finally, the trajectories adapted to the task constraints are generated based on the weights. We conducted experiments on the handwritten dataset LASA and compared the accuracy error with the original HSMM method. The results show that the BHSMM can generate trajectories that satisfy the position and velocity constraints and is more robust to changes in the demonstration data than the HSMM. In addition, satisfactory results are obtained in trajectory generation for real robot handwriting. [ABSTRACT FROM AUTHOR]

We propose a structured prediction approach for robot imitation learning from demonstrations. Among various tools for robot imitation learning, supervised learning has been observed to have a prominent role. Structured prediction is a form of supervised learning that enables learning models to operate on output spaces with complex structures. Through the lens of structured prediction, we show how robots can learn to imitate trajectories belonging to not only Euclidean spaces but also Riemannian manifolds. Exploiting ideas from information theory, we propose a class of loss functions based on the f -divergence to measure the information loss between the demonstrated and reproduced probabilistic trajectories. Different types of f -divergence will result in different policies, which we call imitation modes. Furthermore, our approach enables the incorporation of spatial and temporal trajectory modulation, which is necessary for robots to be adaptive to the change in working conditions. We benchmark our algorithm against state-of-the-art methods in terms of trajectory reproduction and adaptation. The quantitative evaluation shows that our approach outperforms other algorithms regarding both accuracy and efficiency. We also report real-world experimental results on learning manifold trajectories in a polishing task with a KUKA LWR robot arm, illustrating the effectiveness of our algorithmic framework. [ABSTRACT FROM AUTHOR]