Search

Foundation models often struggle with uncertainty when faced with new situations in online decision-making, necessitating scalable and efficient exploration to resolve this uncertainty. We introduce GPT-HyperAgent, an augmentation of GPT with HyperAgent for uncertainty-aware, scalable exploration in contextual bandits, a fundamental online decision problem involving natural language input. We prove that HyperAgent achieves fast incremental uncertainty estimation with $\tilde{O}(\log T)$ per-step computational complexity over $T$ periods under the linear realizable assumption. Our analysis demonstrates that HyperAgent's regret order matches that of exact Thompson sampling in linear contextual bandits, closing a significant theoretical gap in scalable exploration. Empirical results in real-world contextual bandit tasks, such as automated content moderation with human feedback, validate the practical effectiveness of GPT-HyperAgent for safety-critical decisions. Our code is open-sourced at \url{https://github.com/szrlee/GPT-HyperAgent/}., Comment: 43 pages. Presentation at ICML 2024 Workshops: (1) Aligning Reinforcement Learning Experimentalists and Theorists; (2) Automated Reinforcement Learning: Exploring Meta-Learning, AutoML, and LLMs

Learning policies from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making and avoiding unsafe and costly online interactions. However, real-world data collected from sensors or humans often contains noise and errors, posing a significant challenge for existing offline RL methods. Our study indicates that traditional offline RL methods based on temporal difference learning tend to underperform Decision Transformer (DT) under data corruption, especially when the amount of data is limited. This suggests the potential of sequential modeling for tackling data corruption in offline RL. To further unleash the potential of sequence modeling methods, we propose Robust Decision Transformer (RDT) by incorporating several robust techniques. Specifically, we introduce Gaussian weighted learning and iterative data correction to reduce the effect of corrupted data. Additionally, we leverage embedding dropout to enhance the model's resistance to erroneous inputs. Extensive experiments on MoJoCo, KitChen, and Adroit tasks demonstrate RDT's superior performance under diverse data corruption compared to previous methods. Moreover, RDT exhibits remarkable robustness in a challenging setting that combines training-time data corruption with testing-time observation perturbations. These results highlight the potential of robust sequence modeling for learning from noisy or corrupted offline datasets, thereby promoting the reliable application of offline RL in real-world tasks.

Rate-Splitting Multiple Access (RSMA) has emerged as a potent and reliable multiple access and interference management technique in wireless communications. While downlink Multiple-Input Multiple-Ouput (MIMO) RSMA has been widely investigated, uplink MIMO RSMA has not been fully explored. In this paper, we investigate the performance of uplink RSMA in short-packet communications with perfect Channel State Information at Transmitter (CSIT) and Channel State Information at Receiver (CSIR). We propose an uplink MIMO RSMA framework and optimize both precoders and combiners with Max-Min Fairness (MMF) metric and Finite Blocklength (FBL) constraints. Due to the high coupling between precoders and combiners, we apply the Alternating Optimization (AO) to decompose the optimization problem into two subproblems. To tackle these subproblems, we propose a Successive Convex Approximation (SCA)-based approach. Additionally, we introduce a low-complexity scheme to design the decoding order at the receiver. Subsequently, the Physical (PHY)-layer of the uplink MIMO RSMA architecture is designed and evaluated using multi-user Link-Level Simulations (LLS), accounting for finite constellation modulation, finite length polar codes, message splitting, adaptive modulation and coding, and Successive Interference Cancellation (SIC) at the receiver. Numerical results demonstrate that applying RSMA in uplink MIMO with FBL constraints not only achieves MMF gains over conventional transmission schemes such as Space Division Multiple Access (SDMA) and Non-orthogonal Multiple Access (NOMA) but also exhibits robustness to network loads. The benefits of splitting messages from multiple users are also illustrated. LLS results confirm the improved max-min throughput benefits of RSMA over SDMA and NOMA.

We present the Thought Graph as a novel framework to support complex reasoning and use gene set analysis as an example to uncover semantic relationships between biological processes. Our framework stands out for its ability to provide a deeper understanding of gene sets, significantly surpassing GSEA by 40.28% and LLM baselines by 5.38% based on cosine similarity to human annotations. Our analysis further provides insights into future directions of biological processes naming, and implications for bioinformatics and precision medicine., Comment: 4 pages. Accepted by Web Conf 2024

Routine visual inspections of concrete structures are imperative for upholding the safety and integrity of critical infrastructure. Such visual inspections sometimes happen under low-light conditions, e.g., checking for bridge health. Crack segmentation under such conditions is challenging due to the poor contrast between cracks and their surroundings. However, most deep learning methods are designed for well-illuminated crack images and hence their performance drops dramatically in low-light scenes. In addition, conventional approaches require many annotated low-light crack images which is time-consuming. In this paper, we address these challenges by proposing CrackNex, a framework that utilizes reflectance information based on Retinex Theory to help the model learn a unified illumination-invariant representation. Furthermore, we utilize few-shot segmentation to solve the inefficient training data problem. In CrackNex, both a support prototype and a reflectance prototype are extracted from the support set. Then, a prototype fusion module is designed to integrate the features from both prototypes. CrackNex outperforms the SOTA methods on multiple datasets. Additionally, we present the first benchmark dataset, LCSD, for low-light crack segmentation. LCSD consists of 102 well-illuminated crack images and 41 low-light crack images. The dataset and code are available at https://github.com/zy1296/CrackNex., Comment: 8 pages, 4 figures, IEEE International Conference on Robotics and Automation (ICRA) 2024

In this letter, we investigate the performance of Max Minimum Fairness (MMF) for uplink Rate-Splitting Multiple Access (RSMA) in short-packet communications. Specifically, considering a Single-Input Single-Output (SISO) Multiple Access Channel (MAC), we optimize the transmit power allocation between the splitting user messages to maximize the minimum rate among users with Finite Blocklength (FBL) constraints. To tackle this problem, we propose a Successive Convex Approximation (SCA)-based approach. Additionally, we introduce a low-complexity scheme to design the decoding order at the receiver. Numerical results show that RSMA outperforms conventional transmission schemes such as Non-orthogonal Multiple Access (NOMA) in terms of MMF., Comment: 5 pages, 4 figures

We propose HyperAgent, a reinforcement learning (RL) algorithm based on the hypermodel framework for exploration in RL. HyperAgent allows for the efficient incremental approximation of posteriors associated with an optimal action-value function ($Q^\star$) without the need for conjugacy and follows the greedy policies w.r.t. these approximate posterior samples. We demonstrate that HyperAgent offers robust performance in large-scale deep RL benchmarks. It can solve Deep Sea hard exploration problems with episodes that optimally scale with problem size and exhibits significant efficiency gains in the Atari suite. Implementing HyperAgent requires minimal code addition to well-established deep RL frameworks like DQN. We theoretically prove that, under tabular assumptions, HyperAgent achieves logarithmic per-step computational complexity while attaining sublinear regret, matching the best known randomized tabular RL algorithm., Comment: Proceedings of the $\mathit{41}^{st}$ International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024. Copyright 2024 by the author(s). Invited talk in Informs Optimization Conference 2024 and International Symposium on Mathematical Programming 2024

Research on Multi-rotor Aerial Vehicles (MAVs) has experienced remarkable advancements over the past two decades, propelling the field forward at an accelerated pace. Through the implementation of motion control and the integration of specialized mechanisms, researchers have unlocked the potential of MAVs to perform a wide range of tasks in diverse scenarios. Notably, the literature has highlighted the distinctive attributes of MAVs that endow them with a competitive edge in physical interaction when compared to other robotic systems. In this survey, we present a categorization of the various types of physical interactions in which MAVs are involved, supported by comprehensive case studies. We examine the approaches employed by researchers to address different challenges using MAVs and their applications, including the development of different types of controllers to handle uncertainties inherent in these interactions. By conducting a thorough analysis of the strengths and limitations associated with different methodologies, as well as engaging in discussions about potential enhancements, this survey aims to illuminate the path for future research focusing on MAVs with high actuation capabilities.

Multimode fibers (MMFs) have found wide application across various fields, such as optical communications, mode-locked lasers, and endoscopy. However, the practical use of MMFs is limited by the challenges posed by fiber bending, which leads to mode coupling. In this study, we present evidence that MMFs possess principal modes, named curved principal modes, that can resist significant bending. These curved principal modes are identified by extending the Wigner-Smith operator to curved MMFs, and are demonstrated for arbitrary bending by numerical simulations. These findings have substantial implications for mode-divide-multiplexed optical fiber communications, MMF-based endoscopy, and other related applications., Comment: 8 pages, 3 figures

Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in real-world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios., Comment: Accepted by ICLR 2024

Rural communities in remote areas often encounter significant challenges when it comes to accessing emergency healthcare services and essential supplies due to a lack of adequate transportation infrastructure. The situation is further exacerbated by poorly maintained, damaged, or flooded roads, making it arduous for rural residents to obtain the necessary aid in critical situations. Limited budgets and technological constraints pose additional obstacles, hindering the prompt response of local rescue teams during emergencies. The transportation of crucial resources, such as medical supplies and food, plays a vital role in saving lives in these situations. In light of these obstacles, our objective is to improve accessibility and alleviate the suffering of vulnerable populations by automating transportation tasks using low-cost robotic systems. We propose a low-cost, easy-to-build blimp robot (UAVs), that can significantly enhance the efficiency and effectiveness of local emergency responses., Comment: IEEE Global Humanitarian Technology Conference (GHTC 2023)

We present an aerial vehicle composed of a custom quadrotor with tilted rotors and a helium balloon, called SBlimp. We propose a novel control strategy that takes advantage of the natural stable attitude of the blimp to control translational motion. Different from cascade controllers in the literature that controls attitude to achieve desired translational motion, our approach directly controls the linear velocity regardless of the heading orientation of the vehicle. As a result, the vehicle swings during the translational motion. We provide a planar analysis of the dynamic model, demonstrating stability for our controller. Our design is evaluated in numerical simulations with different physical factors and validated with experiments using a real-world prototype, showing that the SBlimp is able to achieve stable translation regardless of its orientation.

Dexterous manipulation of objects through fine control of physical contacts is essential for many important tasks of daily living. A fundamental ability underlying fine contact control is compliant control, \textit{i.e.}, controlling the contact forces while moving. For robots, the most widely explored approaches heavily depend on models of manipulated objects and expensive sensors to gather contact location and force information needed for real-time control. The models are difficult to obtain, and the sensors are costly, hindering personal robots' adoption in our homes and businesses. This study performs model-free reinforcement learning of a normal contact force controller on a robotic manipulation system built with a low-cost, information-poor tactile sensor. Despite the limited sensing capability, our force controller can be combined with a motion controller to enable fine contact interactions during object manipulation. Promising results are demonstrated in non-prehensile, dexterous manipulation experiments.

Traditional aerial vehicles have limitations in their capabilities due to actuator constraints, such as motor saturation. The hardware components and their arrangement are designed to satisfy specific requirements and are difficult to modify during operation. To address this problem, we introduce a versatile modular multi-rotor vehicle that can change its capabilities by reconfiguration. Our modular robot consists of homogeneous cuboid modules, propelled by quadrotors with tilted rotors. Depending on the number of modules and their configuration, the robot can expand its actuation capabilities. In this paper, we build a mathematical model for the actuation capability of a modular multi-rotor vehicle and develop methods to determine if a vehicle is capable of satisfying a task requirement. Based on this result, we find the optimal configurations for a given task. Our approach is validated in realistic 3D simulations, showing that our modular system can adapt to tasks with varying requirements.

Lack of texture often causes ambiguity in matching, and handling this issue is an important challenge in optical flow estimation. Some methods insert stacked transformer modules that allow the network to use global information of cost volume for estimation. But the global information aggregation often incurs serious memory and time costs during training and inference, which hinders model deployment. We draw inspiration from the traditional local region constraint and design the local similarity aggregation (LSA) and the shifted local similarity aggregation (SLSA). The aggregation for cost volume is implemented with lightweight modules that act on the feature maps. Experiments on the final pass of Sintel show the lower cost required for our approach while maintaining competitive performance.

In this letter, we investigate Rate-Splitting Multiple Access (RSMA) for an uplink communication system with finite blocklength. Considering a two-user Single-Input Single-Output (SISO) Multiple Access Channel (MAC), we study the impact of Signal-to-Noise Ratio (SNR), blocklength, power allocation and target rate on the error probability performance of RSMA where one user message is split. We demonstrate that RSMA can improve the error probability performance significantly compared to Non-Orthogonal Multiple Access (NOMA) and RSMA can have a larger rate region than NOMA., Comment: 5 pages, 4 figures

We consider the problem of policy transfer between two Markov Decision Processes (MDPs). We introduce a lemma based on existing theoretical results in reinforcement learning to measure the relativity gap between two arbitrary MDPs, that is the difference between any two cumulative expected returns defined on different policies and environment dynamics. Based on this lemma, we propose two new algorithms referred to as Relative Policy Optimization (RPO) and Relative Transition Optimization (RTO), which offer fast policy transfer and dynamics modelling, respectively. RPO transfers the policy evaluated in one environment to maximize the return in another, while RTO updates the parameterized dynamics model to reduce the gap between the dynamics of the two environments. Integrating the two algorithms results in the complete Relative Policy-Transition Optimization (RPTO) algorithm, in which the policy interacts with the two environments simultaneously, such that data collections from two environments, policy and transition updates are completed in one closed loop to form a principled learning framework for policy transfer. We demonstrate the effectiveness of RPTO on a set of MuJoCo continuous control tasks by creating policy transfer problems via variant dynamics., Comment: Accepted by AAAI 2024

Malware continues to evolve rapidly, and more than 450,000 new samples are captured every day, which makes manual malware analysis impractical. However, existing deep learning detection models need manual feature engineering or require high computational overhead for long training processes, which might be laborious to select feature space and difficult to retrain for mitigating model aging. Therefore, a crucial requirement for a detector is to realize automatic and efficient detection. In this paper, we propose a lightweight malware detection model called SeqNet which could be trained at high speed with low memory required on the raw binaries. By avoiding contextual confusion and reducing semantic loss, SeqNet maintains the detection accuracy when reducing the number of parameters to only 136K. We demonstrate the effectiveness of our methods and the low training cost requirement of SeqNet in our experiments. Besides, we make our datasets and codes public to stimulate further academic research.

We present a design, model, and control for a novel jumping-flying robot that is called PogoDrone. The robot is composed of a quadrotor with a passive mechanism for jumping. The robot can continuously jump in place or fly like a normal quadrotor. Jumping in place allows the robot to quickly move and operate very close to the ground. For instance, in agricultural applications, the jumping mechanism allows the robot to take samples of soil. We propose a hybrid controller that switches from attitude to position control to allow the robot to fall horizontally and recover to the original position. We compare the jumping mode with the hovering mode to analyze the energy consumption. In simulations, we evaluate the effect of different factors on energy consumption. In real experiments, we show that our robot can repeatedly impact the ground, jump, and fly in a physical environment.

The existing generative adversarial fusion methods generally concatenate source images and extract local features through convolution operation, without considering their global characteristics, which tends to produce an unbalanced result and is biased towards the infrared image or visible image. Toward this end, we propose a novel end-to-end mode based on generative adversarial training to achieve better fusion balance, termed as \textit{interactive compensatory attention fusion network} (ICAFusion). In particular, in the generator, we construct a multi-level encoder-decoder network with a triple path, and adopt infrared and visible paths to provide additional intensity and gradient information. Moreover, we develop interactive and compensatory attention modules to communicate their pathwise information, and model their long-range dependencies to generate attention maps, which can more focus on infrared target perception and visible detail characterization, and further increase the representation power for feature extraction and feature reconstruction. In addition, dual discriminators are designed to identify the similar distribution between fused result and source images, and the generator is optimized to produce a more balanced result. Extensive experiments illustrate that our ICAFusion obtains superior fusion performance and better generalization ability, which precedes other advanced methods in the subjective visual description and objective metric evaluation. Our codes will be public at \url{https://github.com/Zhishe-Wang/ICAFusion}, Comment: 13pages,12 figures

Traditional aerial vehicles have specific characteristics to perform specific tasks but designing a versatile vehicle that can adapt depending on the task is still a challenge. Based on modularity, we propose an aerial robotic system that can increase its payload capacity and actuated degrees of freedom by reconfiguring heterogeneous modules to adapt to different task specifications. The system consists of cuboid modules propelled by quadrotors with tilted rotors. We present two module designs with different actuation properties. By assembling different types of modules, H-ModQuad can increase its actuated degrees of freedom from 4 to 5 and 6 depending on its configuration. By extending the concept of actuation ellipsoids, we find the body frame of a vehicle with which the controller can maximize the maximum thrust. We use polytopes to represent the actuation capability of the vehicles and examine them against task requirements. We derive the modular vehicles' dynamics and propose a general control strategy that applies for all possible numbers of actuated degrees of freedom. The design is validated with simulations and experiments using actual robots, showing that the modular vehicles provide different actuation properties.