Search

In this note, we study distributed time-varying optimization for a multi-agent system. We first focus on a class of time-varying quadratic cost functions, and develop a new distributed algorithm that integrates an average estimator and an adaptive optimizer, with both bridged by a Dead Zone Algorithm. Based on a composite Lyapunov function and finite escape-time analysis, we prove the closed-loop global asymptotic convergence to the optimal solution under mild assumptions. Particularly, the introduction of the estimator relaxes the requirement for the Hessians of cost functions, and the integrated design eliminates the waiting time required in the relevant literature for estimating global parameter during algorithm implementation. We then extend this result to a more general class of time-varying cost functions. Two examples are used to verify the proposed designs., Comment: 11 pages, 7 figures

GWnext 2024 was a meeting held in the Kavli Institute for Astronomy and Astrophysics at Peking University in March $4^\text{th} - 8^\text{th}$, 2024. In the meeting researchers at different career stages -- with a particular focus on early career scientists -- working on the different aspects of gravitational wave (GW) astronomy gathered to discuss the current status as well as prospects of the field. The meeting was divided into three core sessions: Astrophysics, GW Theory, and Detection. Each session consisted of introductory talks and extended discussion sessions. Moreover, there was a poster session where students could present their results. In this paper, we summarize the results presented during the meeting and present the most important outcomes.

The development of robotic systems for palletization in logistics scenarios is of paramount importance, addressing critical efficiency and precision demands in supply chain management. This paper investigates the application of Reinforcement Learning (RL) in enhancing task planning for such robotic systems. Confronted with the substantial challenge of a vast action space, which is a significant impediment to efficiently apply out-of-the-shelf RL methods, our study introduces a novel method of utilizing supervised learning to iteratively prune and manage the action space effectively. By reducing the complexity of the action space, our approach not only accelerates the learning phase but also ensures the effectiveness and reliability of the task planning in robotic palletization. The experimental results underscore the efficacy of this method, highlighting its potential in improving the performance of RL applications in complex and high-dimensional environments like logistics palletization., Comment: 8 pages, 8 figures

Efficiency and reliability are critical in robotic bin-picking as they directly impact the productivity of automated industrial processes. However, traditional approaches, demanding static objects and fixed collisions, lead to deployment limitations, operational inefficiencies, and process unreliability. This paper introduces a Dynamic Bin-Picking Framework (DBPF) that challenges traditional static assumptions. The DBPF endows the robot with the reactivity to pick multiple moving arbitrary objects while avoiding dynamic obstacles, such as the moving bin. Combined with scene-level pose generation, the proposed pose selection metric leverages the Tendency-Aware Manipulability Network optimizing suction pose determination. Heuristic task-specific designs like velocity-matching, dynamic obstacle avoidance, and the resight policy, enhance the picking success rate and reliability. Empirical experiments demonstrate the importance of these components. Our method achieves an average 84% success rate, surpassing the 60% of the most comparable baseline, crucially, with zero collisions. Further evaluations under diverse dynamic scenarios showcase DBPF's robust performance in dynamic bin-picking. Results suggest that our framework offers a promising solution for efficient and reliable robotic bin-picking under dynamics., Comment: 8 pages, 5 figures. This paper has been accepted by IEEE RA-L on 2024-03-24. See the supplementary video at youtube: https://youtu.be/n5af2VsKhkg

Reinforcement learning (RL) is a versatile framework for optimizing long-term goals. Although many real-world problems can be formalized with RL, learning and deploying a performant RL policy requires a system designed to address several important challenges, including the exploration-exploitation dilemma, partial observability, dynamic action spaces, and safety concerns. While the importance of these challenges has been well recognized, existing open-source RL libraries do not explicitly address them. This paper introduces Pearl, a Production-Ready RL software package designed to embrace these challenges in a modular way. In addition to presenting benchmarking results, we also highlight examples of Pearl's ongoing industry adoption to demonstrate its advantages for production use cases. Pearl is open sourced on GitHub at github.com/facebookresearch/pearl and its official website is pearlagent.github.io.

This paper studies distributed continuous-time optimization for time-varying quadratic cost functions with uncertain parameters. We first propose a centralized adaptive optimization algorithm using partial information of the cost function. It can be seen that even if there are uncertain parameters in the cost function, exact optimization can still be achieved. To solve this problem in a distributed manner when different local cost functions have identical Hessians, we propose a novel distributed algorithm that cascades the fixed-time average estimator and the distributed optimizer. We remove the requirement for the upper bounds of certain complex functions by integrating state-based gains in the proposed design. We further extend this result to address the distributed optimization where the time-varying cost functions have nonidentical Hessians. We prove the convergence of all the proposed algorithms in the global sense. Numerical examples verify the proposed algorithms., Comment: 11 pages,10 figures

Learning contact-rich manipulation skills is essential. Such skills require the robots to interact with the environment with feasible manipulation trajectories and suitable compliance control parameters to enable safe and stable contact. However, learning these skills is challenging due to data inefficiency in the real world and the sim-to-real gap in simulation. In this paper, we introduce a hybrid offline-online framework to learn robust manipulation skills. We employ model-free reinforcement learning for the offline phase to obtain the robot motion and compliance control parameters in simulation \RV{with domain randomization}. Subsequently, in the online phase, we learn the residual of the compliance control parameters to maximize robot performance-related criteria with force sensor measurements in real time. To demonstrate the effectiveness and robustness of our approach, we provide comparative results against existing methods for assembly, pivoting, and screwing tasks., Comment: Conference on Robot Learning (CoRL) 2023

Space-based gravitational wave detectors such as TianQin, LISA, and TaiJi have the potential to outperform themselves through joint observation. To achieve this, it is desirable to practice joint data analysis in advance on simulated data that encodes the intrinsic correlation among the signals found in different detectors that operate simultaneously. In this paper, we introduce \texttt{GWSpace}, a package that can simulate the joint detection data from TianQin, LISA, and TaiJi. The software is not a groundbreaking work that starts from scratch. Rather, we use as many open-source resources as possible, tailoring them to the needs of simulating the multi-mission science data and putting everything into a ready-to-go and easy-to-use package. We shall describe the main components, the construction, and a few examples of application of the package. A common coordinate system, namely the Solar System Barycenter (SSB) coordinate system, is utilized to calculate spacecraft orbits for all three missions. The paper also provides a brief derivation of the detection process and outlines the general waveform of sources detectable by these detectors., Comment: 24 pages, 13 figures, GWSpace will be uploaded at https://github.com/TianQinSYSU/GWSpace

The millihertz gravitational wave band is expected to be opened by space-borne detectors like TianQin. Various mechanisms can produce short outbursts of gravitational waves, whose actual waveform can be hard to model. In order to identify such gravitational wave bursts and not to misclassify them as noise transients, we proposed a proof-of-principle energy excess method, that utilized the signal-insensitive channel to veto noise transients. We perform a test on simulated data, and for bursts with a signal-to-noise ratio of 20, even with the contamination of noise transient, our methods can reach a detection efficiency of 97.4% under a false alarm rate of once per year. However, more frequent occurrences of noise transients would lower the detection efficiency., Comment: 13 pages, 9 figures, comments welcome

As the core of artificial intelligence applications, the research of convolution has become a hot topic in high performance computing. With the rapid development of the emerging SW26010 processor in artificial intelligence, there is an urgent need for high-performance convolution algorithms on the processor. However, the current support of convolution on SW26010 is still rudimentary. The only studies provide sufficient runtime peak performance but lack the adaptability to various convolution scenes. To perfect convolution algorithms on SW26010, we propose a multi-grained matrix-multiplication-mapping convolution algorithm called MG3MConv, which targets the architectural features of SW26010. MG3MConv supports diversified mapping schemes of convolution tasks based on the concept of the thread block proposed in this paper. All the architecture-oriented optimization methods are elaborately designed from four levels to fully exploit the hardware efficiency of SW26010. The experiments show that the hardware efficiency of MG3MConv can reach 84.78% in max, which is 1.75 times compared with that of cuDNN based on NVIDIA K80m GPU. Moreover, MG3MConv can overperform cuDNN in most convolution scenes. We also use six representative CNNs as real-world cases, and the hardware efficiency of MG3MConv reaches up to 67.04% on the VGG network model, which is 1.37 times and 1.96 times that of cuDNN and swDNN, respectively.

In this paper, a deep reinforcement learning based method is proposed to obtain optimal policies for optimal infinite-horizon control of probabilistic Boolean control networks (PBCNs). Compared with the existing literatures, the proposed method is model-free, namely, the system model and the initial states needn't to be known. Meanwhile, it is suitable for large-scale PBCNs. First, we establish the connection between deep reinforcement learning and optimal infinite-horizon control, and structure the problem into the framework of the Markov decision process. Then, PBCNs are defined as large-scale or small-scale, depending on whether the memory of the action-values exceeds the RAM of the computer. Based on the newly introduced definition, Q-learning (QL) and double deep Q-network (DDQN) are applied to the optimal infinite-horizon control of small-scale and large-scale PBCNs, respectively. Meanwhile, the optimal state feedback controllers are designed. Finally, two examples are presented, which are a small-scale PBCN with 3 nodes, and a large-scale one with 28 nodes. To verify the convergence of QL and DDQN, the optimal control policy and the optimal action-values, which are obtained from both the algorithms, are compared with the ones based on a model-based method named policy iteration. Meanwhile, the performance of QL is compared with DDQN in the small-scale PBCN.

Reinforcement learning often suffer from the sparse reward issue in real-world robotics problems. Learning from demonstration (LfD) is an effective way to eliminate this problem, which leverages collected expert data to aid online learning. Prior works often assume that the learning agent and the expert aim to accomplish the same task, which requires collecting new data for every new task. In this paper, we consider the case where the target task is mismatched from but similar with that of the expert. Such setting can be challenging and we found existing LfD methods can not effectively guide learning in mismatched new tasks with sparse rewards. We propose conservative reward shaping from demonstration (CRSfD), which shapes the sparse rewards using estimated expert value function. To accelerate learning processes, CRSfD guides the agent to conservatively explore around demonstrations. Experimental results of robot manipulation tasks show that our approach outperforms baseline LfD methods when transferring demonstrations collected in a single task to other different but similar tasks., Comment: 11 pages, 5 figures, CoRL 2022

Learning generalizable insertion skills in a data-efficient manner has long been a challenge in the robot learning community. While the current state-of-the-art methods with reinforcement learning (RL) show promising performance in acquiring manipulation skills, the algorithms are data-hungry and hard to generalize. To overcome the issues, in this paper we present Prim-LAfD, a simple yet effective framework to learn and adapt primitive-based insertion skills from demonstrations. Prim-LAfD utilizes black-box function optimization to learn and adapt the primitive parameters leveraging prior experiences. Human demonstrations are modeled as dense rewards guiding parameter learning. We validate the effectiveness of the proposed method on eight peg-hole and connector-socket insertion tasks. The experimental results show that our proposed framework takes less than one hour to acquire the insertion skills and as few as fifteen minutes to adapt to an unseen insertion task on a physical robot., Comment: 6 pages, 4 figures

Conventional theoretical studies on the ground-state laser cooling of a trapped ion have mostly focused on the weak sideband coupling (WSC) regime, where the cooling rate is inverse proportional to the linewidth of the excited state. In a recent work~[New J. Phys. 23, 023018 (2021)], we proposed a theoretical framework to study the ground state cooling of a trapped ion in the strong sideband coupling (SSC) regime, under the assumption of a vanishing carrier transition. Here we extend this analysis to more general situations with nonvanishing carrier transitions, where we show that by properly tuning the coupling lasers a cooling rate proportional to the linewidth can be achieved. Our theoretical predictions closely agree with the corresponding exact solutions in the SSC regime, which provide an important theoretical guidance for sideband cooling experiments., Comment: 7 pages, 4 figures

Humans are capable of abstracting various tasks as different combinations of multiple attributes. This perspective of compositionality is vital for human rapid learning and adaption since previous experiences from related tasks can be combined to generalize across novel compositional settings. In this work, we aim to achieve zero-shot policy generalization of Reinforcement Learning (RL) agents by leveraging the task compositionality. Our proposed method is a meta- RL algorithm with disentangled task representation, explicitly encoding different aspects of the tasks. Policy generalization is then performed by inferring unseen compositional task representations via the obtained disentanglement without extra exploration. The evaluation is conducted on three simulated tasks and a challenging real-world robotic insertion task. Experimental results demonstrate that our proposed method achieves policy generalization to unseen compositional tasks in a zero-shot manner., Comment: 7 pages, 9 figures

The capture of compact objects by massive black holes in galaxies or dwarf galaxies will generate short gravitational wave signals, called extreme-mass-ratio bursts (EMRBs), before evolving into extreme-mass-ratio inspirals. Their detection will provide an investigation of the black hole properties and shed light on astronomy and astrophysics. In this work, we investigate the detection number of the TianQin observatory on EMRBs. Our result shows that TianQin can detect tens of EMRBs events during its mission lifetime. For those detected events, we use the Fisher information matrix to quantify these uncertainties in the inference of their parameters. We consider the possible network of TianQin+LISA and study how a network can improve parameter estimation. The result shows that, for most sources, the CO mass, the MBH mass, and the MBH spin can be determined with an accuracy of the order $10^{-1}$ and the sky localization can be determined with an accuracy of 10 square degrees. We further explore the gravitational wave background generated by those unsolved EMRBs and conclude that it is about $10^6$ times weaker than TianQin's sensitivity and thus it can be ignored.

Incremental learning is one paradigm to enable model building and updating at scale with streaming data. For end-to-end automatic speech recognition (ASR) tasks, the absence of human annotated labels along with the need for privacy preserving policies for model building makes it a daunting challenge. Motivated by these challenges, in this paper we use a cloud based framework for production systems to demonstrate insights from privacy preserving incremental learning for automatic speech recognition (ILASR). By privacy preserving, we mean, usage of ephemeral data which are not human annotated. This system is a step forward for production levelASR models for incremental/continual learning that offers near real-time test-bed for experimentation in the cloud for end-to-end ASR, while adhering to privacy-preserving policies. We show that the proposed system can improve the production models significantly(3%) over a new time period of six months even in the absence of human annotated labels with varying levels of weak supervision and large batch sizes in incremental learning. This improvement is 20% over test sets with new words and phrases in the new time period. We demonstrate the effectiveness of model building in a privacy-preserving incremental fashion for ASR while further exploring the utility of having an effective teacher model and use of large batch sizes., Comment: 9 pages