Search

Offline reinforcement learning has received extensive attention from scholars because it avoids the interaction between the agent and the environment by learning a policy through a static dataset. However, general reinforcement learning methods cannot get satisfactory results in offline reinforcement learning due to the out-of-distribution state actions that the dataset cannot cover during training. To solve this problem, the policy regularization method that tries to directly clone policies used in static datasets has received numerous studies due to its simplicity and effectiveness. However, policy constraint methods make the agent choose the corresponding actions in the static dataset. This type of constraint is usually over-conservative, which results in suboptimal policies, especially in low-quality static datasets. In this paper, a hypercube policy regularization framework is proposed, this method alleviates the constraints of policy constraint methods by allowing the agent to explore the actions corresponding to similar states in the static dataset, which increases the effectiveness of algorithms in low-quality datasets. It was also theoretically demonstrated that the hypercube policy regularization framework can effectively improve the performance of original algorithms. In addition, the hypercube policy regularization framework is combined with TD3-BC and Diffusion-QL for experiments on D4RL datasets which are called TD3-BC-C and Diffusion-QL-C. The experimental results of the score demonstrate that TD3-BC-C and Diffusion-QL-C perform better than state-of-the-art algorithms like IQL, CQL, TD3-BC and Diffusion-QL in most D4RL environments in approximate time.

Strong substrate reflections and complex scattering effects present significant challenges for diffraction tomography in metrology and inspection applications. To address these issues, we introduce a reflection-mode diffraction tomography technique for imaging strongly scattering samples on a reflective substrate using intensity-only measurements. Our technique leverages the modified Born series to model complex wave interactions with fast and stable convergence, further incorporating Bloch and perfect electric conductor boundary conditions for improved accuracy. The adjoint method is used for efficient gradient computation in solving the inverse problem. Validated on a reflection-mode LED array microscope, we achieve high-resolution reconstructions of dual-layer targets and phase structures through a scattering fiber layer, demonstrating the technique's potential for challenging metrology and inspection tasks.

In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy. Additionally, we also investigate the scaling laws and learning rate schedule of mixture of experts models, providing valuable insights and guidances for future model development and optimization. The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications. Codes: https://github.com/Tencent/Hunyuan-Large Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large, Comment: 17 pages, 4 Figures

Chirality is ubiquitous in nature and manifests in a wide range of phenomena including chemical reactions, biological processes, and quantum transport of electrons. In quantum materials, the chirality of fermions, given by the relative directions between the electron spin and momentum, is connected to the band topology of electronic states. Here, we show that in structurally chiral materials like CoSi, the orbital angular momentum (OAM) serves as the main driver of a nontrivial band topology in this new class of unconventional topological semimetals, even when spin-orbit coupling is negligible. A nontrivial orbital-momentum locking of multifold chiral fermions in the bulk leads to a pronounced OAM texture of the helicoid Fermi arcs at the surface. Our findings highlight the pivotal role of the orbital degree of freedom for the chirality and topology of electron states, in general, and pave the way towards the application of topological chiral semimetals in orbitronic devices.

In this paper, we study an inverse reinforcement learning problem that involves learning the reward function of a learning agent using trajectory data collected while this agent is learning its optimal policy. To address this problem, we propose an inverse reinforcement learning method that allows us to estimate the policy parameters of the learning agent which can then be used to estimate its reward function. Our method relies on a new variant of the behavior cloning algorithm, which we call bundle behavior cloning, and uses a small number of trajectories generated by the learning agent's policy at different points in time to learn a set of policies that match the distribution of actions observed in the sampled trajectories. We then use the cloned policies to train a neural network model that estimates the reward function of the learning agent. We provide a theoretical analysis to show a complexity result on bound guarantees for our method that beats standard behavior cloning as well as numerical experiments for a reinforcement learning problem that validate the proposed method.

In this paper, we address the challenge of heterogeneous data distributions in cross-silo federated learning by introducing a novel algorithm, which we term Cross-silo Robust Clustered Federated Learning (CS-RCFL). Our approach leverages the Wasserstein distance to construct ambiguity sets around each client's empirical distribution that capture possible distribution shifts in the local data, enabling evaluation of worst-case model performance. We then propose a model-agnostic integer fractional program to determine the optimal distributionally robust clustering of clients into coalitions so that possible biases in the local models caused by statistically heterogeneous client datasets are avoided, and analyze our method for linear and logistic regression models. Finally, we discuss a federated learning protocol that ensures the privacy of client distributions, a critical consideration, for instance, when clients are healthcare institutions. We evaluate our algorithm on synthetic and real-world healthcare data., Comment: 8 pages, 3 figures, Accepted to IEEE CDC 2024

Advancements in deep multi-agent reinforcement learning (MARL) have positioned it as a promising approach for decision-making in cooperative games. However, it still remains challenging for MARL agents to learn cooperative strategies for some game environments. Recently, large language models (LLMs) have demonstrated emergent reasoning capabilities, making them promising candidates for enhancing coordination among the agents. However, due to the model size of LLMs, it can be expensive to frequently infer LLMs for actions that agents can take. In this work, we propose You Only LLM Once for MARL (YOLO-MARL), a novel framework that leverages the high-level task planning capabilities of LLMs to improve the policy learning process of multi-agents in cooperative games. Notably, for each game environment, YOLO-MARL only requires one time interaction with LLMs in the proposed strategy generation, state interpretation and planning function generation modules, before the MARL policy training process. This avoids the ongoing costs and computational time associated with frequent LLMs API calls during training. Moreover, the trained decentralized normal-sized neural network-based policies operate independently of the LLM. We evaluate our method across three different environments and demonstrate that YOLO-MARL outperforms traditional MARL algorithms.

Search-based motion planning algorithms have been widely utilized for unmanned aerial vehicles (UAVs). However, deploying these algorithms on real UAVs faces challenges due to limited onboard computational resources. The algorithms struggle to find solutions in high-dimensional search spaces and require considerable time to ensure that the trajectories are dynamically feasible. This paper incorporates the lazy search concept into search-based planning algorithms to address the critical issue of real-time planning for collision-free and dynamically feasible trajectories on UAVs. We demonstrate that the lazy search motion planning algorithm can efficiently find optimal trajectories and significantly improve computational efficiency.

The core of quantum metrology lies in utilizing entanglement to enhance measurement precision beyond standard quantum limit. Here, we utilize the Floquet-engineered two-axis twisting (TAT) and turn dynamics to generate non-Gaussian states for quantum metrology. By employing both analytically semi-classical and quantum approaches, we find that the desired $N$-particle non-Gaussian state can be produced within a remarkably short time $t_\mathrm{opt}\propto \ln{N}/{N}$, and its quantum Fisher information $F^\mathrm{opt}_\mathrm{Q}\propto N^2$ approaches the Heisenberg limit. Moreover, using the Floquet-engineered anti-TAT-and-turn, we may implement an efficient interaction-based readout protocol to extract the signal encoded in this non-Gaussian state. This Floquet-engineered anti-TAT-and-turn approach offers a viable method to achieve effective time-reversal dynamics for improving measurement precision and resilience against detection noise, all without the need to invert the sign of the nonlinear interaction. This study paves the way for achieving entanglement-enhanced quantum metrology via rapid generation of cat-like states at high particle numbers through continuous Floquet engineering., Comment: 17 pages, 7 figures

A recent analysis of the LHCb data [Phys. Rev. D 105 (2022) L031503] obtained a sizable negative effective range for the $X(3872)$. This has attracted intensive discussions on whether $X(3872)$ can be deemed as a $D\bar{D}^*$ molecular state. This work explicitly demonstrates that the negative effective range of the $X(3872)$ does not contradict the molecular picture, adopting an effective field theory formulation of the $D\bar{D}^*$ interaction that can simultaneously reproduce the binding energy and effective range of the $X(3872)$. We elaborate on the implications of the large negative effective range of $X(3872)$ and the small binding energy on the underlying $D\bar{D}^*$ interaction. Such results are relevant for a better understanding of hadronic molecules and their binding mechanism., Comment: 6 pages, 4 figures; Comments and suggestions are welcome

We introduce Synthetic Playground (SynPlay), a new synthetic human dataset that aims to bring out the diversity of human appearance in the real world. We focus on two factors to achieve a level of diversity that has not yet been seen in previous works: i) realistic human motions and poses and ii) multiple camera viewpoints towards human instances. We first use a game engine and its library-provided elementary motions to create games where virtual players can take less-constrained and natural movements while following the game rules (i.e., rule-guided motion design as opposed to detail-guided design). We then augment the elementary motions with real human motions captured with a motion capture device. To render various human appearances in the games from multiple viewpoints, we use seven virtual cameras encompassing the ground and aerial views, capturing abundant aerial-vs-ground and dynamic-vs-static attributes of the scene. Through extensive and carefully-designed experiments, we show that using SynPlay in model training leads to enhanced accuracy over existing synthetic datasets for human detection and segmentation. The benefit of SynPlay becomes even greater for tasks in the data-scarce regime, such as few-shot and cross-domain learning tasks. These results clearly demonstrate that SynPlay can be used as an essential dataset with rich attributes of complex human appearances and poses suitable for model pretraining. SynPlay dataset comprising over 73k images and 6.5M human instances, is available for download at https://synplaydataset.github.io/., Comment: Project Page: https://synplaydataset.github.io/

Since the birth of quantum theory, it has been controversial that whether real numbers are adequate to describe its formalism. Recently, the imaginary unit $i$ has been experimentally proven to be indispensable for quantum mechanics. It motivates us to study the differences in detection power between real and complex entanglement witnesses (EWs), and analyze the detection power of real EWs under local equivalences. We show that a real EW must detect a real entangled state, and conversely a real entangled state must be detected by a real EW. We present a necessary and sufficient condition for the entangled states detected by real EWs, and give a specific example which implies the detection limitations of real EWs. Then, we conjecture that all entangled states are detected by the EWs locally equivalent to real ones. We prove the conjecture for all states with non-positive partial transpose. We also derive a necessary and sufficient condition for the complex PPT (positive-partial-transpose) entangled states detected by the EWs locally equivalent to real ones. We further prove the conjecture for a family of two-quqart PPT entangled states. Another way to figure out the conjecture is to check whether a counterexample exists. We propose an equivalent method to examine the existence of a counterexample from a set-theoretic perspective, and provide some supporting evidence of non-existence. Finally, we derive some results on local projections of EWs with product projectors., Comment: 17 pages, 2 figures

We determine Gr\"{o}bner bases for the presentation ideals of multi-Rees algebras and their special fiber rings. Specifically, we focus on modules that are direct sums of ideals generated by either maximal minors of a two-sided ladder matrix or unit interval determinantal ideals. Our analysis reveals that the multi-blowup algebras are Koszul Cohen--Macaulay normal domains, possess rational singularities in characteristic zero, and are F-rational in positive characteristic.

Recently, interesting empirical phenomena known as Neural Collapse have been observed during the final phase of training deep neural networks for classification tasks. We examine this issue when the feature dimension d is equal to the number of classes K. We demonstrate that two popular unconstrained feature models are strict saddle functions, with every critical point being either a global minimum or a strict saddle point that can be exited using negative curvatures. The primary findings conclusively confirm the conjecture on the unconstrained feature models in previous articles.

In this paper, we describe our approaches to TREC Real-Time Summarization of Twitter. We focus on real time push notification scenario, which requires a system monitors the stream of sampled tweets and returns the tweets relevant and novel to given interest profiles. Dirichlet score with and with very little smoothing (baseline) are employed to classify whether a tweet is relevant to a given interest profile. Using metrics including Mean Average Precision (MAP, cumulative gain (CG) and discount cumulative gain (DCG), the experiment indicates that our approach has a good performance. It is also desired to remove the redundant tweets from the pushing queue. Due to the precision limit, we only describe the algorithm in this paper., Comment: This paper was accepted to International Conference on Artificial Intelligence and Electromechanical Automation 2024

This article investigates a control system within the context of six-generation wireless networks. The control performance optimization confronts the technical challenges that arise from the intricate interactions between communication and control sub-systems, asking for a co-design. Accounting for the system dynamics, we formulate the sequential co-design decision-makings of communication and control over the discrete time horizon as a Markov decision process, for which a practical offline learning framework is proposed. Our proposed framework integrates large language models into the elements of reinforcement learning. We present a case study on the age of semantics-aware communication and control co-design to showcase the potentials from our proposed learning framework. Furthermore, we discuss the open issues remaining to make our proposed offline learning framework feasible for real-world implementations, and highlight the research directions for future explorations.

Unmanned Surface Vehicles (USVs) are pivotal in marine exploration, but their sensors' accuracy is compromised by the dynamic marine environment. Traditional calibration methods fall short in these conditions. This paper introduces a deep learning architecture that predicts changes in the USV's dynamic metacenter and refines sensors' extrinsic parameters in real time using a Time-Sequence General Regression Neural Network (GRNN) with Euler angles as input. Simulation data from Unity3D ensures robust training and testing. Experimental results show that the Time-Sequence GRNN achieves the lowest mean squared error (MSE) loss, outperforming traditional neural networks. This method significantly enhances sensor calibration for USVs, promising improved data accuracy in challenging maritime conditions. Future work will refine the network and validate results with real-world data., Comment: Accepted by The 9th Asia-Pacific Conference on Intelligent Robot Systems (ACIRS 2024)

We present a framework for diversifying human poses in a synthetic dataset for aerial-view human detection. Our method firstly constructs a set of novel poses using a pose generator and then alters images in the existing synthetic dataset to assume the novel poses while maintaining the original style using an image translator. Since images corresponding to the novel poses are not available in training, the image translator is trained to be applicable only when the input and target poses are similar, thus training does not require the novel poses and their corresponding images. Next, we select a sequence of target novel poses from the novel pose set, using Dijkstra's algorithm to ensure that poses closer to each other are located adjacently in the sequence. Finally, we repeatedly apply the image translator to each target pose in sequence to produce a group of novel pose images representing a variety of different limited body movements from the source pose. Experiments demonstrate that, regardless of how the synthetic data is used for training or the data size, leveraging the pose-diversified synthetic dataset in training generally presents remarkably better accuracy than using the original synthetic dataset on three aerial-view human detection benchmarks (VisDrone, Okutama-Action, and ICG) in the few-shot regime.

Aerial-view human detection has a large demand for large-scale data to capture more diverse human appearances compared to ground-view human detection. Therefore, synthetic data can be a good resource to expand data, but the domain gap with real-world data is the biggest obstacle to its use in training. As a common solution to deal with the domain gap, the sim2real transformation is used, and its quality is affected by three factors: i) the real data serving as a reference when calculating the domain gap, ii) the synthetic data chosen to avoid the transformation quality degradation, and iii) the synthetic data pool from which the synthetic data is selected. In this paper, we investigate the impact of these factors on maximizing the effectiveness of synthetic data in training in terms of improving learning performance and acquiring domain generalization ability--two main benefits expected of using synthetic data. As an evaluation metric for the second benefit, we introduce a method for measuring the distribution gap between two datasets, which is derived as the normalized sum of the Mahalanobis distances of all test data. As a result, we have discovered several important findings that have never been investigated or have been used previously without accurate understanding. We expect that these findings can break the current trend of either naively using or being hesitant to use synthetic data in machine learning due to the lack of understanding, leading to more appropriate use in future research.

This paper addresses the challenge of collision-free motion planning in automated navigation within complex environments. Utilizing advancements in Deep Reinforcement Learning (DRL) and sensor technologies like LiDAR, we propose the TD3-DWA algorithm, an innovative fusion of the traditional Dynamic Window Approach (DWA) with the Twin Delayed Deep Deterministic Policy Gradient (TD3). This hybrid algorithm enhances the efficiency of robotic path planning by optimizing the sampling interval parameters of DWA to effectively navigate around both static and dynamic obstacles. The performance of the TD3-DWA algorithm is validated through various simulation experiments, demonstrating its potential to significantly improve the reliability and safety of autonomous navigation systems.

This paper explores the integration of strategic optimization methods in search advertising, focusing on ad ranking and bidding mechanisms within E-commerce platforms. By employing a combination of reinforcement learning and evolutionary strategies, we propose a dynamic model that adjusts to varying user interactions and optimizes the balance between advertiser cost, user relevance, and platform revenue. Our results suggest significant improvements in ad placement accuracy and cost efficiency, demonstrating the model's applicability in real-world scenarios., Comment: Accepted by 2024 5th International Conference on Electronic communication and Artificial Intelligence (ICECAI 2024)

Although quantum entanglement is an important resource, its characterization is quite challenging. The partial transposition is a common method to detect bipartite entanglement. In this paper, the authors study the partial-transpose(PT)-moments of two-qubit states,and completely describe the whole region, composed of the second and third PT-moments, for all two-qubit states. Furthermore, they determine the accurate region corresponding to all entangled two-qubit states. The states corresponding to those boundary points of the whole region, and to the border lines between separable and entangled states are analyzed. As an application, they characterize the entangled region of PT-moments for the two families of Werner states and Bell-diagonal states. The relations between entanglement and the pairs of PT-moments are revealed from these typical examples. They also numerically plot the whole region of possible PT-moments for all two-qubit X-states, and find that this region is almost the same as the whole region of PT-moments for all two-qubit states. Moreover, they extend their results to detect the entanglement of multiqubit states. By utilizing the PT-moment-based method to characterize the entanglement of the multiqubit states mixed by the GHZ and W states, they propose an operational way of verifying the genuine entanglement in such states., Comment: 31 pages, LaTeX, 9 figures

The reduced cost and computational and calibration requirements of monocular cameras make them ideal positioning sensors for mobile robots, albeit at the expense of any meaningful depth measurement. Solutions proposed by some scholars to this localization problem involve fusing pose estimates from convolutional neural networks (CNNs) with pose estimates from geometric constraints on motion to generate accurate predictions of robot trajectories. However, the distribution of attitude estimation based on CNN is not uniform, resulting in certain translation problems in the prediction of robot trajectories. This paper proposes improving these CNN-based pose estimates by propagating a SE(3) uniform distribution driven by a particle filter. The particles utilize the same motion model used by the CNN, while updating their weights using CNN-based estimates. The results show that while the rotational component of pose estimation does not consistently improve relative to CNN-based estimation, the translational component is significantly more accurate. This factor combined with the superior smoothness of the filtered trajectories shows that the use of particle filters significantly improves the performance of CNN-based localization algorithms.

In order to solve the problem of frequent deceleration of unmanned vehicles when approaching obstacles, this article uses a Deep Q-Network (DQN) and its extension, the Double Deep Q-Network (DDQN), to develop a local navigation system that adapts to obstacles while maintaining optimal speed planning. By integrating improved reward functions and obstacle angle determination methods, the system demonstrates significant enhancements in maneuvering capabilities without frequent decelerations. Experiments conducted in simulated environments with varying obstacle densities confirm the effectiveness of the proposed method in achieving more stable and efficient path planning.

Path planning is an important problem with the the applications in many aspects, such as video games, robotics etc. This paper proposes a novel method to address the problem of Deep Reinforcement Learning (DRL) based path planning for a mobile robot. We design DRL-based algorithms, including reward functions, and parameter optimization, to avoid time-consuming work in a 2D environment. We also designed an Two-way search hybrid A* algorithm to improve the quality of local path planning. We transferred the designed algorithm to a simple embedded environment to test the computational load of the algorithm when running on a mobile robot. Experiments show that when deployed on a robot platform, the DRL-based algorithm in this article can achieve better planning results and consume less computing resources.

We propose the piecemeal quantum telescope through bit-by-bit iteration using different baselines. It improves precision exponentially with number of baselines, and it works robustly under large observation errors such as statistical errors, channel noise, operational errors and so on. For example, under the noisy channel of random phase drifts, our method can provide an exponential precision provided that the detected data is not entirely noisy. Being fault tolerant to statistical error, it requests only a small number of incident single-photons in detecting the star angle with exponential precision. As a result, it requests to detect only a few hundreds of photons from the target star for a precision breaking classical limit by 4 to 5 magnitude orders. This demonstrates a super efficiency., Comment: 6 figures

Offline reinforcement learning learns from a static dataset without interacting with environments, which ensures security and thus owns a good application prospect. However, directly applying naive reinforcement learning algorithm usually fails in an offline environment due to inaccurate Q value approximation caused by out-of-distribution (OOD) state-actions. It is an effective way to solve this problem by penalizing the Q-value of OOD state-actions. Among the methods of punishing OOD state-actions, count-based methods have achieved good results in discrete domains in a simple form. Inspired by it, a novel pseudo-count method for continuous domains called Grid-Mapping Pseudo-Count method (GPC) is proposed by extending the count-based method from discrete to continuous domains. Firstly, the continuous state and action space are mapped to discrete space using Grid-Mapping, then the Q-values of OOD state-actions are constrained through pseudo-count. Secondly, the theoretical proof is given to show that GPC can obtain appropriate uncertainty constraints under fewer assumptions than other pseudo-count methods. Thirdly, GPC is combined with Soft Actor-Critic algorithm (SAC) to get a new algorithm called GPC-SAC. Lastly, experiments on D4RL datasets are given to show that GPC-SAC has better performance and less computational cost than other algorithms that constrain the Q-value.