Search

Deep Reinforcement Learning (DRL) has achieved remarkable success in solving complex decision-making problems by combining the representation capabilities of deep learning with the decision-making power of reinforcement learning. However, learning in sparse reward environments remains challenging due to insufficient feedback to guide the optimization of agents, especially in real-life environments with high-dimensional states. To tackle this issue, experience replay is commonly introduced to enhance learning efficiency through past experiences. Nonetheless, current methods of experience replay, whether based on uniform or prioritized sampling, frequently struggle with suboptimal learning efficiency and insufficient utilization of samples. This paper proposes a novel approach, diversity-based experience replay (DBER), which leverages the deterministic point process to prioritize diverse samples in state realizations. We conducted extensive experiments on Robotic Manipulation tasks in MuJoCo, Atari games, and realistic in-door environments in Habitat. The results show that our method not only significantly improves learning efficiency but also demonstrates superior performance in sparse reward environments with high-dimensional states, providing a simple yet effective solution for this field.

Nowadays transformer-based Large Language Models (LLM) for code generation tasks usually apply sampling and filtering pipelines. Due to the sparse reward problem in code generation tasks caused by one-token incorrectness, transformer-based models will sample redundant programs till they find a correct one, leading to low efficiency. To overcome the challenge, we incorporate Experience Replay (ER) in the fine-tuning phase, where codes and programs produced are stored and will be replayed to give the LLM agent a chance to learn from past experiences. Based on the spirit of ER, we introduce a novel approach called BTP pipeline which consists of three phases: beam search sampling, testing phase, and prioritized experience replay phase. The approach makes use of failed programs collected by code models and replays programs with high Possibility and Pass-rate Prioritized value (P2Value) from the replay buffer to improve efficiency. P2Value comprehensively considers the possibility of transformers' output and pass rate and can make use of the redundant resources caused by the problem that most programs collected by LLMs fail to pass any tests. We empirically apply our approach in several LLMs, demonstrating that it enhances their performance in code generation tasks and surpasses existing baselines.

For radio transients, an inverted spectrum (defined as $\alpha > 0$ for a power law spectrum $S_{\nu}\propto \nu^{\alpha}$) constrains physical properties, which in principle can be a useful criterion for selecting specific targets of interest in a transient search. To test and develop this concept, we have searched epoch 1 of the Very Large Array Sky Survey (VLASS; 3.0 GHz) and the VLITE Commensal Sky Survey (VCSS; 340 MHz) for radio transients with inverted spectra. We discover a sample of 22 inverted-spectra transient candidates that are not associated with cataloged active galactic nuclei (AGNs). To the best of our knowledge, none of our candidates have transient counterparts at other wavelengths, and only one has previously been reported as a radio transient. We find that our candidates evolve slowly over years and show either highly inverted spectra or peaked spectra over $\sim$1--3 GHz. Within our sample, nine candidates are matched to optical centers of galaxies and have estimated radio spectral luminosities of $L_{3.0\mathrm{GHz}}\sim10^{30}-10^{33}\,\mathrm{erg}\,\mathrm{s}^{-1}\,\mathrm{Hz}^{-1}$. Based on the observed properties, we find the most plausible transient classification for our candidates to be relativistic tidal disruption events. However, we are unable to fully rule out variable or transient AGNs with highly inverted spectra. Upon examining physical constraints, we confirm that mainly relativistic transients (on-axis or off-axis) with equipartition energy $E_{\mathrm{eq}}\gtrsim10^{49}-10^{53}\,\mathrm{erg}$ are expected from searching VLASS and VCSS based on inverted spectra. The obtainable physical constraints, however, can be weak due to degeneracy introduced by viewing angle., Comment: 34 pages, 10 figures, 6 tables. Submitted to ApJ. Comments welcome!

This paper studies a robust stochastic control problem with a monotone mean-variance cost functional and random coefficients. The main technique is to find the saddle point through two backward stochastic differential equations (BSDEs) with unbounded coefficients. We further show that the robust stochastic control problem shares the same optimal control and optimal value with the stochastic control problem with a mean-variance cost functional. The results obtained are then applied to monotone mean-variance and mean-variance portfolio selection problems and monotone mean-variance and mean-variance investment-reinsurance problems., Comment: arXiv admin note: text overlap with arXiv:2212.14188 by other authors

In this paper, we study non-homogeneous stochastic linear-quadratic (LQ) optimal control problems with multi-dimensional state and regime switching. We focus on the corresponding stochastic Riccati equation, which is the same as that one in homogeneous stochastic LQ optimal control problem, and the adjoint backward stochastic differential equation (BSDE), which arises from the non-homogeneous terms in the state equation and cost functional. Both stochastic Riccati equation and adjoint BSDE are solved by the contraction mapping method, and are used to represent the closed-loop optimal control and the optimal value of our problems.

As Graph Neural Networks (GNNs) become popular, libraries like PyTorch-Geometric (PyG) and Deep Graph Library (DGL) are proposed; these libraries have emerged as the de facto standard for implementing GNNs because they provide graph-oriented APIs and are purposefully designed to manage the inherent sparsity and irregularity in graph structures. However, these libraries show poor scalability on multi-core processors, which under-utilizes the available platform resources and limits the performance. This is because GNN training is a resource-intensive workload with high volume of irregular data accessing, and existing libraries fail to utilize the memory bandwidth efficiently. To address this challenge, we propose ARGO, a novel runtime system for GNN training that offers scalable performance. ARGO exploits multi-processing and core-binding techniques to improve platform resource utilization. We further develop an auto-tuner that searches for the optimal configuration for multi-processing and core-binding. The auto-tuner works automatically, making it completely transparent from the user. Furthermore, the auto-tuner allows ARGO to adapt to various platforms, GNN models, datasets, etc. We evaluate ARGO on two representative GNN models and four widely-used datasets on two platforms. With the proposed autotuner, ARGO is able to select a near-optimal configuration by exploring only 5% of the design space. ARGO speeds up state-of-the-art GNN libraries by up to 5.06x and 4.54x on a four-socket Ice Lake machine with 112 cores and a two-socket Sapphire Rapids machine with 64 cores, respectively. Finally, ARGO can seamlessly integrate into widely-used GNN libraries (e.g., DGL, PyG) with few lines of code and speed up GNN training., Comment: To appear in IEEE International Parallel and Distributed Processing Symposium (IPDPS) 2024

In recent times, there is an increased interest in the identification and re-identification of people at long distances, such as from rooftop cameras, UAV cameras, street cams, and others. Such recognition needs to go beyond face and use whole-body markers such as gait. However, datasets to train and test such recognition algorithms are not widely prevalent, and fewer are labeled. This paper introduces DIOR -- a framework for data collection, semi-automated annotation, and also provides a dataset with 14 subjects and 1.649 million RGB frames with 3D/2D skeleton gait labels, including 200 thousands frames from a long range camera. Our approach leverages advanced 3D computer vision techniques to attain pixel-level accuracy in indoor settings with motion capture systems. Additionally, for outdoor long-range settings, we remove the dependency on motion capture systems and adopt a low-cost, hybrid 3D computer vision and learning pipeline with only 4 low-cost RGB cameras, successfully achieving precise skeleton labeling on far-away subjects, even when their height is limited to a mere 20-25 pixels within an RGB frame. On publication, we will make our pipeline open for others to use.

This paper deals with the long time behavior of the optimal solution of stochastic backward linear-quadratic optimal control problem over the finite time horizon. Both weak and strong turnpike properties are established under appropriate conditions, including stabilizability condition. The key ingredients are to formulate the corresponding static optimization problem and determine the correction processes. However, our techniques are quite different from stochastic (forward) linear-quadratic case.

Implicit Neural Representations (INRs) have recently exhibited immense potential in the field of scientific visualization for both data generation and visualization tasks. However, these representations often consist of large multi-layer perceptrons (MLPs), necessitating millions of operations for a single forward pass, consequently hindering interactive visual exploration. While reducing the size of the MLPs and employing efficient parametric encoding schemes can alleviate this issue, it compromises generalizability for unseen parameters, rendering it unsuitable for tasks such as temporal super-resolution. In this paper, we introduce HyperINR, a novel hypernetwork architecture capable of directly predicting the weights for a compact INR. By harnessing an ensemble of multiresolution hash encoding units in unison, the resulting INR attains state-of-the-art inference performance (up to 100x higher inference bandwidth) and can support interactive photo-realistic volume visualization. Additionally, by incorporating knowledge distillation, exceptional data and visualization generation quality is achieved, making our method valuable for real-time parameter exploration. We validate the effectiveness of the HyperINR architecture through a comprehensive ablation study. We showcase the versatility of HyperINR across three distinct scientific domains: novel view synthesis, temporal super-resolution of volume data, and volume rendering with dynamic global shadows. By simultaneously achieving efficiency and generalizability, HyperINR paves the way for applying INR in a wider array of scientific visualization applications.

This paper investigates the stochastic linear-quadratic (LQ, for short) optimal control problems with non-Markovian regime switching in a finite time horizon where the state equation is multi-dimensional. Similar to the classical stochastic LQ problems, we establish the relationship between the stochastic LQ optimal control problems with non-Markovian regime switching and the related extended stochastic Riccati equations. To solve the extended stochastic Riccati equations, we construct a monotone Piccard iterative sequence and present the link bewteen this sequence and solutions of a family of forward-backward stochastic differential equations. Relying on $L^p$ estimates for FBSDEs, we show that the extended stochastic Riccati equation has a solution. This partially addresses one question left in Hu et al. (Ann. Appl. Probab. 32(1): 426-460, 2022). Finally, the stochastic LQ optimal control problems with non-Markovian regime switching is solved.

The exact nature of the luminous fast blue optical transient AT 2018cow is still debated. In this first of a two-paper series, we present a detailed analysis of three Hubble Space Telescope (HST) observations of AT 2018cow covering $\sim$50--60 days post-discovery in combination with other observations throughout the first two months, and derive significantly improved constraints of the late thermal properties. By modeling the spectral energy distributions (SEDs), we confirm that the UV-optical emission over 50--60 days was still a smooth blackbody (i.e., optically thick) with a high temperature ($T_{\mathrm{BB}}\sim15000\,\mathrm{K}$) and small radius ($R_{\mathrm{BB}}\lesssim1000\,R_\odot$). Additionally, we report for the first time a break in the bolometric light curve: the thermal luminosity initially declined at a rate of $L_{\mathrm{BB}}\propto t^{-2.40}$, but faded much faster at $t^{-3.06}$ after day 13. Reexamining possible late-time power sources, we disfavor significant contributions from radioactive decay based on the required $^{56}$Ni mass and lack of UV line blanketing in the HST SEDs. We argue that the commonly proposed interaction with circumstellar material may face significant challenges in explaining the late thermal properties, particularly the effects of the optical depth. Alternatively, we find that continuous outflow/wind driven by a central engine can still reasonably explain the combination of a receding photosphere, optically thick and rapidly fading emission, and intermediate-width lines. However, the rapid fading may have further implications on the power output and structure of the system. Our findings may support the hypothesis that AT 2018cow and other "Cow-like transients" are powered mainly by accretion onto a central engine., Comment: 22 pages, 10 figures. This is paper I of a two-paper series. v2: updated to the accepted version

In this second of a two-paper series, we present a detailed analysis of three HST observations taken $\sim$2--4 years post-discovery, examining the evolution of a UV-bright underlying source at the precise position of AT 2018cow. While observations at $\sim$2--3 years post-discovery revealed an exceptionally blue ($L_\nu\propto \nu^{1.99}$) underlying source with relatively stable optical brightness, fading in the NUV was observed at year 4, indicating flattening in the spectrum (to $L_\nu\propto \nu^{1.64}$). The resulting spectral energy distributions can be described by an extremely hot but small blackbody, and the fading may be intrinsic (cooling) or extrinsic (increased absorption). Considering possible scenarios and explanations, we disfavor significant contributions from stellar sources and dust formation based on the observed color and brightness. By comparing the expected power and the observed luminosity, we rule out interaction with the known radio-producing circumstellar material as well as magnetar spin down with $B\sim10^{15}\,\mathrm{G}$ as possible power sources, though we cannot rule out the possible existence of a denser CSM component (e.g., previously ejected hydrogen envelope) or a magnetar with $B\lesssim10^{14}\,\mathrm{G}$. Finally, we find that a highly-inclined precessing accretion disk can reasonably explain the color, brightness, and evolution of the underlying source. However, a major uncertainty in this scenario is the mass of the central black hole (BH), as both stellar-mass and intermediate-mass BHs face notable challenges that cannot be explained by our simple disk model, and further observations and theoretical works are needed to fully constrain the nature of this underlying source., Comment: 24 pages, 12 figures. This is paper II of a two-paper series. v2: updated to the accepted version. Note a mistake in v1 in the caption of Figure 6 is now fixed: removed the word "dereddened" because the plotted observed magnitudes are directly from Table 1 which are not extinction-corrected (the analyses are not affected by this)

A fundamental challenge in robot perception is the coupling of the sensor pose and robot pose. This has led to research in active vision where robot pose is changed to reorient the sensor to areas of interest for perception. Further, egomotion such as jitter, and external effects such as wind and others affect perception requiring additional effort in software such as image stabilization. This effect is particularly pronounced in micro-air vehicles and micro-robots who typically are lighter and subject to larger jitter but do not have the computational capability to perform stabilization in real-time. We present a novel microelectromechanical (MEMS) mirror LiDAR system to change the field of view of the LiDAR independent of the robot motion. Our design has the potential for use on small, low-power systems where the expensive components of the LiDAR can be placed external to the small robot. We show the utility of our approach in simulation and on prototype hardware mounted on a UAV. We believe that this LiDAR and its compact movable scanning design provide mechanisms to decouple robot and sensor geometry allowing us to simplify robot perception. We also demonstrate examples of motion compensation using IMU and external odometry feedback in hardware., Comment: This paper is published in IEEE Transactions on Robotics

This study verified the effectiveness of Donald Trump's Twitter campaign in guiding agen-da-setting and deflecting political risk and examined Trump's Twitter communication strategy and explores the communication effects of his tweet content during Covid-19 pandemic. We collected all tweets posted by Trump on the Twitter platform from January 1, 2020 to December 31, 2020.We used Ordinary Least Squares (OLS) regression analysis with a fixed effects model to analyze the existence of the Twitter strategy. The correlation between the number of con-firmed daily Covid-19 diagnoses and the number of particular thematic tweets was investigated using time series analysis. Empirical analysis revealed Twitter's strategy is used to divert public attention from negative Covid-19 reports during the epidemic, and it posts a powerful political communication effect on Twitter. However, findings suggest that Trump did not use false claims to divert political risk and shape public opinion., Comment: 14 pages, 6 tables

We present observations of an extreme radio flare, VT J024345.70-284040.08, hereafter VT J0243, from the nucleus of a galaxy with evidence for historic Seyfert activity at redshift $z=0.074$. Between NRAO VLA Sky Survey observations in 1993 to VLA Sky Survey observations in 2018, VT J0243 rose from a ${\sim}$GHz radio luminosity of $\nu L_\nu \lesssim 10^{38}$ erg s$^{-1}$ to $\nu L_\nu{\sim}10^{40}$ erg s$^{-1}$, and still continues to brighten. The radio spectral energy distribution (SED) evolution is consistent with a nascent jet that has slowed over ${\sim}3000$ days with an average $0.1 < \langle \beta \rangle < 0.6$. The jet is energetic (${\sim}10^{51-52}$ erg), and had a radius ${\sim}0.7$ pc in Dec. 2021. X-ray observations suggest a persistent or evolving corona, possibly associated with an accretion disk, and IR and optical observations constrain any high-energy counterpart to be sub-Eddington. VT J0243 may be an example of a young, off-axis radio jet from a slowly evolving tidal disruption event. Other more mysterious triggers for the accretion enhancement and jet launching are possible. In either case, VT J0243 is a unique example of a nascent jet, highlighting the unknown connection between supermassive black holes, the properties of their accretion flows, and jet launching., Comment: 20 pages, 5 figures, 3 tables. Submitted to ApJ

Sepsis is a life-threatening medical emergency, which is a major cause of death worldwide and the second highest cause of mortality in the United States. Researching the optimal control treatment or intervention strategy on the comprehensive sepsis system is key in reducing mortality. For this purpose, first, this paper improves a complex nonlinear sepsis model proposed in our previous work. Then, bifurcation analyses are conducted for each sepsis subsystem to study the model behaviors under some system parameters. The bifurcation analysis results also further indicate the necessity of control treatment and intervention therapy. If the sepsis system is without adding any control under some parameter and initial system value settings, the system will perform persistent inflammation outcomes as time goes by. Therefore, we develop our complex improved nonlinear sepsis model into a sepsis optimal control model, and then use some effective biomarkers recommended in existing clinic practices as optimization objective function to measure the development of sepsis. Besides that, a Bayesian optimization algorithm by combining Recurrent neural network (RNN-BO algorithm) is introduced to predict the optimal control strategy for the studied sepsis optimal control system. The difference between the RNN-BO algorithm from other optimization algorithms is that once given any new initial system value setting (initial value is associated with the initial conditions of patients), the RNN-BO algorithm is capable of quickly predicting a corresponding time-series optimal control based on the historical optimal control data for any new sepsis patient. To demonstrate the effectiveness and efficiency of the RNN-BO algorithm on solving the optimal control solution on the complex nonlinear sepsis system, some numerical simulations are implemented by comparing with other optimization algorithms in this paper., Comment: 25 pages, 7 figures, 1 table

Bayesian Optimization algorithm has become a promising approach for nonlinear global optimization problems and many machine learning applications. Over the past few years, improvements and enhancements have been brought forward and they have shown some promising results in solving the complex dynamic problems, systems of ordinary differential equations where the objective functions are computationally expensive to evaluate. Besides, the straightforward implementation of the Bayesian Optimization algorithm performs well merely for optimization problems with 10-20 dimensions. The study presented in this paper proposes a new high dimensional Bayesian Optimization algorithm combining Recurrent neural networks, which is expected to predict the optimal solution for the global optimization problems with high dimensional or time series decision models. The proposed RNN-BO algorithm can solve the optimal control problems in the lower dimension space and then learn from the historical data using the recurrent neural network to learn the historical optimal solution data and predict the optimal control strategy for any new initial system value setting. In addition, accurately and quickly providing the optimal control strategy is essential to effectively and efficiently control the epidemic spread while minimizing the associated financial costs. Therefore, to verify the effectiveness of the proposed algorithm, computational experiments are carried out on a deterministic SEIR epidemic model and a stochastic SIS optimal control model. Finally, we also discuss the impacts of different numbers of the RNN layers and training epochs on the trade-off between solution quality and related computational efforts., Comment: 16 pages, 9 figures, 2 tables

In this paper, we study multi-dimensional reflected backward stochastic differential equations with diagonally quadratic generators. Using the comparison theorem for diagonally quadratic BSDEs which is established recently in [14], we obtain the existence and uniqueness of a solution by a penalization method. Moreover, we provide a comparison theorem.

The present paper is devoted to study multi-dimensional coupled FBSDEs with diagonally quadratic generators. Relying on a comparison result obtained in [20], we provide conditions under which there exists a global solution. As a byproduct, we further give a comparison result for this global solution.