Search

Evaluating policies using off-policy data is crucial for applying reinforcement learning to real-world problems such as healthcare and autonomous driving. Previous methods for off-policy evaluation (OPE) generally suffer from high variance or irreducible bias, leading to unacceptably high prediction errors. In this work, we introduce STAR, a framework for OPE that encompasses a broad range of estimators -- which include existing OPE methods as special cases -- that achieve lower mean squared prediction errors. STAR leverages state abstraction to distill complex, potentially continuous problems into compact, discrete models which we call abstract reward processes (ARPs). Predictions from ARPs estimated from off-policy data are provably consistent (asymptotically correct). Rather than proposing a specific estimator, we present a new framework for OPE and empirically demonstrate that estimators within STAR outperform existing methods. The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases., Comment: Accepted at the Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024)

Novel reinforcement learning algorithms, or improvements on existing ones, are commonly justified by evaluating their performance on benchmark environments and are compared to an ever-changing set of standard algorithms. However, despite numerous calls for improvements, experimental practices continue to produce misleading or unsupported claims. One reason for the ongoing substandard practices is that conducting rigorous benchmarking experiments requires substantial computational time. This work investigates the sources of increased computation costs in rigorous experiment designs. We show that conducting rigorous performance benchmarks will likely have computational costs that are often prohibitive. As a result, we argue for using an additional experimentation paradigm to overcome the limitations of benchmarking., Comment: 19 pages, 13 figures, The Forty-first International Conference on Machine Learning (ICML 2024)

Perfect colouring of isonemal fabrics by thin and thick striping of warp and weft with more than two colours is examined where the cells with warps and wefts of the same colour do not appear along diagonal lines (not twilly redundancy). In species 33 to 39, all fabrics of specific orders (linear periods) can be perfectly coloured by thin striping with satin redundancy. In fewer species, all fabrics of specific orders can be perfectly coloured by thick striping in two different ways with doubled satin redundancy. The isonemal fabric 6-1-1 can be used as a redundancy configuration. All these colourings allow the coloured weaving of flat tori and -- a few of them -- cubes.

We present ICU-Sepsis, an environment that can be used in benchmarks for evaluating reinforcement learning (RL) algorithms. Sepsis management is a complex task that has been an important topic in applied RL research in recent years. Therefore, MDPs that model sepsis management can serve as part of a benchmark to evaluate RL algorithms on a challenging real-world problem. However, creating usable MDPs that simulate sepsis care in the ICU remains a challenge due to the complexities involved in acquiring and processing patient data. ICU-Sepsis is a lightweight environment that models personalized care of sepsis patients in the ICU. The environment is a tabular MDP that is widely compatible and is challenging even for state-of-the-art RL algorithms, making it a valuable tool for benchmarking their performance. However, we emphasize that while ICU-Sepsis provides a standardized environment for evaluating RL algorithms, it should not be used to draw conclusions that guide medical practice., Comment: Reinforcement Learning Conference 2024

Modern transient surveys now routinely discover flares resulting from tidal disruption events (TDEs) which occur when stars, typically $\sim0.5-2$ M$_{\odot}$, are ripped apart after passing too close to a supermassive black hole. We present three examples of a new class of extreme nuclear transients (ENTs) that we interpret as the tidal disruption of intermediate mass ($\sim3-10$ M$_{\odot}$) stars. Each is coincident with their host-galaxy nucleus and exhibits a smooth ($<10$% excess variability), luminous ($2-7\times10^{45}$ erg s$^{-1}$), and long-lived ($>150$ days) flare. ENTs are extremely rare ($\geq1\times10^{-3}$ Gpc$^{-1}$ yr$^{-1}$) compared to any other known class of transients. They are at least twice as energetic ($0.5-2.5\times 10^{53}$ erg) as any other known transient and these extreme energetics rule out stellar origins., Comment: Submitted to Science

We present detailed radio observations of the tidal disruption event (TDE) ASASSN-19bt/AT2019ahk, obtained with the Australia Telescope Compact Array (ATCA), the Atacama Large Millimeter/submillimeter Array (ALMA), and the MeerKAT radio telescopes, spanning 40 to 1464 days after the onset of the optical flare. We find that ASASSN-19bt displays unusual radio evolution compared to other TDEs, as the peak brightness of its radio emission increases rapidly until 457 days post-optical discovery and then plateaus. Using a generalized approach to standard equipartition techniques, we estimate the energy and corresponding physical parameters for two possible emission geometries: a non-relativistic spherical outflow and a relativistic outflow observed from an arbitrary viewing angle. We find that the non-relativistic solution implies a continuous energy rise in the outflow from $E\sim10^{46}$ erg to $E\sim10^{49}$ erg with $\beta \approx 0.05$, while the off-axis relativistic jet solution instead suggests $E\approx10^{52}$ erg with $\Gamma\sim10$ erg at late times in the maximally off-axis case. We find that neither model provides a holistic explanation for the origin and evolution of the radio emission, emphasizing the need for more complex models. ASASSN-19bt joins the population of TDEs that display unusual radio emission at late times. Conducting long-term radio observations of these TDEs, especially during the later phases, will be crucial for understanding how these types of radio emission in TDEs are produced., Comment: 25 pages. Submitted to ApJ

We report the discovery and characterisation of TIC 350842552 ("Zvrk"), an apparently isolated, rapidly-rotating ($P_\text{rot} \sim 99\ \mathrm{d}$) red giant observed by TESS in its Southern Continuous Viewing Zone. The star's fast surface rotation is independently verified by the use of p-mode asteroseismology, strong periodicity in TESS and ASAS-SN photometry, and measurements of spectroscopic rotational broadening. A two-component fit to APOGEE spectra indicates a coverage fraction of its surface features consistent with the amplitude of the photometric rotational signal. Variations in the amplitude of its photometric modulations over time suggest the evolution of its surface morphology, and therefore enhanced magnetic activity. We further develop and deploy new asteroseismic techniques to characterise radial differential rotation, and find weak evidence for rotational shear within Zvrk's convective envelope. This feature, in combination with such a high surface rotation rate, is incompatible with models of angular-momentum transport in single-star evolution. Spectroscopic abundance estimates also indicate a high lithium abundance, among other chemical anomalies. Taken together, all of these suggest a planet-ingestion scenario for the formation of this rotational configuration, various models for which we examine in detail., Comment: 31 pages, 17 figures. Accepted for publication in ApJ

Binaries containing a compact object orbiting a supermassive black hole are thought to be precursors of gravitational wave events, but their identification has been extremely challenging. Here, we report quasi-periodic variability in X-ray absorption which we interpret as quasi-periodic outflows (QPOuts) from a previously low-luminosity active galactic nucleus after an outburst, likely caused by a stellar tidal disruption. We rule out several models based on observed properties and instead show using general relativistic magnetohydrodynamic simulations that QPOuts, separated by roughly 8.3 days, can be explained with an intermediate-mass black hole secondary on a mildly eccentric orbit at a mean distance of about 100 gravitational radii from the primary. Our work suggests that QPOuts could be a new way to identify intermediate/extreme-mass ratio binary candidates., Comment: Accepted for publication in Science Advances. We report a new supermassive black hole phenomenon that we call quasi-periodic outflows (QPOuts)

In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in unexpected credit assignment to preceding states. From this investigation emerges the concept of a novel value function, which we refer to as the \emph{bidirectional value function}. Unlike traditional state value functions, bidirectional value functions account for both future expected returns (rewards anticipated from the current state onward) and past expected returns (cumulative rewards from the episode's start to the present). We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation. In particular, our results indicate that the proposed learning approach can, in certain challenging contexts, perform policy evaluation more rapidly than TD($\lambda$) -- a method that learns forward value functions, $v^\pi$, \emph{directly}. Overall, our findings present a new perspective on eligibility traces and potential advantages associated with the novel value function it inspires, especially for policy evaluation., Comment: Accepted in The 38th Annual AAAI Conference on Artificial Intelligence

Designing reward functions for efficiently guiding reinforcement learning (RL) agents toward specific behaviors is a complex task. This is challenging since it requires the identification of reward structures that are not sparse and that avoid inadvertently inducing undesirable behaviors. Naively modifying the reward structure to offer denser and more frequent feedback can lead to unintended outcomes and promote behaviors that are not aligned with the designer's intended goal. Although potential-based reward shaping is often suggested as a remedy, we systematically investigate settings where deploying it often significantly impairs performance. To address these issues, we introduce a new framework that uses a bi-level objective to learn \emph{behavior alignment reward functions}. These functions integrate auxiliary rewards reflecting a designer's heuristics and domain knowledge with the environment's primary rewards. Our approach automatically determines the most effective way to blend these types of feedback, thereby enhancing robustness against heuristic reward misspecification. Remarkably, it can also adapt an agent's policy optimization process to mitigate suboptimalities resulting from limitations and biases inherent in the underlying RL algorithms. We evaluate our method's efficacy on a diverse set of tasks, from small-scale experiments to high-dimensional control challenges. We investigate heuristic auxiliary rewards of varying quality -- some of which are beneficial and others detrimental to the learning process. Our results show that our framework offers a robust and principled way to integrate designer-specified heuristics. It not only addresses key shortcomings of existing approaches but also consistently leads to high-performing solutions, even when given misaligned or poorly-specified auxiliary reward functions., Comment: (Spotlight) Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)

Representation learning is increasingly employed to generate representations that are predictive across multiple downstream tasks. The development of representation learning algorithms that provide strong fairness guarantees is thus important because it can prevent unfairness towards disadvantaged groups for all downstream prediction tasks. To prevent unfairness towards disadvantaged groups in all downstream tasks, it is crucial to provide representation learning algorithms that provide fairness guarantees. In this paper, we formally define the problem of learning representations that are fair with high confidence. We then introduce the Fair Representation learning with high-confidence Guarantees (FRG) framework, which provides high-confidence guarantees for limiting unfairness across all downstream models and tasks, with user-defined upper bounds. After proving that FRG ensures fairness for all downstream models and tasks with high probability, we present empirical evaluations that demonstrate FRG's effectiveness at upper bounding unfairness for multiple downstream models and tasks.

We demonstrate an approach to two-dimensional electronic spectroscopy (2DES) that combines the benefits of shot-to-shot detection at high-repetition rates with the simplicity of a broadband white light continuum input and conventional optical elements to generate phase-locked pump pulse pairs. We demonstrate this through mutual synchronization between the laser repetition rate, acousto-optical deflector (AOD), pump delay stage and the CCD line camera, which allows rapid scanning of pump optical delay synchronously with the laser repetition rate while the delay stage is moved at a constant velocity. The resulting shot-to-shot detection scheme is repetition rate scalable and only limited by the CCD line rate and the maximum stage velocity. Using this approach, we demonstrate measurement of an averaged 2DES absorptive spectrum in as much as 1.2 seconds of continuous sample exposure per 2D spectrum. We achieve a signal-to-noise ratio (SNR) of 6.8 for optical densities down to 0.05 with 11.6 seconds of averaging at 100 kHz laser repetition rate. Combining rapid scanning of mechanical delay lines with shot-to-shot detection as demonstrated here provides a viable alternative to acousto-optic pulse shaping (AOPS) approaches that is repetition-rate scalable, has comparable throughput and sensitivity, and minimizes sample exposure per 2D spectrum with promising micro-spectroscopy applications., Comment: 27 pages, 6 figures

The analysis of core-collapse supernova (CCSN) environments can provide important information on the life cycle of massive stars and constrain the progenitor properties of these powerful explosions. The MUSE instrument at the VLT enables detailed local environment constraints of the progenitors of large samples of CCSNe. Using a homogeneous SN sample from the ASAS-SN survey has enabled us to perform a minimally biased statistical analysis of CCSN environments. We analyze 111 galaxies observed by MUSE that hosted 112 CCSNe detected or discovered by the ASAS-SN survey between 2014 and 2018. The majority of the galaxies were observed by the the AMUSING survey. Here we analyze the immediate environment around the SN locations and compare the properties between the different CCSN types and their light curves. We used stellar population synthesis and spectral fitting techniques to derive physical parameters for all HII regions detected within each galaxy, including the star formation rate (SFR), H$\alpha$ equivalent width (EW), oxygen abundance, and extinction. We found that stripped-envelope (SE) SNe occur in environments with a higher median SFR, H$\alpha$ EW, and oxygen abundances than SNe II and SNe IIn/Ibn. The distributions of SNe II and IIn are very similar, indicating that these events explode in similar environments. For the SESNe, SNe Ic have higher median SFRs, H$\alpha$ EWs, and oxygen abundances than SNe Ib. SNe IIb have environments with similar SFRs and H$\alpha$ EWs to SNe Ib, and similar oxygen abundances to SNe Ic. We also show that the postmaximum decline rate, $s$, of SNe II correlates with the H$\alpha$ EW, and that the luminosity and the $\Delta m_{15}$ parameter of SESNe correlate with the oxygen abundance, H$\alpha$ EW, and SFR at their environments. This suggests a connection between the explosion mechanisms of these events to their environment properties., Comment: 27 pages, 16 figures, accepted at A&A

For reinforcement learning systems to be widely adopted, their users must understand and trust them. We present a theoretical analysis of explaining reinforcement learning using Shapley values, following a principled approach from game theory for identifying the contribution of individual players to the outcome of a cooperative game. We call this general framework Shapley Values for Explaining Reinforcement Learning (SVERL). Our analysis exposes the limitations of earlier uses of Shapley values in reinforcement learning. We then develop an approach that uses Shapley values to explain agent performance. In a variety of domains, SVERL produces meaningful explanations that match and supplement human intuition., Comment: 12 pages, 9 figures. Accepted at ICML 2023

Model predictive control (MPC) is promising for fueling and core density feedback control in nuclear fusion tokamaks, where the primary actuators, frozen hydrogen fuel pellets fired into the plasma, are discrete. Previous density feedback control approaches have only approximated pellet injection as a continuous input due to the complexity that it introduces. In this letter, we model plasma density and pellet injection as a hybrid system and propose two MPC strategies for density control: mixed-integer (MI) MPC using a conventional mixed-integer programming (MIP) solver and MPC utilizing our novel modification of the penalty term homotopy (PTH) algorithm. By relaxing the integer requirements, the PTH algorithm transforms the MIP problem into a series of continuous optimization problems, reducing computational complexity. Our novel modification to the PTH algorithm ensures that it can handle path constraints, making it viable for constrained hybrid MPC in general. Both strategies perform well with regards to reference tracking without violating path constraints and satisfy the computation time limit for real-time control of the pellet injection system. However, the computation time of the PTH-based MPC strategy consistently outpaces the conventional MI-MPC strategy.

Coagent networks for reinforcement learning (RL) [Thomas and Barto, 2011] provide a powerful and flexible framework for deriving principled learning rules for arbitrary stochastic neural networks. The coagent framework offers an alternative to backpropagation-based deep learning (BDL) that overcomes some of backpropagation's main limitations. For example, coagent networks can compute different parts of the network \emph{asynchronously} (at different rates or at different times), can incorporate non-differentiable components that cannot be used with backpropagation, and can explore at levels higher than their action spaces (that is, they can be designed as hierarchical networks for exploration and/or temporal abstraction). However, the coagent framework is not just an alternative to BDL; the two approaches can be blended: BDL can be combined with coagent learning rules to create architectures with the advantages of both approaches. This work generalizes the coagent theory and learning rules provided by previous works; this generalization provides more flexibility for network architecture design within the coagent framework. This work also studies one of the chief disadvantages of coagent networks: high variance updates for networks that have many coagents and do not use backpropagation. We show that a coagent algorithm with a policy network that does not use backpropagation can scale to a challenging RL domain with a high-dimensional state and action space (the MuJoCo Ant environment), learning reasonable (although not state-of-the-art) policies. These contributions motivate and provide a more general theoretical foundation for future work that studies coagent networks.

Some galaxies show little to no sign of active galactic nucleus (AGN) activity, yet exhibit strong coronal emission lines (CLs) relative to common narrow emission lines. Many of these CLs have ionization potentials of $\geq 100$ eV, thus requiring strong extreme UV and/or soft X-ray flux. It has long been thought that such events are powered by tidal disruption events (TDEs), but owing to a lack of detailed multi-wavelength follow-up, such a connection has not been firmly made. Here we compare coronal line emitters (CLEs) and TDEs in terms of their host-galaxy and transient properties. We find that the mid-infrared (MIR) colors of CLE hosts in quiescence are similar to TDE hosts. Additionally, many CLEs show evidence of a large dust reprocessing echo in their mid-infrared colors, a sign of significant dust in the nucleus. The stellar masses and star formation rates of the CLE hosts are largely consistent with TDE hosts, with many CLEs residing within the green valley. The blackbody properties of CLEs and TDEs are similar, with some CLEs showing hot (T $\geq 40,000$ K) blackbody temperatures. Finally, the location of CLEs on the peak-luminosity/decline-rate parameter space is much closer to TDEs than many other major classes of nuclear transients. Combined, these provide strong evidence to confirm the previous claims that CLEs are indeed TDEs in gas-rich environments. We additionally propose a stricter threshold of CL flux $\geq 1/3$ $\times$ [O III] flux to better exclude AGNs from the sample of CLEs., Comment: Updated to match MNRAS accepted version