Search

Policy search methods are crucial in reinforcement learning, offering a framework to address continuous state-action and partially observable problems. However, the complexity of exploring vast policy spaces can lead to significant inefficiencies. Reducing the policy space through policy compression emerges as a powerful, reward-free approach to accelerate the learning process. This technique condenses the policy space into a smaller, representative set while maintaining most of the original effectiveness. Our research focuses on determining the necessary sample size to learn this compressed set accurately. We employ R\'enyi divergence to measure the similarity between true and estimated policy distributions, establishing error bounds for good approximations. To simplify the analysis, we employ the $l_1$ norm, determining sample size requirements for both model-based and model-free settings. Finally, we correlate the error bounds from the $l_1$ norm with those from R\'enyi divergence, distinguishing between policies near the vertices and those in the middle of the policy space, to determine the lower and upper bounds for the required sample sizes.

Achieving the no-regret property for Reinforcement Learning (RL) problems in continuous state and action-space environments is one of the major open problems in the field. Existing solutions either work under very specific assumptions or achieve bounds that are vacuous in some regimes. Furthermore, many structural assumptions are known to suffer from a provably unavoidable exponential dependence on the time horizon $H$ in the regret, which makes any possible solution unfeasible in practice. In this paper, we identify local linearity as the feature that makes Markov Decision Processes (MDPs) both learnable (sublinear regret) and feasible (regret that is polynomial in $H$). We define a novel MDP representation class, namely Locally Linearizable MDPs, generalizing other representation classes like Linear MDPs and MDPS with low inherent Belmman error. Then, i) we introduce Cinderella, a no-regret algorithm for this general representation class, and ii) we show that all known learnable and feasible MDP families are representable in this class. We first show that all known feasible MDPs belong to a family that we call Mildly Smooth MDPs. Then, we show how any mildly smooth MDP can be represented as a Locally Linearizable MDP by an appropriate choice of representation. This way, Cinderella is shown to achieve state-of-the-art regret bounds for all previously known (and some new) continuous MDPs for which RL is learnable and feasible.

The measurement of cosmic rays at energies exceeding 100 TeV per nucleon is crucial for enhancing the understanding of high-energy particle propagation and acceleration models in the Galaxy. HERD is a space-borne calorimetric experiment that aims to extend the current direct measurements of cosmic rays to unexplored energies. The payload is scheduled to be installed on the Chinese Space Station in 2027. The primary peculiarity of the instrument is its capability to measure particles coming from all directions, with the main detector being a deep, homogeneous, 3D calorimeter. The active elements are read out using two independent systems: one based on wavelength shifter fibers coupled to CMOS cameras, and the other based on photo-diodes read-out with custom front-end electronics. A large calorimeter prototype was tested in 2023 during an extensive beam test campaign at CERN. In this paper, the performance of the calorimeter for high-energy electron beams, as obtained from the photo-diode system data, is presented. The prototype demonstrated excellent performance, e.g., an energy resolution better than 1% for electrons at 250 GeV. A comparison between beam test data and Monte Carlo simulation data is also presented.

This paper presents the second-place methodology in the Volvo Discovery Challenge at ECML-PKDD 2024, where we used Long Short-Term Memory networks and pseudo-labeling to predict maintenance needs for a component of Volvo trucks. We processed the training data to mirror the test set structure and applied a base LSTM model to label the test data iteratively. This approach refined our model's predictive capabilities and culminated in a macro-average F1-score of 0.879, demonstrating robust performance in predictive maintenance. This work provides valuable insights for applying machine learning techniques effectively in industrial settings., Comment: 2nd place at ECML-PKDD Discovery Challenge https://www.hh.se/english/about-the-university/events/discovery-challenge-ecml-pkdd-2024.html

Electrical resistivity tomography is a suitable technique for non-invasive monitoring of municipal solid waste landfills, but accurate sensitivity analysis is necessary to evaluate the effectiveness and reliability of geoelectrical investigations and to properly design data acquisition. Commonly, a thin high-resistivity membrane in placed underneath the waste to prevent leachate leakage. In the construction of a numerical framework for sensitivity computation, taking into account the actual dimensions of the electrodes and, in particular, of the membrane, can lead to extremely high computational costs. In this work, we present a novel approach for numerically computing sensitivity effectively by adopting a mixed-dimensional framework, where the membrane is approximated as a 2D object and the electrodes as 1D objects. The code is first validated against analytical expressions for simple 4-electrode arrays and a homogeneous medium. It is then tested in simplified landfill models, where a 2D box-shaped liner separates the landfill body from the surrounding media, and 48 electrodes are used. The results show that electrodes arranged linearly along both sides of the perimeter edges of the box-shaped liner are promising for detecting liner damage, with sensitivity increasing by 2-3 orders of magnitude, even for damage as small as one-sixth of the electrode spacing in diameter. Good results are also obtained when simulating an electrical connection between the landfill and the surrounding media that is not due to liner damage. The next steps involve evaluating the minimum number of configurations needed to achieve suitable sensitivity with a manageable field effort and validating the modeling results with downscaled laboratory tests.

Constrained Reinforcement Learning (CRL) tackles sequential decision-making problems where agents are required to achieve goals by maximizing the expected return while meeting domain-specific constraints, which are often formulated as expected costs. In this setting, policy-based methods are widely used since they come with several advantages when dealing with continuous-control problems. These methods search in the policy space with an action-based or parameter-based exploration strategy, depending on whether they learn directly the parameters of a stochastic policy or those of a stochastic hyperpolicy. In this paper, we propose a general framework for addressing CRL problems via gradient-based primal-dual algorithms, relying on an alternate ascent/descent scheme with dual-variable regularization. We introduce an exploration-agnostic algorithm, called C-PG, which exhibits global last-iterate convergence guarantees under (weak) gradient domination assumptions, improving and generalizing existing results. Then, we design C-PGAE and C-PGPE, the action-based and the parameter-based versions of C-PG, respectively, and we illustrate how they naturally extend to constraints defined in terms of risk measures over the costs, as it is often requested in safety-critical scenarios. Finally, we numerically validate our algorithms on constrained control problems, and compare them with state-of-the-art baselines, demonstrating their effectiveness., Comment: Accepted NeurIPS 2024

This paper presents a bridge between the theories of wonderful models associated with toric arrangements and wonderful models associated with hyperplane arrangements. In a previous work, the same authors noticed that the model of the toric arrangement of type $A_{n-1}$ is isomorphic to the one of the hyperplane arrangement of type $A_{n}$; it is natural to ask if there exist similar isomorphisms between other families of arrangements. The aim of this paper is to study one such family, namely the family of arrangements defined by graphs. The main result states that there is indeed an isomorphism between the model of a toric arrangement defined by a graph $\Gamma$ and the model of a hyperplane arrangement defined by the cone of $\Gamma$, provided that a suitable building set is chosen., Comment: 19 pages, 2 figures

The paper studies the existence of periodic solutions of a perturbed relativistic Kepler problem of the type \begin{equation*} \dfrac{\mathrm{d}}{\mathrm{d}t}\left(\frac{m\dot{x}}{\sqrt{1-|\dot{x}|^{2}/c^{2}}}\right) = -\alpha\frac{x}{|x|^{3}} + \varepsilon \, \nabla_{x} U(t,x), \qquad x \in \mathbb{R}^d\setminus\{0\}, \end{equation*} with $d=2$ or $d=3$, bifurcating, for $\varepsilon$ small enough, from the set of circular solutions of the unperturbed system. Both the case of the fixed-period problem (assuming that $U$ is $T$-periodic in time) and the case of the fixed-energy problem (assuming that $U$ is independent of time) are considered., Comment: 31 pages

We consider the problem of learning an $\varepsilon$-optimal policy in a general class of continuous-space Markov decision processes (MDPs) having smooth Bellman operators. Given access to a generative model, we achieve rate-optimal sample complexity by performing a simple, \emph{perturbed} version of least-squares value iteration with orthogonal trigonometric polynomials as features. Key to our solution is a novel projection technique based on ideas from harmonic analysis. Our~$\widetilde{\mathcal{O}}(\epsilon^{-2-d/(\nu+1)})$ sample complexity, where $d$ is the dimension of the state-action space and $\nu$ the order of smoothness, recovers the state-of-the-art result of discretization approaches for the special case of Lipschitz MDPs $(\nu=0)$. At the same time, for $\nu\to\infty$, it recovers and greatly generalizes the $\mathcal{O}(\epsilon^{-2})$ rate of low-rank MDPs, which are more amenable to regression approaches. In this sense, our result bridges the gap between two popular but conflicting perspectives on continuous-space MDPs.

Importance sampling (IS) represents a fundamental technique for a large surge of off-policy reinforcement learning approaches. Policy gradient (PG) methods, in particular, significantly benefit from IS, enabling the effective reuse of previously collected samples, thus increasing sample efficiency. However, classically, IS is employed in RL as a passive tool for re-weighting historical samples. However, the statistical community employs IS as an active tool combined with the use of behavioral distributions that allow the reduction of the estimate variance even below the sample mean one. In this paper, we focus on this second setting by addressing the behavioral policy optimization (BPO) problem. We look for the best behavioral policy from which to collect samples to reduce the policy gradient variance as much as possible. We provide an iterative algorithm that alternates between the cross-entropy estimation of the minimum-variance behavioral policy and the actual policy optimization, leveraging on defensive IS. We theoretically analyze such an algorithm, showing that it enjoys a convergence rate of order $O(\epsilon^{-4})$ to a stationary point, but depending on a more convenient variance term w.r.t. standard PG methods. We then provide a practical version that is numerically validated, showing the advantages in the policy gradient estimation variance and on the learning speed.

Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. They learn stochastic parametric (hyper)policies by either exploring in the space of actions or in the space of parameters. Stochastic controllers, however, are often undesirable from a practical perspective because of their lack of robustness, safety, and traceability. In common practice, stochastic (hyper)policies are learned only to deploy their deterministic version. In this paper, we make a step towards the theoretical understanding of this practice. After introducing a novel framework for modeling this scenario, we study the global convergence to the best deterministic policy, under (weak) gradient domination assumptions. Then, we illustrate how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy. Finally, we quantitatively compare action-based and parameter-based exploration, giving a formal guise to intuitive results., Comment: Accepted to ICML 2024

We study the problem of online learning in contextual bandit problems where the loss function is assumed to belong to a known parametric function class. We propose a new analytic framework for this setting that bridges the Bayesian theory of information-directed sampling due to Russo and Van Roy (2018) and the worst-case theory of Foster, Kakade, Qian, and Rakhlin (2021) based on the decision-estimation coefficient. Drawing from both lines of work, we propose a algorithmic template called Optimistic Information-Directed Sampling and show that it can achieve instance-dependent regret guarantees similar to the ones achievable by the classic Bayesian IDS method, but with the major advantage of not requiring any Bayesian assumptions. The key technical innovation of our analysis is introducing an optimistic surrogate model for the regret and using it to define a frequentist version of the Information Ratio of Russo and Van Roy (2018), and a less conservative version of the Decision Estimation Coefficient of Foster et al. (2021). Keywords: Contextual bandits, information-directed sampling, decision estimation coefficient, first-order regret bounds.

We show that, in one spatial and arbitrary jump dimension, the averaged solution of a Marcustype SPDE with pure jump L\'evy transport noise satisfies a dissipative deterministic equation involving a fractional Laplace-type operator. To this end, we identify the correct associated L\'evy measure for the driving noise. We consider this a first step in the direction of a non-local version of enhanced dissipation, a phenomenon recently proven to occur for Brownian transport noise and the associated local parabolic PDE by the first author. Moreover, we present numerical simulations, supporting the fact that dissipation occurs for the averaged solution, with a behavior akin to the diffusion due to a fractional Laplacian, but not in a pathwise sense., Comment: 10 pages, 6 figures

Obtaining no-regret guarantees for reinforcement learning (RL) in the case of problems with continuous state and/or action spaces is still one of the major open challenges in the field. Recently, a variety of solutions have been proposed, but besides very specific settings, the general problem remains unsolved. In this paper, we introduce a novel structural assumption on the Markov decision processes (MDPs), namely $\nu-$smoothness, that generalizes most of the settings proposed so far (e.g., linear MDPs and Lipschitz MDPs). To face this challenging scenario, we propose two algorithms for regret minimization in $\nu-$smooth MDPs. Both algorithms build upon the idea of constructing an MDP representation through an orthogonal feature map based on Legendre polynomials. The first algorithm, \textsc{Legendre-Eleanor}, archives the no-regret property under weaker assumptions but is computationally inefficient, whereas the second one, \textsc{Legendre-LSVI}, runs in polynomial time, although for a smaller class of problems. After analyzing their regret properties, we compare our results with state-of-the-art ones from RL theory, showing that our algorithms achieve the best guarantees.

Prompted by the relevant problem of temperature inversion (i.e. gradient of density anti-correlated to the gradient of temperature) in astrophysics, we introduce a novel method to model a gravitationally confined multi-component collisionless plasma in contact with a fluctuating thermal boundary. We focus on systems with anti-correlated (inverted) density and temperature profiles, with applications to solar physics. The dynamics of the plasma is analytically described via the coupling of an appropriated coarse-grained distribution function and temporally coarse-grained Vlasov dynamics. We derive a stationary solution of the system and predict the inverted density and temperature profiles of the two-species for scenarios relevant for the corona. We validate our method by comparing the analytical results with kinetic numerical simulations of the plasma dynamics in the context of the two-species Hamiltonian mean-field model (HMF). Finally, we apply our theoretical framework to the problem of the temperature inversion in the solar corona obtaining density and temperature profiles in remarkably good agreement with the observations., Comment: 27 pages, 12 Figs. Version accepted for publication in JPP

The numerical study of relativistic magnetohydrodynamics (MHD) plays a crucial role in high-energy astrophysics, but unfortunately is computationally demanding, given the complex physics involved (high Lorentz factor flows, extreme magnetization, curved spacetimes near compact objects) and the large variety of spatial scales needed to resolve turbulent motions. A great benefit comes from the porting of existing codes running on standard processors to GPU-based platforms. However, this usually requires a drastic rewriting of the original code, the use of specific languages like CUDA, and a complex analysis of data management and optimization of parallel processes. Here we describe the porting of the ECHO code for special and general relativistic MHD to accelerated devices, simply based on native Fortran language built-in constructs, especially 'do concurrent' loops, few OpenACC directives, and the straightforward data management provided by the Unified Memory option of NVIDIA compilers.Thanks to these very minor modifications to the original code, the new version of ECHO runs at least 16 times faster on GPU platforms compared to CPU-based ones. The chosen benchmark is the 3D propagation of a relativistic MHD Alfv\'en wave, for which strong and weak scaling tests performed on the LEONARDO pre-exascale supercomputer at CINECA are provided (using up to 256 nodes corresponding to 1024 GPUs, and over 14 billion cells). Finally, an example of high-resolution relativistic MHD Alfv\'enic turbulence simulation is shown, demonstrating the potential for astrophysical plasmas of the new GPU-based version of ECHO., Comment: Accepted for publication on Fluids, MDPI, 17 pages

This paper introduces novel alternate training procedures for hard-parameter sharing Multi-Task Neural Networks (MTNNs). Traditional MTNN training faces challenges in managing conflicting loss gradients, often yielding sub-optimal performance. The proposed alternate training method updates shared and task-specific weights alternately, exploiting the multi-head architecture of the model. This approach reduces computational costs, enhances training regularization, and improves generalization. Convergence properties similar to those of the classical stochastic gradient method are established. Empirical experiments demonstrate delayed overfitting, improved prediction, and reduced computational demands. In summary, our alternate training procedures offer a promising advancement for the training of hard-parameter sharing MTNNs.

Trajectory Optimization (TO) and Reinforcement Learning (RL) are powerful and complementary tools to solve optimal control problems. On the one hand, TO can efficiently compute locally-optimal solutions, but it tends to get stuck in local minima if the problem is not convex. On the other hand, RL is typically less sensitive to non-convexity, but it requires a much higher computational effort. Recently, we have proposed CACTO (Continuous Actor-Critic with Trajectory Optimization), an algorithm that uses TO to guide the exploration of an actor-critic RL algorithm. In turns, the policy encoded by the actor is used to warm-start TO, closing the loop between TO and RL. In this work, we present an extension of CACTO exploiting the idea of Sobolev learning. To make the training of the critic network faster and more data efficient, we enrich it with the gradient of the Value function, computed via a backward pass of the differential dynamic programming algorithm. Our results show that the new algorithm is more efficient than the original CACTO, reducing the number of TO episodes by a factor ranging from 3 to 10, and consequently the computation time. Moreover, we show that CACTO-SL helps TO to find better minima and to produce more consistent results.

Detailed measurements of the spectral structure of cosmic-ray electrons and positrons from 10.6 GeV to 7.5 TeV are presented from over 7 years of observations with the CALorimetric Electron Telescope (CALET) on the International Space Station. Because of the excellent energy resolution (a few percent above 10 GeV) and the outstanding e/p separation (10$^5$), CALET provides optimal performance for a detailed search of structures in the energy spectrum. The analysis uses data up to the end of 2022, and the statistics of observed electron candidates has increased more than 3 times since the last publication in 2018. By adopting an updated boosted decision tree analysis, a sufficient proton rejection power up to 7.5 TeV is achieved, with a residual proton contamination less than 10%. The observed energy spectrum becomes gradually harder in the lower energy region from around 30 GeV, consistently with AMS-02, but from 300 to 600 GeV it is considerably softer than the spectra measured by DAMPE and Fermi-LAT. At high energies, the spectrum presents a sharp break around 1 TeV, with a spectral index change from -3.15 to -3.91, and a broken power law fitting the data in the energy range from 30 GeV to 4.8 TeV better than a single power law with 6.9 sigma significance, which is compatible with the DAMPE results. The break is consistent with the expected effects of radiation loss during the propagation from distant sources (except the highest energy bin). We have fitted the spectrum with a model consistent with the positron flux measured by AMS-02 below 1 TeV and interpreted the electron + positron spectrum with possible contributions from pulsars and nearby sources. Above 4.8 TeV, a possible contribution from known nearby supernova remnants, including Vela, is addressed by an event-by-event analysis providing a higher proton-rejection power than a purely statistical analysis., Comment: main text: 7 pages, 4 figures; supplemental material: 10 pages, 8 figures, 1 table

The problem of relevance ranking consists of sorting a set of objects with respect to a given criterion. Since users may prefer different relevance criteria, the ranking algorithms should be adaptable to the user needs. Two main approaches exist in literature for the task of learning to rank: 1) a score function, learned by examples, which evaluates the properties of each object yielding an absolute relevance value that can be used to order the objects or 2) a pairwise approach, where a "preference function" is learned using pairs of objects to define which one has to be ranked first. In this paper, we present SortNet, an adaptive ranking algorithm which orders objects using a neural network as a comparator. The neural network training set provides examples of the desired ordering between pairs of items and it is constructed by an iterative procedure which, at each iteration, adds the most informative training examples. Moreover, the comparator adopts a connectionist architecture that is particularly suited for implementing a preference function. We also prove that such an architecture has the universal approximation property and can implement a wide class of functions. Finally, the proposed algorithm is evaluated on the LETOR dataset showing promising performances in comparison with other state of the art algorithms., Comment: The 31st Annual International ACM SIGIR Conference (SIGIR 2008) - Workshop: Learning to Rank for Information Retrieval (LR4IR), Singapore, July 20-24 2008 - ISBN:978-16-05581-64-4

In this paper we report the detection of five strong Gamma-Ray Bursts (GRBs) by the High-Energy Particle Detector (HEPD-01) mounted on board the China Seismo-Electromagnetic Satellite (CSES-01), operational since 2018 on a Sun-synchronous polar orbit at a $\sim$ 507 km altitude and 97$^\circ$ inclination. HEPD-01 was designed to detect high-energy electrons in the energy range 3 - 100 MeV, protons in the range 30 - 300 MeV, and light nuclei in the range 30 - 300 MeV/n. Nonetheless, Monte Carlo simulations have shown HEPD-01 is sensitive to gamma-ray photons in the energy range 300 keV - 50 MeV, even if with a moderate effective area above $\sim$ 5 MeV. A dedicated time correlation analysis between GRBs reported in literature and signals from a set of HEPD-01 trigger configuration masks has confirmed the anticipated detector sensitivity to high-energy photons. A comparison between the simultaneous time profiles of HEPD-01 electron fluxes and photons from GRB190114C, GRB190305A, GRB190928A, GRB200826B and GRB211211A has shown a remarkable similarity, in spite of the different energy ranges. The high-energy response, with peak sensitivity at about 2 MeV, and moderate effective area of the detector in the actual flight configuration explain why these five GRBs, characterised by a fluence above $\sim$ 3 $\times$ 10$^{-5}$ erg cm$^{-2}$ in the energy interval 300 keV - 50 MeV, have been detected., Comment: Accepted for publication in The Astrophysical Journal (ApJ)

We study the problem of offline policy optimization in stochastic contextual bandit problems, where the goal is to learn a near-optimal policy based on a dataset of decision data collected by a suboptimal behavior policy. Rather than making any structural assumptions on the reward function, we assume access to a given policy class and aim to compete with the best comparator policy within this class. In this setting, a standard approach is to compute importance-weighted estimators of the value of each policy, and select a policy that minimizes the estimated value up to a "pessimistic" adjustment subtracted from the estimates to reduce their random fluctuations. In this paper, we show that a simple alternative approach based on the "implicit exploration" estimator of \citet{Neu2015} yields performance guarantees that are superior in nearly all possible terms to all previous results. Most notably, we remove an extremely restrictive "uniform coverage" assumption made in all previous works. These improvements are made possible by the observation that the upper and lower tails importance-weighted estimators behave very differently from each other, and their careful control can massively improve on previous results that were all based on symmetric two-sided concentration inequalities. We also extend our results to infinite policy classes in a PAC-Bayesian fashion, and showcase the robustness of our algorithm to the choice of hyper-parameters by means of numerical simulations.

JWST has ushered in an era of unprecedented ability to characterize exoplanetary atmospheres. While there are over 5,000 confirmed planets, more than 4,000 TESS planet candidates are still unconfirmed and many of the best planets for atmospheric characterization may remain to be identified. We present a sample of TESS planets and planet candidates that we identify as "best-in-class" for transmission and emission spectroscopy with JWST. These targets are sorted into bins across equilibrium temperature $T_{\mathrm{eq}}$ and planetary radius $R{_\mathrm{p}}$ and are ranked by transmission and emission spectroscopy metric (TSM and ESM, respectively) within each bin. In forming our target sample, we perform cuts for expected signal size and stellar brightness, to remove sub-optimal targets for JWST. Of the 194 targets in the resulting sample, 103 are unconfirmed TESS planet candidates, also known as TESS Objects of Interest (TOIs). We perform vetting and statistical validation analyses on these 103 targets to determine which are likely planets and which are likely false positives, incorporating ground-based follow-up from the TESS Follow-up Observation Program (TFOP) to aid the vetting and validation process. We statistically validate 23 TOIs, marginally validate 33 TOIs to varying levels of confidence, deem 29 TOIs likely false positives, and leave the dispositions for 4 TOIs as inconclusive. 14 of the 103 TOIs were confirmed independently over the course of our analysis. We provide our final best-in-class sample as a community resource for future JWST proposals and observations. We intend for this work to motivate formal confirmation and mass measurements of each validated planet and encourage more detailed analysis of individual targets by the community., Comment: Submitted to AJ. Machine-readable versions of Tables 2 and 3 are included. 40 pages, 7 figures, 3 tables

Current understanding of the kinetic-scale turbulence in weakly-collisional plasmas still remains elusive. We employ a general framework in which the turbulent energy transfer is envisioned as a scale-to-scale Langevin process. Fluctuations in the sub-ion range show a global scale invariance, thus suggesting a homogeneous energy repartition. In this Letter, we interpret such a feature by linking the drift term of the Langevin equation to scaling properties of fluctuations. Theoretical expectations are verified on solar wind observations and numerical simulations thus giving relevance to the proposed framework for understanding kinetic-scale turbulence in space plasmas., Comment: Accepted for publication in Physical Review Research

We present the observation of a charge-sign dependent solar modulation of galactic cosmic rays (GCRs) with the CALorimetric Electron Telescope onboard the International Space Station over 6 yr, corresponding to the positive polarity of the solar magnetic field. The observed variation of proton count rate is consistent with the neutron monitor count rate, validating our methods for determining the proton count rate. It is observed by the CALorimetric Electron Telescope that both GCR electron and proton count rates at the same average rigidity vary in anticorrelation with the tilt angle of the heliospheric current sheet, while the amplitude of the variation is significantly larger in the electron count rate than in the proton count rate. We show that this observed charge-sign dependence is reproduced by a numerical ``drift model'' of the GCR transport in the heliosphere. This is a clear signature of the drift effect on the long-term solar modulation observed with a single detector., Comment: main text: 6 pages, 3 figures, supplemental material: 8 pages, 3 figures, 1 table

Offline Reinforcement Learning (RL) aims to learn a near-optimal policy from a fixed dataset of transitions collected by another policy. This problem has attracted a lot of attention recently, but most existing methods with strong theoretical guarantees are restricted to finite-horizon or tabular settings. In constrast, few algorithms for infinite-horizon settings with function approximation and minimal assumptions on the dataset are both sample and computationally efficient. Another gap in the current literature is the lack of theoretical analysis for the average-reward setting, which is more challenging than the discounted setting. In this paper, we address both of these issues by proposing a primal-dual optimization method based on the linear programming formulation of RL. Our key contribution is a new reparametrization that allows us to derive low-variance gradient estimators that can be used in a stochastic optimization scheme using only samples from the behavior policy. Our method finds an $\varepsilon$-optimal policy with $O(\varepsilon^{-4})$ samples, improving on the previous $O(\varepsilon^{-5})$, while being computationally efficient for infinite-horizon discounted and average-reward MDPs with realizable linear function approximation and partial coverage. Moreover, to the best of our knowledge, this is the first theoretical result for average-reward offline RL.

The forward $\eta$ mesons production has been observed by the Large Hadron Collider forward (LHCf) experiment in proton-proton collision at $\sqrt{s}$=13 TeV. This paper presents the measurement of the inclusive production rate of $\eta$ in $p_T<$ 1.1 GeV/c, expressed as a function of the Feynman-x variable. These results are compared with the predictions of several hadronic interaction models commonly used for the modelling of the air showers produced by ultra-high energy cosmic rays. This is both the first measurement of $\eta$ mesons from LHCf and the first time a particle containing strange quarks has been observed in the forward region for high-energy collisions. These results will provide a powerful constraint on hadronic interaction models for the purpose of improving the understanding of the processes underlying the air showers produced in the Earth's atmosphere by ultra-energetic cosmic rays., Comment: 18 pages, 7 figures, 4 tables