Search

Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent's performance. In this paper, we present "Beyond The Rainbow" (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC, with a human-normalized interquartile mean (IQM) of 7.4 on atari-60. Beyond Atari, we demonstrate BTR's capability to handle complex 3D games, successfully training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algorithmic changes. Designing BTR with computational efficiency in mind, agents can be trained using a desktop PC on 200 million Atari frames within 12 hours. Additionally, we conduct detailed ablation studies of each component, analzying the performance and impact using numerous measures., Comment: 9 main pages, 26 total. Currently under review at ICLR

Anti-viral therapies are typically designed to target only the current strains of a virus. Game theoretically, this corresponds to a short-sighted, or myopic, response. However, therapy-induced selective pressures act on viruses to drive the emergence of mutated strains, against which initial therapies have reduced efficacy. Building on a computational model of binding between antibodies and viral antigens (the Absolut! framework), we design and implement a genetic simulation of viral evolutionary escape. Crucially, this allows our antibody optimisation algorithm to consider and influence the entire escape curve of the virus, i.e. to guide (or "shape") the viral evolution. This is inspired by opponent shaping which, in general-sum learning, accounts for the adaptation of the co-player rather than playing a myopic best response. Hence we call the optimised antibodies shapers. Within our simulations, we demonstrate that our shapers target both current and simulated future viral variants, outperforming the antibodies chosen in a myopic way. Furthermore, we show that shapers exert specific evolutionary pressure on the virus compared to myopic antibodies. Altogether, shapers modify the evolutionary trajectories of viral strains and minimise the viral escape compared to their myopic counterparts. While this is a simplified model, we hope that our proposed paradigm will facilitate the discovery of better long-lived vaccines and antibody therapies in the future, enabled by rapid advancements in the capabilities of simulation tools. Our code is available at https://github.com/olakalisz/antibody-shapers., Comment: Preprint

Future reward estimation is a core component of reinforcement learning agents; i.e., Q-value and state-value functions, predicting an agent's sum of future rewards. Their scalar output, however, obfuscates when or what individual future rewards an agent may expect to receive. We address this by modifying an agent's future reward estimator to predict their next N expected rewards, referred to as Temporal Reward Decomposition (TRD). This unlocks novel explanations of agent behaviour. Through TRD we can: estimate when an agent may expect to receive a reward, the value of the reward and the agent's confidence in receiving it; measure an input feature's temporal importance to the agent's action decisions; and predict the influence of different actions on future rewards. Furthermore, we show that DQN agents trained on Atari environments can be efficiently retrained to incorporate TRD with minimal impact on performance., Comment: 7 pages + 3 pages of supplementary material. Published at ECAI 2024

Reinforcement Learning (RL) is a continuously growing field that has the potential to revolutionize many areas of artificial intelligence. However, despite its promise, RL research is often hindered by the lack of standardization in environment and algorithm implementations. This makes it difficult for researchers to compare and build upon each other's work, slowing down progress in the field. Gymnasium is an open-source library that provides a standard API for RL environments, aiming to tackle this issue. Gymnasium's main feature is a set of abstractions that allow for wide interoperability between environments and training algorithms, making it easier for researchers to develop and test RL algorithms. In addition, Gymnasium provides a collection of easy-to-use environments, tools for easily customizing environments, and tools to ensure the reproducibility and robustness of RL research. Through this unified framework, Gymnasium significantly streamlines the process of developing and testing RL algorithms, enabling researchers to focus more on innovation and less on implementation details. By providing a standardized platform for RL research, Gymnasium helps to drive forward the field of reinforcement learning and unlock its full potential. Gymnasium is available online at https://github.com/Farama-Foundation/Gymnasium

A characterization of the finite-dimensional Leibniz algebras with an abelian subalgebra of codimension two over a field $\mathbb{F}$ of characteristic $p\neq2$ is given. In short, a finite-dimensional Leibniz algebra of dimension $n$ with an abelian subalgebra of codimension two is solvable and contains an abelian ideal of codimension at most two or it is a direct sum of a Lie one-dimensional solvable extension of the Heisenberg algebra $\mathfrak{h}(\mathbb{F})$ and $\mathbb{F}^{n-4}$ or a direct sum of a $3$-dimensional simple Lie algebra and $\mathbb{F}^{n-3}$ or a Leibniz one-dimensional solvable extension of the algebra $\mathfrak{h}(\mathbb{F}) \oplus \mathbb{F}^{n-4}$.

This paper studies the abelian subalgebras and ideals of maximal dimension of Poisson algebras $\mathcal{P}$ of dimension $n$. We introduce the invariants $\alpha$ and $\beta$ for Poisson algebras, which correspond to the dimension of an abelian subalgebra and ideal of maximal dimension, respectively. We prove that these invariants coincide if $\alpha(\mathcal{P}) = n-1$. We characterize the Poisson algebras with $\alpha(\mathcal{P}) = n-2$ over arbitrary fields. In particular, we characterize Lie algebras $L$ with $\alpha(L) = n-2$. We also show that $\alpha(\mathcal{P}) = n-2$ for nilpotent Poisson algebras implies $\beta(\mathcal{P})=n-2$. Finally, we study these invariants for various distinguished Poisson algebras, providing us with several examples and counterexamples.

The boundless possibility of neural networks which can be used to solve a problem -- each with different performance -- leads to a situation where a Deep Learning expert is required to identify the best neural network. This goes against the hope of removing the need for experts. Neural Architecture Search (NAS) offers a solution to this by automatically identifying the best architecture. However, to date, NAS work has focused on a small set of datasets which we argue are not representative of real-world problems. We introduce eight new datasets created for a series of NAS Challenges: AddNIST, Language, MultNIST, CIFARTile, Gutenberg, Isabella, GeoClassing, and Chesseract. These datasets and challenges are developed to direct attention to issues in NAS development and to encourage authors to consider how their models will perform on datasets unknown to them at development time. We present experimentation using standard Deep Learning methods as well as the best results from challenge participants.

This paper is concerned with generalising the results for Lie $CT$-algebras to Leibniz algebras. In some cases our results give a generalisation even for the case of a Lie algebra. Results on $A$-algebras are used to show every Leibniz $CT$-algebra over an algebraically closed field of characteristic different from 2,3 is solvable or is isomorphic to $sl_2(F)$. A characterisation is then given for solvable Leibniz $CT$-algebras. It is also shown that the class of solvable Leibniz $CT$-algebras is factor closed.

Policy Mirror Descent (PMD) is a popular framework in reinforcement learning, serving as a unifying perspective that encompasses numerous algorithms. These algorithms are derived through the selection of a mirror map and enjoy finite-time convergence guarantees. Despite its popularity, the exploration of PMD's full potential is limited, with the majority of research focusing on a particular mirror map -- namely, the negative entropy -- which gives rise to the renowned Natural Policy Gradient (NPG) method. It remains uncertain from existing theoretical studies whether the choice of mirror map significantly influences PMD's efficacy. In our work, we conduct empirical investigations to show that the conventional mirror map choice (NPG) often yields less-than-optimal outcomes across several standard benchmark environments. Using evolutionary strategies, we identify more efficient mirror maps that enhance the performance of PMD. We first focus on a tabular environment, i.e. Grid-World, where we relate existing theoretical bounds with the performance of PMD for a few standard mirror maps and the learned one. We then show that it is possible to learn a mirror map that outperforms the negative entropy in more complex environments, such as the MinAtar suite. Our results suggest that mirror maps generalize well across various environments, raising questions about how to best match a mirror map to an environment's structure and characteristics.

In many Reinforcement Learning (RL) papers, learning curves are useful indicators to measure the effectiveness of RL algorithms. However, the complete raw data of the learning curves are rarely available. As a result, it is usually necessary to reproduce the experiments from scratch, which can be time-consuming and error-prone. We present Open RL Benchmark, a set of fully tracked RL experiments, including not only the usual data such as episodic return, but also all algorithm-specific and system metrics. Open RL Benchmark is community-driven: anyone can download, use, and contribute to the data. At the time of writing, more than 25,000 runs have been tracked, for a cumulative duration of more than 8 years. Open RL Benchmark covers a wide range of RL libraries and reference implementations. Special care is taken to ensure that each experiment is precisely reproducible by providing not only the full parameters, but also the versions of the dependencies used to generate it. In addition, Open RL Benchmark comes with a command-line interface (CLI) for easy fetching and generating figures to present the results. In this document, we include two case studies to demonstrate the usefulness of Open RL Benchmark in practice. To the best of our knowledge, Open RL Benchmark is the first RL benchmark of its kind, and the authors hope that it will improve and facilitate the work of researchers in the field., Comment: Under review

This paper starts by showing that for algebras in a certain class the concepts of weak nilpotency and nilpotency coincide. It goes on to describe some solvability and nilpotency properties of bicommutative algebras, of assosymmetric and of Novikov algebras and to introduce a Frattini theory for all of them. A description is also given for semisimple bicommutative algebras over any field.

This paper examines whether the concept of an almost-algebraic Lie algebra developed by Auslander and Brezin in \cite{ab} can be introduced for Leibniz algebras. Two possible analogues are considered: almost-reductive and almost-algebraic Leibniz algebras. For Lie algebras these two concepts are the same, but that is not the case for Leibniz algebras, the class of almost-algebraic Leibniz algebras strictly containing that of the almost-reductive ones. Various properties of these two classes of algebras are obtained, together with some relationships to $\phi$-free, elementary, $E$-algebras and $A$-algebras.

We present the Minigrid and Miniworld libraries which provide a suite of goal-oriented 2D and 3D environments. The libraries were explicitly created with a minimalistic design paradigm to allow users to rapidly develop new environments for a wide range of research-specific needs. As a result, both have received widescale adoption by the RL community, facilitating research in a wide range of areas. In this paper, we outline the design philosophy, environment details, and their world generation API. We also showcase the additional capabilities brought by the unified API between Minigrid and Miniworld through case studies on transfer learning (for both RL agents and humans) between the different observation spaces. The source code of Minigrid and Miniworld can be found at https://github.com/Farama-Foundation/{Minigrid, Miniworld} along with their documentation at https://{minigrid, miniworld}.farama.org/.

We establish quantitative compactness estimates for finite difference schemes used to solve nonlinear conservation laws. These equations involve a flux function $f(k(x,t),u)$, where the coefficient $k(x,t$ is $BV$-regular and may exhibit discontinuities along curves in the $(x,t)$ plane. Our approach, which is technically elementary, relies on a discrete interaction estimate and the existence of one entropy function. While the details are specifically outlined for the Lax-Friedrichs scheme, the same framework can be applied to other difference schemes. Notably, our compactness estimates are new even in the homogeneous case ($k\equiv 1$).

In this paper we continue the study of the subalgebra lattice of a Leibniz algebra. In particular, we find out that solvable Leibniz algebras with an upper semi-modular lattice are either almost-abelian or have an abelian ideal spanned by the elements with square zero. We also study Leibniz algebras in which every subalgebra is a weak quasi-ideal, as well as modular symmetric Leibniz algebras.

Meta-learning, the notion of learning to learn, enables learning systems to quickly and flexibly solve new tasks. This usually involves defining a set of outer-loop meta-parameters that are then used to update a set of inner-loop parameters. Most meta-learning approaches use complicated and computationally expensive bi-level optimisation schemes to update these meta-parameters. Ideally, systems should perform multiple orders of meta-learning, i.e. to learn to learn to learn and so on, to accelerate their own learning. Unfortunately, standard meta-learning techniques are often inappropriate for these higher-order meta-parameters because the meta-optimisation procedure becomes too complicated or unstable. Inspired by the higher-order meta-learning we observe in real-world evolution, we show that using simple population-based evolution implicitly optimises for arbitrarily-high order meta-parameters. First, we theoretically prove and empirically show that population-based evolution implicitly optimises meta-parameters of arbitrarily-high order in a simple setting. We then introduce a minimal self-referential parameterisation, which in principle enables arbitrary-order meta-learning. Finally, we show that higher-order meta-learning improves performance on time series forecasting tasks.

A subalgebra $B$ of a Leibniz algebra $L$ is called a weak c-ideal of $L$ if there is a subideal $C$ of $L$ such that $L=B+C$ and $B\cap C\subseteq B_{L}$ where $B_{L}$ is the largest ideal of $L$ contained in $B.$ This is analogous to the concept of a weakly c-normal subgroup, which has been studied by a number of authors. We obtain some properties of weak c-ideals and use them to give some characterisations of solvable and supersolvable Leibniz algebras generalising previous results for Lie algebras. We note that one-dimensional weak c-ideals are c-ideals, and show that a result of Turner classifying Leibniz algebras in which every one-dimensional subalgebra is a c-ideal is false for general Leibniz algebras, but holds for symmetric ones., Comment: arXiv admin note: substantial text overlap with arXiv:2006.05234

Decision making is a human process that is a fundamental part of competition. As a realisation of decision making, Command and Control, or C2, has been studied in the literature for adversarial populations, yet these models do not explicitly capture competition. Our work here seeks to enrich such competition models by extending ecologically inspired population models to include decision dynamics through coupling with a nonlinear system of oscillators (the Kuramoto equation) to model the action-perception cycle in decision making. Through asymmetric competition models of increasing complexity, we highlight the importance of competitive agility and decision making processes necessary for a population to be successful., Comment: 33 pages

It is a sad reflection of modern academia that code is often ignored after publication -- there is no academic 'kudos' for bug fixes / maintenance. Code is often unavailable or, if available, contains bugs, is incomplete, or relies on out-of-date / unavailable libraries. This has a significant impact on reproducibility and general scientific progress. Neural Architecture Search (NAS) is no exception to this, with some prior work in reproducibility. However, we argue that these do not consider long-term reproducibility issues. We therefore propose a checklist for long-term NAS reproducibility. We evaluate our checklist against common NAS approaches along with proposing how we can retrospectively make these approaches more long-term reproducible., Comment: 4 pages, LaTeX, Typos corrected