Search

Reinforcement learning algorithms typically rely on the assumption that the environment dynamics and value function can be expressed in terms of a Markovian state representation. However, when state information is only partially observable, how can an agent learn such a state representation, and how can it detect when it has found one? We introduce a metric that can accomplish both objectives, without requiring access to--or knowledge of--an underlying, unobservable state space. Our metric, the $\lambda$-discrepancy, is the difference between two distinct temporal difference (TD) value estimates, each computed using TD($\lambda$) with a different value of $\lambda$. Since TD($\lambda$=0) makes an implicit Markov assumption and TD($\lambda$=1) does not, a discrepancy between these estimates is a potential indicator of a non-Markovian state representation. Indeed, we prove that the $\lambda$-discrepancy is exactly zero for all Markov decision processes and almost always non-zero for a broad class of partially observable environments. We also demonstrate empirically that, once detected, minimizing the $\lambda$-discrepancy can help with learning a memory function to mitigate the corresponding partial observability. We then train a reinforcement learning agent that simultaneously constructs two recurrent value networks with different $\lambda$ parameters and minimizes the difference between them as an auxiliary loss. The approach scales to challenging partially observable domains, where the resulting agent frequently performs significantly better (and never performs worse) than a baseline recurrent agent with only a single value network., Comment: GitHub URL: https://github.com/brownirl/lambda_discrepancy; Videos: https://lambda-discrepancy.github.io/

Additive manufacturing is an expanding multidisciplinary field encompassing applications including medical devices, aerospace components, microfabrication strategies, and artificial organs. Among additive manufacturing approaches, light-based printing technologies, including two-photon polymerization, projection micro stereolithography, and volumetric printing, have garnered significant attention due to their speed, resolution and/or potential applications for biofabrication. In this study, we introduce dynamic interface printing (DIP), a new 3D printing approach that leverages an acoustically modulated, constrained air-liquid boundary to rapidly generate cm-scale three-dimensional structures within tens of seconds. Distinct from volumetric approaches, this process eliminates the need for intricate feedback systems, specialized chemistry, or complex optics while maintaining rapid printing speeds. We demonstrate the versatility of this technique across a broad array of materials and intricate geometries, including those that would be impossible to print via conventional layer-by-layer methods. In doing so, we demonstrate the rapid fabrication of complex structures in-situ, overprinting, structural parallelisation, and biofabrication utility. Moreover, we showcase that the formation of surface waves at this boundary enables enhanced mass transport, material flexibility, and permits three-dimensional particle patterning. We therefore anticipate that this approach will be invaluable for applications where high resolution, scalable throughput, and biocompatible printing is required., Comment: 63 Pages, 4 Figures, 18 Supplementary Figures

Neural MMO 2.0 is a massively multi-agent environment for reinforcement learning research. The key feature of this new version is a flexible task system that allows users to define a broad range of objectives and reward signals. We challenge researchers to train agents capable of generalizing to tasks, maps, and opponents never seen during training. Neural MMO features procedurally generated maps with 128 agents in the standard setting and support for up to. Version 2.0 is a complete rewrite of its predecessor with three-fold improved performance and compatibility with CleanRL. We release the platform as free and open-source software with comprehensive documentation available at neuralmmo.github.io and an active community Discord. To spark initial research on this new platform, we are concurrently running a competition at NeurIPS 2023.