Search

Lifting during manual material handling is a major cause of low-back pain (LBP). As an important risk factor that directly influences the risk of LBP, the Load vertical location (LVL) during lifting needs to be measured and controlled. However, existing solutions for LVL measurement are inefficient, inaccurate, and impractical for real-world workplace environments. To address these problems, an unobtrusive wearable system, including smart insoles and smart wristbands, was proposed to measure LVL accurately in workplace environments. Different from traditional methods which rely on Inertial Measurement Unit (IMU) and suffer from integral drifting errors over time, a novel barometer-based LVL measurement method was proposed in this study. To correct the environment-induced LVL measurement errors in the barometer-based method, a novel Known Vertical Location Update (KVLU) method was proposed. This method calibrates the measured LVL using a known wrist vertical location at known postures during frequently used daily activities such as standing and walking. The proposed wearable system achieved a mean absolute error (MAE) of 5.71 cm in LVL measurement. This result indicates that the proposed system has the potential to reliably measure LVL and assess the risk of LBP in manual lifting tasks.

Biomaterial surface engineering and integrating cell-adhesive ligands are crucial in biological research and biotechnological applications. The interplay between cells and their microenvironment, influenced by chemical and physical cues, impacts cellular behavior. Surface modification of biomaterials profoundly affects cellular responses, especially at the cell-surface interface. This work focuses on enhancing cellular activities through material manipulation, emphasizing silanization for further functionalization with bioactive molecules like RGD peptides to improve cell adhesion. The grafting of three distinct silanes onto silicon wafers using both spin coating and immersion methods was investigated. This study sheds light on the effects of different alkyl chain lengths and protecting groups on cellular behavior, providing valuable insights into optimizing silane-based self-assembled monolayers (SAMs) before peptide or protein grafting for the first time. Specifically, it challenges the common use of APTES molecules in this context. These findings advance our understanding of surface modification strategies, paving the way for tailoring biomaterial surfaces to modulate cellular behavior for diverse biotechnological applications.

One-dimensional Coulomb drag has been an essential tool to probe the physics of interacting Tomonaga-Luttinger liquids. To date, most experimental work has focused on the linear regime while the predictions for Luttinger liquids beyond the linear response theory remain largely untested. In this letter, we report measurements of momentum transfer induced Coulomb drag between vertically-coupled quasi-one-dimensional quantum wires in the nonlinear regime. Measurements were performed at ultra-low temperatures between wires only 15 nm apart. Our results reveal a nonlinear dependence of the drag voltage as a function of the drive current superimposed with an oscillatory contribution, in agreement with theoretical predictions for Coulomb drag between Tomonaga-Luttinger liquids. Additionally, the observed current-voltage ($I$-$V$) characteristics exhibit a nonmonotonic temperature dependence, further corroborating the presence of non-Fermi-liquid behavior in our system. These findings are observed both in the single and in the multiple subband regimes and in the presence of disorder, extending the onset of this behavior beyond the clean single channel Tomonaga-Luttinger regime where the predictions were originally formulated., Comment: 5 pages, 4 figures

Coulomb drag is a powerful tool to study interactions in coupled low-dimensional systems. Historically, Coulomb drag has been attributed to a frictional force arising from momentum transfer whose direction is dictated by the current flow. However recent findings in one-dimensional systems have also identified a rectified contribution that is independent of the current flow direction. In this work, we present Coulomb drag measurements between vertically coupled GaAs/AlGaAs quantum wires separated vertically by a hard barrier only 15 nm wide where both momentum transfer and rectified Coulomb drag signals are observed simultaneously, and whose relative magnitude are temperature and gate tunable. Our study opens up the possibility of studying the physical mechanisms behind the onset of both momentum transfer and charge rectification drag simultaneously in a single device, ultimately leading to a better understanding of Luttinger liquids in multi-channel wires and paving the way for the creation of energy harvesting devices., Comment: 9 pages, 5 figures

As speech processing systems in mobile and edge devices become more commonplace, the demand for unintrusive speech quality monitoring increases. Deep learning methods provide high-quality estimates of objective and subjective speech quality metrics. However, their significant computational requirements are often prohibitive on resource-constrained devices. To address this issue, we investigated binary activation maps (BAMs) for speech quality prediction on a convolutional architecture based on DNSMOS. We show that the binary activation model with quantization aware training matches the predictive performance of the baseline model. It further allows using other compression techniques. Combined with 8-bit weight quantization, our approach results in a 25-fold memory reduction during inference, while replacing almost all dot products with summations. Our findings show a path toward substantial resource savings by supporting mixed-precision binary multiplication in hard- and software., Comment: Accepted for Interspeech 2024

Data-driven models for stellar spectra which depend on stellar labels suffer from label systematics which decrease model performance: the "stellar labels gap". To close the stellar labels gap, we present a stellar label independent model for $\textit{Gaia}$ BP/RP (XP) spectra. We develop a novel implementation of a variational auto-encoder; a $\textit{scatter}$ VAE, which learns to generate an XP spectrum and intrinsic scatter without relying on stellar labels. We demonstrate that our model achieves competitive XP spectra reconstructions in comparison to stellar label dependent models. We find that our model learns stellar properties directly from the data itself. We then apply our model to XP/APOGEE giant stars to study the [$\alpha$/M] information in $\textit{Gaia}$ XP. We provide strong evidence that the XP spectra contain meaningful [$\alpha$/M] information by demonstrating that our model learns the $\alpha$-bimodality $\textit{without relying on stellar label correlations}$, for stars with $T_{\rm eff} <$ 5000 K. We publicly release our trained model, codebase and data. Importantly, our stellar label independent model can be implemented for any/all XP spectra because our model performance scales with training object density, not training label density., Comment: 18 pages, 14 figures, submitted to ApJ, data available at DOI: https://zenodo.org/doi/10.5281/zenodo.10951129, v2. Note: significant text overlap with arXiv:2307.06378. Comments welcome!

Despite the recent advances in model compression techniques for deep neural networks, deploying such models on ultra-low-power embedded devices still proves challenging. In particular, quantization schemes for Gated Recurrent Units (GRU) are difficult to tune due to their dependence on an internal state, preventing them from fully benefiting from sub-8bit quantization. In this work, we propose a modular integer quantization scheme for GRUs where the bit width of each operator can be selected independently. We then employ Genetic Algorithms (GA) to explore the vast search space of possible bit widths, simultaneously optimising for model size and accuracy. We evaluate our methods on four different sequential tasks and demonstrate that mixed-precision solutions exceed homogeneous-precision ones in terms of Pareto efficiency. In our results, we achieve a model size reduction between 25% and 55% while maintaining an accuracy comparable with the 8-bit homogeneous equivalent., Comment: Accepted as a full paper by the tinyML Research Symposium 2024

Union County Public Schools (UCPS) received a U.S. Department of Education Teacher and School Leader Incentive Program grant in 2017, supporting the EPIC program aimed at transforming education in 13 low-performing, high-poverty schools. The EPIC initiative had two primary goals: redesigning the Human Capital Management System and Performance-Based Compensation System to attract, retain, and sustain effective educators, and enhancing teacher and school leader effectiveness to positively impact student academic achievement. UCPS implemented a Performance-Based Compensation System (PBCS) that provided monetary incentives to individual teachers and school leaders based on attendance, growth targets, and evaluations. The EPIC impact study evaluated the program's effects on student outcomes using a short-interrupted time series design with comparison (SIT-C), comparing standardized achievement scores in math and reading for Grades 5 and 8, and high school levels, with matched comparison schools. The study spanned five years, tracking proficiency scores before and after EPIC implementation. Results at the elementary level showed that neither the treatment nor comparison groups achieved the projected mean percent proficient in post-intervention years for math. However, the treatment group exceeded the mean in reading, unlike the comparison group. In middle schools, the treatment group surpassed the projected mean percent proficient in both math and reading, while the comparison group did not. At the high school level, neither group achieved the projected mean percent proficient in math post-intervention, but the treatment group exceeded it in English compared to the comparison group. Teacher outcomes indicated improvements in retention and diversity across EPIC schools. Seven of the 13 schools saw a substantial increase in teacher retention, and the percentage of teachers of color increased by 10 points over five years. Teacher attendance showed improvement, with a decrease in average days absent. UCPS's EPIC program demonstrated positive outcomes in teacher retention, diversity, and attendance, with varying impacts on student proficiency across grade levels. The study emphasized the need for cautious interpretation of data due to external factors like the COVID pandemic. [The report was submitted to Union County Public Schools.]

While bisimulation-based approaches hold promise for learning robust state representations for Reinforcement Learning (RL) tasks, their efficacy in offline RL tasks has not been up to par. In some instances, their performance has even significantly underperformed alternative methods. We aim to understand why bisimulation methods succeed in online settings, but falter in offline tasks. Our analysis reveals that missing transitions in the dataset are particularly harmful to the bisimulation principle, leading to ineffective estimation. We also shed light on the critical role of reward scaling in bounding the scale of bisimulation measurements and of the value error they induce. Based on these findings, we propose to apply the expectile operator for representation learning to our offline RL setting, which helps to prevent overfitting to incomplete data. Meanwhile, by introducing an appropriate reward scaling strategy, we avoid the risk of feature collapse in representation space. We implement these recommendations on two state-of-the-art bisimulation-based algorithms, MICo and SimSR, and demonstrate performance gains on two benchmark suites: D4RL and Visual D4RL. Codes are provided at \url{https://github.com/zanghyu/Offline_Bisimulation}., Comment: NeurIPS 2023

Coulomb drag experiments have been an essential tool to study strongly interacting low-dimensional systems. Historically, this effect has been explained in terms of momentum transfer between electrons in the active and the passive layer. Here, we report Coulomb drag measurements between laterally coupled GaAs/AlGaAs quantum wires in the multiple 1D sub-band regime that break Onsager's reciprocity upon both layer and current direction reversal, in contrast to prior 1D Coulomb drag results. The drag signal shows nonlinear I-V characteristics, which are well characterized by a third-order polynomial fit. These findings are qualitatively consistent with a rectified drag signal induced by charge fluctuations. However, the nonmonotonic temperature dependence of this drag signal suggests that strong electron-electron interactions, expected within the Tomonaga-Luttinger liquid framework, remain important and standard interaction models are insufficient to capture the qualitative nature of rectified 1D Coulomb drag., Comment: 8 pages, 7 figures

The Ribbit system is a compact Scheme implementation running on the Ribbit Virtual Machine (RVM) that has been ported to a dozen host languages. It supports a simple Foreign Function Interface (FFI) allowing extensions to the RVM directly from the program's source code. We have extended the system to offer conformance to the R4RS standard while staying as compact as possible. This leads to a R4RS compliant REPL that fits in an 7 KB Linux executable. This paper explains the various issues encountered and our solutions to make, arguably, the smallest R4RS conformant Scheme implementation of all time., Comment: Presented at The 2023 Scheme and Functional Programming Workshop (arXiv:cs/0101200)

Offline policy learning is aimed at learning decision-making policies using existing datasets of trajectories without collecting additional data. The primary motivation for using reinforcement learning (RL) instead of supervised learning techniques such as behavior cloning is to find a policy that achieves a higher average return than the trajectories constituting the dataset. However, we empirically find that when a dataset is dominated by suboptimal trajectories, state-of-the-art offline RL algorithms do not substantially improve over the average return of trajectories in the dataset. We argue this is due to an assumption made by current offline RL algorithms of staying close to the trajectories in the dataset. If the dataset primarily consists of sub-optimal trajectories, this assumption forces the policy to mimic the suboptimal actions. We overcome this issue by proposing a sampling strategy that enables the policy to only be constrained to ``good data" rather than all actions in the dataset (i.e., uniform sampling). We present a realization of the sampling strategy and an algorithm that can be used as a plug-and-play module in standard offline RL algorithms. Our evaluation demonstrates significant performance gains in 72 imbalanced datasets, D4RL dataset, and across three different offline RL algorithms. Code is available at https://github.com/Improbable-AI/dw-offline-rl., Comment: Accepted NeurIPS 2023

Inspired by human conscious planning, we propose Skipper, a model-based reinforcement learning framework utilizing spatio-temporal abstractions to generalize better in novel situations. It automatically decomposes the given task into smaller, more manageable subtasks, and thus enables sparse decision-making and focused computation on the relevant parts of the environment. The decomposition relies on the extraction of an abstracted proxy problem represented as a directed graph, in which vertices and edges are learned end-to-end from hindsight. Our theoretical analyses provide performance guarantees under appropriate assumptions and establish where our approach is expected to be helpful. Generalization-focused experiments validate Skipper's significant advantage in zero-shot generalization, compared to some existing state-of-the-art hierarchical planning methods., Comment: ICLR 2024 Camera-Ready

Using diffusion models to solve inverse problems is a growing field of research. Current methods assume the degradation to be known and provide impressive results in terms of restoration quality and diversity. In this work, we leverage the efficiency of those models to jointly estimate the restored image and unknown parameters of the degradation model such as blur kernel. In particular, we designed an algorithm based on the well-known Expectation-Minimization (EM) estimation method and diffusion models. Our method alternates between approximating the expected log-likelihood of the inverse problem using samples drawn from a diffusion model and a maximization step to estimate unknown model parameters. For the maximization step, we also introduce a novel blur kernel regularization based on a Plug \& Play denoiser. Diffusion models are long to run, thus we provide a fast version of our algorithm. Extensive experiments on blind image deblurring demonstrate the effectiveness of our method when compared to other state-of-the-art approaches.

Although deep learning has made strides in the field of deep noise suppression, leveraging deep architectures on resource-constrained devices still proved challenging. Therefore, we present an early-exiting model based on nsNet2 that provides several levels of accuracy and resource savings by halting computations at different stages. Moreover, we adapt the original architecture by splitting the information flow to take into account the injected dynamism. We show the trade-offs between performance and computational complexity based on established metrics., Comment: Accepted at the MLSP 2023

The recent release of 220+ million BP/RP spectra in $\textit{Gaia}$ DR3 presents an opportunity to apply deep learning models to an unprecedented number of stellar spectra, at extremely low-resolution. The BP/RP dataset is so massive that no previous spectroscopic survey can provide enough stellar labels to cover the BP/RP parameter space. We present an unsupervised, deep, generative model for BP/RP spectra: a $\textit{scatter}$ variational auto-encoder. We design a non-traditional variational auto-encoder which is capable of modeling both $(i)$ BP/RP coefficients and $(ii)$ intrinsic scatter. Our model learns a latent space from which to generate BP/RP spectra (scatter) directly from the data itself without requiring any stellar labels. We demonstrate that our model accurately reproduces BP/RP spectra in regions of parameter space where supervised learning fails or cannot be implemented., Comment: Accepted at the ICML 2023 Workshop on Machine Learning for Astrophysics. 6 pages, 3 figures

Most offline reinforcement learning (RL) algorithms return a target policy maximizing a trade-off between (1) the expected performance gain over the behavior policy that collected the dataset, and (2) the risk stemming from the out-of-distribution-ness of the induced state-action occupancy. It follows that the performance of the target policy is strongly related to the performance of the behavior policy and, thus, the trajectory return distribution of the dataset. We show that in mixed datasets consisting of mostly low-return trajectories and minor high-return trajectories, state-of-the-art offline RL algorithms are overly restrained by low-return trajectories and fail to exploit high-performing trajectories to the fullest. To overcome this issue, we show that, in deterministic MDPs with stochastic initial states, the dataset sampling can be re-weighted to induce an artificial dataset whose behavior policy has a higher return. This re-weighted sampling strategy may be combined with any offline RL algorithm. We further analyze that the opportunity for performance improvement over the behavior policy correlates with the positive-sided variance of the returns of the trajectories in the dataset. We empirically show that while CQL, IQL, and TD3+BC achieve only a part of this potential policy improvement, these same algorithms combined with our reweighted sampling strategy fully exploit the dataset. Furthermore, we empirically demonstrate that, despite its theoretical limitation, the approach may still be efficient in stochastic environments. The code is available at https://github.com/Improbable-AI/harness-offline-rl.

The adoption of self-driving cars will certainly revolutionize our lives, even though they may take more time to become fully autonomous than initially predicted. The first vehicles are already present in certain cities of the world, as part of experimental robot-taxi services. However, most existing studies focus on the navigation part of such vehicles. We currently miss methods, datasets, and studies to assess the in-cabin human component of the adoption of such technology in real-world conditions. This paper proposes an experimental framework to study the activities of occupants of self-driving cars using a multidisciplinary approach (computer vision associated with human and social sciences), particularly non-driving related activities. The framework is composed of an experimentation scenario, and a data acquisition module. We seek firstly to capture real-world data about the usage of the vehicle in the nearest possible, real-world conditions, and secondly to create a dataset containing in-cabin human activities to foster the development and evaluation of computer vision algorithms. The acquisition module records multiple views of the front seats of the vehicle (Intel RGB-D and GoPro cameras); in addition to survey data about the internal states and attitudes of participants towards this type of vehicle before, during, and after the experimentation. We evaluated the proposed framework with the realization of real-world experimentation with 30 participants (1 hour each) to study the acceptance of SDCs of SAE level 4., Comment: This paper is currently under review by the 26th IEEE International Conference on Intelligent Transportation Systems (ITSC 2023)

Speech enhancement techniques improve the quality or the intelligibility of an audio signal by removing unwanted noise. It is used as preprocessing in numerous applications such as speech recognition, hearing aids, broadcasting and telephony. The evaluation of such algorithms often relies on reference-based objective metrics that are shown to correlate poorly with human perception. In order to evaluate audio quality as perceived by human observers it is thus fundamental to resort to subjective quality assessment. In this paper, a user evaluation based on crowdsourcing (subjective) and the Comparison Category Rating (CCR) method is compared against the DNSMOS, ViSQOL and 3QUEST (objective) metrics. The overall quality scores of three speech enhancement algorithms from real time communications (RTC) are used in the comparison using the P.808 toolkit. Results indicate that while the CCR scale allows participants to identify differences between processed and unprocessed audio samples, two groups of preferences emerge: some users rate positively by focusing on noise suppression processing, while others rate negatively by focusing mainly on speech quality. We further present results on the parameters, size considerations and speaker variations that are critical and should be considered when designing the CCR-based crowdsourcing evaluation., Comment: Published at ICASSP 2023

Decision Transformer-based decision-making agents have shown the ability to generalize across multiple tasks. However, their performance relies on massive data and computation. We argue that this inefficiency stems from the forgetting phenomenon, in which a model memorizes its behaviors in parameters throughout training. As a result, training on a new task may deteriorate the model's performance on previous tasks. In contrast to LLMs' implicit memory mechanism, the human brain utilizes distributed memory storage, which helps manage and organize multiple skills efficiently, mitigating the forgetting phenomenon. Inspired by this, we propose a working memory module to store, blend, and retrieve information for different downstream tasks. Evaluation results show that the proposed method improves training efficiency and generalization in Atari games and Meta-World object manipulation tasks. Moreover, we demonstrate that memory fine-tuning further enhances the adaptability of the proposed architecture., Comment: Accepted at ICML 2024

Monitoring pesticide concentration is very important for public authorities given the major concerns for environmental safety and the likelihood for increased public health risks. An important aspect of this process consists in locating abnormal signals, from a large amount of collected data. This kind of data is usually complex since it suffers from limits of quantification leading to left censored observations, and from the sampling procedure which is irregular in time and space across measuring stations. The present manuscript tackles precisely the issue of detecting spatio-temporal collective anomalies in pesticide concentration levels, and introduces a novel methodology for dealing with spatio-temporal heterogeneity. The latter combines a change-point detection procedure applied to the series of maximum daily values across all stations, and a clustering step aimed at a spatial segmentation of the stations. Limits of quantification are handled in the change-point procedure, by supposing an underlying left-censored parametric model, piece-wise stationary. Spatial segmentation takes into account the geographical conditions, and may be based on river network, wind directions, etc. Conditionally to the temporal segment and the spatial cluster, one may eventually analyse the data and identify contextual anomalies. The proposed procedure is illustrated in detail on a data set containing the prosulfocarb concentration levels in surface waters in Centre-Val de Loire region., Comment: Environmetrics, In press

Offline reinforcement learning (RL) struggles in environments with rich and noisy inputs, where the agent only has access to a fixed dataset without environment interactions. Past works have proposed common workarounds based on the pre-training of state representations, followed by policy training. In this work, we introduce a simple, yet effective approach for learning state representations. Our method, Behavior Prior Representation (BPR), learns state representations with an easy-to-integrate objective based on behavior cloning of the dataset: we first learn a state representation by mimicking actions from the dataset, and then train a policy on top of the fixed representation, using any off-the-shelf Offline RL algorithm. Theoretically, we prove that BPR carries out performance guarantees when integrated into algorithms that have either policy improvement guarantees (conservative algorithms) or produce lower bounds of the policy values (pessimistic algorithms). Empirically, we show that BPR combined with existing state-of-the-art Offline RL algorithms leads to significant improvements across several offline control benchmarks. The code is available at \url{https://github.com/bit1029public/offline_bpr}., Comment: ICLR 2023

Goal-conditioned reinforcement learning (RL) is a promising direction for training agents that are capable of solving multiple tasks and reach a diverse set of objectives. How to \textit{specify} and \textit{ground} these goals in such a way that we can both reliably reach goals during training as well as generalize to new goals during evaluation remains an open area of research. Defining goals in the space of noisy and high-dimensional sensory inputs poses a challenge for training goal-conditioned agents, or even for generalization to novel goals. We propose to address this by learning factorial representations of goals and processing the resulting representation via a discretization bottleneck, for coarser goal specification, through an approach we call DGRL. We show that applying a discretizing bottleneck can improve performance in goal-conditioned RL setups, by experimentally evaluating this method on tasks ranging from maze environments to complex robotic navigation and manipulation. Additionally, we prove a theorem lower-bounding the expected return on out-of-distribution goals, while still allowing for specifying goals with expressive combinatorial structure., Comment: Neurips 2022

Plug & Play methods combine proximal algorithms with denoiser priors to solve inverse problems. These methods rely on the computability of the proximal operator of the data fidelity term. In this paper, we propose a Plug & Play framework based on linearized ADMM that allows us to bypass the computation of intractable proximal operators. We demonstrate the convergence of the algorithm and provide results on restoration tasks such as super-resolution and deblurring with non-uniform blur.

In this paper, we investigate whether artificial agents can develop a shared language in an ecological setting where communication relies on a sensory-motor channel. To this end, we introduce the Graphical Referential Game (GREG) where a speaker must produce a graphical utterance to name a visual referent object while a listener has to select the corresponding object among distractor referents, given the delivered message. The utterances are drawing images produced using dynamical motor primitives combined with a sketching library. To tackle GREG we present CURVES: a multimodal contrastive deep learning mechanism that represents the energy (alignment) between named referents and utterances generated through gradient ascent on the learned energy landscape. We demonstrate that CURVES not only succeeds at solving the GREG but also enables agents to self-organize a language that generalizes to feature compositions never seen during training. In addition to evaluating the communication performance of our approach, we also explore the structure of the emerging language. Specifically, we show that the resulting language forms a coherent lexicon shared between agents and that basic compositional rules on the graphical productions could not explain the compositional generalization.

Ultra-light dark matter (ULDM) refers to a class of theories, including ultra-light axions, in which particles with mass $m_{\psi} < 10^{-20}\, \rm{eV}$ comprise a significant fraction of the dark matter. A galactic scale de Broglie wavelength distinguishes these theories from cold dark matter (CDM), suppressing the overall abundance of structure on sub-galactic scales, and producing wave-like interference phenomena in the density profiles of halos. With the aim of constraining the particle mass, we analyze the flux ratios in a sample of eleven quadruple-image strong gravitational lenses. We account for the suppression of the halo mass function and concentration-mass relation predicted by ULDM theories, and the wave-like fluctuations in the host halo density profile, calibrating the model for the wave interference against numerical simulations of galactic-scale halos. We show that the granular structure of halo density profiles, in particular, the amplitude of the fluctuations, significantly impacts image flux ratios, and therefore inferences on the particle mass derived from these data. We infer relative likelihoods of CDM to ULDM of 8:1, 7:1, 6:1, and 4:1 for particle masses $\log_{10}(m_\psi/\rm{eV})\in[-22.5,-22.25], [-22.25,-22.0],[-22.0,-21.75], [-21.75,-21.5]$, respectively. Repeating the analysis and omitting fluctuations associated with the wave interference effects, we obtain relative likelihoods of CDM to ULDM with a particle mass in the same ranges of 98:1, 48:1, 26:1 and 18:1, highlighting the significant perturbation to image flux ratios associated with the fluctuations. Nevertheless, our results disfavor the lightest particle masses with $m_{\psi} < 10^{-21.5}\,\rm{eV}$, adding to mounting pressure on ultra-light axions as a viable dark matter candidate., Comment: 17 pages, 10 figures. Accepted for publication in MNRAS

We address the problem of evaluating the quality of self-supervised learning (SSL) models without access to supervised labels, while being agnostic to the architecture, learning algorithm or data manipulation used during training. We argue that representations can be evaluated through the lens of expressiveness and learnability. We propose to use the Intrinsic Dimension (ID) to assess expressiveness and introduce Cluster Learnability (CL) to assess learnability. CL is measured in terms of the performance of a KNN classifier trained to predict labels obtained by clustering the representations with K-means. We thus combine CL and ID into a single predictor -- CLID. Through a large-scale empirical study with a diverse family of SSL algorithms, we find that CLID better correlates with in-distribution model performance than other competing recent evaluation schemes. We also benchmark CLID on out-of-domain generalization, where CLID serves as a predictor of the transfer performance of SSL models on several visual classification tasks, yielding improvements with respect to the competing baselines.

Most theoretically motivated work in the offline reinforcement learning setting requires precise uncertainty estimates. This requirement restricts the algorithms derived in that work to the tabular and linear settings where such estimates exist. In this work, we develop a novel method for incorporating scalable uncertainty estimates into an offline reinforcement learning algorithm called deep-SPIBB that extends the SPIBB family of algorithms to environments with larger state and action spaces. We use recent innovations in uncertainty estimation from the deep learning community to get more scalable uncertainty estimates to plug into deep-SPIBB. While these uncertainty estimates do not allow for the same theoretical guarantees as in the tabular case, we argue that the SPIBB mechanism for incorporating uncertainty is more robust and flexible than pessimistic approaches that incorporate the uncertainty as a value function penalty. We bear this out empirically, showing that deep-SPIBB outperforms pessimism based approaches with access to the same uncertainty estimates and performs at least on par with a variety of other strong baselines across several environments and datasets.

Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL). RCSL algorithms learn the distribution of actions conditioned on both the state and the return of the trajectory. Then they define a policy by conditioning on achieving high return. In this paper, we provide a rigorous study of the capabilities and limitations of RCSL, something which is crucially missing in previous work. We find that RCSL returns the optimal policy under a set of assumptions that are stronger than those needed for the more traditional dynamic programming-based algorithms. We provide specific examples of MDPs and datasets that illustrate the necessity of these assumptions and the limits of RCSL. Finally, we present empirical evidence that these limitations will also cause issues in practice by providing illustrative experiments in simple point-mass environments and on datasets from the D4RL benchmark.

A central object of study in Reinforcement Learning (RL) is the Markovian policy, in which an agent's actions are chosen from a memoryless probability distribution, conditioned only on its current state. The family of Markovian policies is broad enough to be interesting, yet simple enough to be amenable to analysis. However, RL often involves more complex policies: ensembles of policies, policies over options, policies updated online, etc. Our main contribution is to prove that the occupancy measure of any non-Markovian policy, i.e., the distribution of transition samples collected with it, can be equivalently generated by a Markovian policy. This result allows theorems about the Markovian policy class to be directly extended to its non-Markovian counterpart, greatly simplifying proofs, in particular those involving replay buffers and datasets. We provide various examples of such applications to the field of Reinforcement Learning., Comment: 9p+sup. mat

We propose a state-of-the-art method for super-resolution with non-uniform blur. Single-image super-resolution methods seek to restore a high-resolution image from blurred, subsampled, and noisy measurements. Despite their impressive performance, existing techniques usually assume a uniform blur kernel. Hence, these techniques do not generalize well to the more general case of non-uniform blur. Instead, in this paper, we address the more realistic and computationally challenging case of spatially-varying blur. To this end, we first propose a fast deep plug-and-play algorithm, based on linearized ADMM splitting techniques, which can solve the super-resolution problem with spatially-varying blur. Second, we unfold our iterative algorithm into a single network and train it end-to-end. In this way, we overcome the intricacy of manually tuning the parameters involved in the optimization scheme. Our algorithm presents remarkable performance and generalizes well after a single training to a large family of spatially-varying blur kernels, noise levels and scale factors.

Humans have the capability, aided by the expressive compositionality of their language, to learn quickly by demonstration. They are able to describe unseen task-performing procedures and generalize their execution to other contexts. In this work, we introduce DescribeWorld, an environment designed to test this sort of generalization skill in grounded agents, where tasks are linguistically and procedurally composed of elementary concepts. The agent observes a single task demonstration in a Minecraft-like grid world, and is then asked to carry out the same task in a new map. To enable such a level of generalization, we propose a neural agent infused with hierarchical latent language--both at the level of task inference and subtask planning. Our agent first generates a textual description of the demonstrated unseen task, then leverages this description to replicate it. Through multiple evaluation scenarios and a suite of generalization tests, we find that agents that perform text-based inference are better equipped for the challenge under a random split of tasks.

In Reinforcement Learning, the optimal action at a given state is dependent on policy decisions at subsequent states. As a consequence, the learning targets evolve with time and the policy optimization process must be efficient at unlearning what it previously learnt. In this paper, we discover that the policy gradient theorem prescribes policy updates that are slow to unlearn because of their structural symmetry with respect to the value target. To increase the unlearning speed, we study a novel policy update: the gradient of the cross-entropy loss with respect to the action maximizing $q$, but find that such updates may lead to a decrease in value. Consequently, we introduce a modified policy update devoid of that flaw, and prove its guarantees of convergence to global optimality in $\mathcal{O}(t^{-1})$ under classic assumptions. Further, we assess standard policy updates and our cross-entropy policy updates along six analytical dimensions. Finally, we empirically validate our theoretical findings., Comment: 9p+appendix, accepted to AISTATS 2022

SARSA, a classical on-policy control algorithm for reinforcement learning, is known to chatter when combined with linear function approximation: SARSA does not diverge but oscillates in a bounded region. However, little is known about how fast SARSA converges to that region and how large the region is. In this paper, we make progress towards this open problem by showing the convergence rate of projected SARSA to a bounded region. Importantly, the region is much smaller than the region that we project into, provided that the magnitude of the reward is not too large. Existing works regarding the convergence of linear SARSA to a fixed point all require the Lipschitz constant of SARSA's policy improvement operator to be sufficiently small; our analysis instead applies to arbitrary Lipschitz constants and thus characterizes the behavior of linear SARSA for a new regime., Comment: ICML 2023

We report low-temperature magneto-transport measurements of an undoped Si/SiGe asymmetric double quantum well heterostructure. The density in both layers is tuned independently utilizing a top and a bottom gate, allowing the investigation of quantum wells at both imbalanced and matched densities. Integer quantum Hall states at total filling factor $\nu_{\text{T}} = 1$ and $\nu_{\text{T}} = 2$ are observed in both density regimes, and the evolution of their excitation gaps is reported as a function of density. The $\nu_{\text{T}} = 1$ gap evolution departs from the behavior generally observed for valley splitting in the single layer regime. Furthermore, by comparing the $\nu_{\text{T}} = 2$ gap to the single particle tunneling energy, $\Delta_{\text{SAS}}$, obtained from Schr\"{o}dinger-Poisson (SP) simulations, evidence for the onset of spontaneous inter-layer coherence (SIC) is observed for a relative filling fraction imbalance smaller than ${\sim}50\%$, Comment: 6 pages, 3 figures, accepted in APL

In this paper, we establish the global optimality and convergence rate of an off-policy actor critic algorithm in the tabular setting without using density ratio to correct the discrepancy between the state distribution of the behavior policy and that of the target policy. Our work goes beyond existing works on the optimality of policy gradient methods in that existing works use the exact policy gradient for updating the policy parameters while we use an approximate and stochastic update step. Our update step is not a gradient update because we do not use a density ratio to correct the state distribution, which aligns well with what practitioners do. Our update is approximate because we use a learned critic instead of the true value function. Our update is stochastic because at each step the update is done for only the current state action pair. Moreover, we remove several restrictive assumptions from existing works in our analysis. Central to our work is the finite sample analysis of a generic stochastic approximation algorithm with time-inhomogeneous update operators on time-inhomogeneous Markov chains, based on its uniform contraction properties., Comment: Journal of Machine Learning Research 2022

Three-dimensional turbulence is usually studied experimentally by using a spatially localized forcing at large scales (e.g. via rotating blades or oscillating grids), often in a deterministic way. Here, we report an original technique where the fluid is forced in volume, randomly in space and time, using small magnetic particles remotely driven. Such a forcing generates almost no mean flow and is closer to those of direct numerical simulations of isotropic homogeneous turbulence. We compute the energy spectra and structure functions using local and spatiotemporal flow velocity measurements. The energy dissipation rate is also evaluated consistently in five different ways. Our experimental results confirm the stationary, homogeneous and isotropic features of such turbulence, and in particular the Tennekes' model for which the Tennekes' constant is experimentally estimated., Comment: submitted to Phys. Rev. Fluids (Letters)

In Batched Multi-Armed Bandits (BMAB), the policy is not allowed to be updated at each time step. Usually, the setting asserts a maximum number of allowed policy updates and the algorithm schedules them so that to minimize the expected regret. In this paper, we describe a novel setting for BMAB, with the following twist: the timing of the policy update is not controlled by the BMAB algorithm, but instead the amount of data received during each batch, called \textit{crowd}, is influenced by the past selection of arms. We first design a near-optimal policy with approximate knowledge of the parameters that we prove to have a regret in $\mathcal{O}(\sqrt{\frac{\ln x}{x}}+\epsilon)$ where $x$ is the size of the crowd and $\epsilon$ is the parameter error. Next, we implement a UCB-inspired algorithm that guarantees an additional regret in $\mathcal{O}\left(\max(K\ln T,\sqrt{T\ln T})\right)$, where $K$ is the number of arms and $T$ is the horizon., Comment: 31 pages

The policy gradient theorem states that the policy should only be updated in states that are visited by the current policy, which leads to insufficient planning in the off-policy states, and thus to convergence to suboptimal policies. We tackle this planning issue by extending the policy gradient theory to policy updates with respect to any state density. Under these generalized policy updates, we show convergence to optimality under a necessary and sufficient condition on the updates' state densities, and thereby solve the aforementioned planning issue. We also prove asymptotic convergence rates that significantly improve those in the policy gradient literature. To implement the principles prescribed by our theory, we propose an agent, Dr Jekyll & Mr Hyde (JH), with a double personality: Dr Jekyll purely exploits while Mr Hyde purely explores. JH's independent policies allow to record two separate replay buffers: one on-policy (Dr Jekyll's) and one off-policy (Mr Hyde's), and therefore to update JH's models with a mixture of on-policy and off-policy updates. More than an algorithm, JH defines principles for actor-critic algorithms to satisfy the requirements we identify in our analysis. We extensively test on finite MDPs where JH demonstrates a superior ability to recover from converging to a suboptimal policy without impairing its speed of convergence. We also implement a deep version of the algorithm and test it on a simple problem where it shows promising results., Comment: accepted to NeurIPS as a poster

By the age of two, children tend to assume that new word categories are based on objects' shape, rather than their color or texture; this assumption is called the shape bias. They are thought to learn this bias by observing that their caregiver's language is biased towards shape based categories. This presents a chicken and egg problem: if the shape bias must be present in the language in order for children to learn it, how did it arise in language in the first place? In this paper, we propose that communicative efficiency explains both how the shape bias emerged and why it persists across generations. We model this process with neural emergent language agents that learn to communicate about raw pixelated images. First, we show that the shape bias emerges as a result of efficient communication strategies employed by agents. Second, we show that pressure brought on by communicative need is also necessary for it to persist across generations; simply having a shape bias in an agent's input language is insufficient. These results suggest that, over and above the operation of other learning strategies, the shape bias in human learners may emerge and be sustained by communicative pressures., Comment: Accepted at CoNLL 2021

We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning (RL) setting. We consider the scenario where: (i) we have a dataset collected under a known baseline policy, (ii) multiple reward signals are received from the environment inducing as many objectives to optimize. We present an SPI formulation for this RL setting that takes into account the preferences of the algorithm's user for handling the trade-offs for different reward signals while ensuring that the new policy performs at least as well as the baseline policy along each individual objective. We build on traditional SPI algorithms and propose a novel method based on Safe Policy Iteration with Baseline Bootstrapping (SPIBB, Laroche et al., 2019) that provides high probability guarantees on the performance of the agent in the true environment. We show the effectiveness of our method on a synthetic grid-world safety task as well as in a real-world critical care context to learn a policy for the administration of IV fluids and vasopressors to treat sepsis.

The ability to cultivate the cyanobacterium Arhtrospira platensis in artificially lightened photobioreactors using high energetic efficiency (quasi-monochromatic) red LED was investigated. In order to reach the same maximal productivities as with the polychromatic lightening control conditions (red + blue, P/2e- = 1.275), the need to work with an optimal range of wavelength around 620 nm was first established on batch and continuous cultures. The long-term physiological and kinetic behavior was then verified in a continuous photobioreactor illuminated only with red (620 nm) LED, showing that the maximum productivities can be maintained over 30 residence times with only minor changes in the pigment content of the cells corresponding to a well-known adaptation mechanism of the photosystems, but without any effect on growth and stoichiometry. For both poly and monochromatic incident light inputs, a predictive spectral knowledge model was proposed and validated for the first time, allowing the calculation of the kinetics and stoichiometry observed in any photobioreactor cultivating A. platensis, or other cyanobacteria if the parameters were updated. It is shown that the photon flux (with a specified wavelength) must be used instead of light energy flux as a relevant control variable for the growth. The experimental and theoretical results obtained in this study demonstrate that it is possible to save the energy consumed by the lightening device of photobioreactors using red LED, the spectral range of which is defined according to the action spectrum of photosynthesis. This appears to be crucial information for applications in which the energy must be rationalized, as it is the case for life support systems in closed environments like a permanent spatial base or a submarine.

We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e., there is a $\gamma^t$ term in the actor update for the transition observed at time $t$ in a trajectory and the critic is a discounted value function. Practitioners, however, usually ignore the discounting ($\gamma^t$) for the actor while using a discounted critic. We investigate this mismatch in two scenarios. In the first scenario, we consider optimizing an undiscounted objective $(\gamma = 1)$ where $\gamma^t$ disappears naturally $(1^t = 1)$. We then propose to interpret the discounting in critic in terms of a bias-variance-representation trade-off and provide supporting empirical results. In the second scenario, we consider optimizing a discounted objective ($\gamma < 1$) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results., Comment: AAMAS 2022

Orthogonal Monte Carlo (OMC) is a very effective sampling algorithm imposing structural geometric conditions (orthogonality) on samples for variance reduction. Due to its simplicity and superior performance as compared to its Quasi Monte Carlo counterparts, OMC is used in a wide spectrum of challenging machine learning applications ranging from scalable kernel methods to predictive recurrent neural networks, generative models and reinforcement learning. However theoretical understanding of the method remains very limited. In this paper we shed new light on the theoretical principles behind OMC, applying theory of negatively dependent random variables to obtain several new concentration results. We also propose a novel extensions of the method leveraging number theory techniques and particle algorithms, called Near-Orthogonal Monte Carlo (NOMC). We show that NOMC is the first algorithm consistently outperforming OMC in applications ranging from kernel methods to approximating distances in probabilistic metric spaces., Comment: 22 pages, 4 figures

We introduce a new way to implicitly represent swept volumes in 3D. We first implicitize the base volume and then apply the time-dependent rigid transformation to build an implicit representation of the swept volume. This way, we build a fitting representation, where geometric details generated by the rigid transformation and details of the base volume itself are both taken into account. This representation relies on a partition of a boundary box of the swept volume according to portions of the space related either to parts of the base volume or to time span of the transformation.

In fluorescence microscopy live-cell imaging, there is a critical trade-off between the signal-to-noise ratio and spatial resolution on one side, and the integrity of the biological sample on the other side. To obtain clean high-resolution (HR) images, one can either use microscopy techniques, such as structured-illumination microscopy (SIM), or apply denoising and super-resolution (SR) algorithms. However, the former option requires multiple shots that can damage the samples, and although efficient deep learning based algorithms exist for the latter option, no benchmark exists to evaluate these algorithms on the joint denoising and SR (JDSR) tasks. To study JDSR on microscopy data, we propose such a novel JDSR dataset, Widefield2SIM (W2S), acquired using a conventional fluorescence widefield and SIM imaging. W2S includes 144,000 real fluorescence microscopy images, resulting in a total of 360 sets of images. A set is comprised of noisy low-resolution (LR) widefield images with different noise levels, a noise-free LR image, and a corresponding high-quality HR SIM image. W2S allows us to benchmark the combinations of 6 denoising methods and 6 SR methods. We show that state-of-the-art SR networks perform very poorly on noisy inputs. Our evaluation also reveals that applying the best denoiser in terms of reconstruction error followed by the best SR method does not necessarily yield the best final result. Both quantitative and qualitative results show that SR networks are sensitive to noise and the sequential application of denoising and SR algorithms is sub-optimal. Lastly, we demonstrate that SR networks retrained end-to-end for JDSR outperform any combination of state-of-the-art deep denoising and SR networks, Comment: ECCVW 2020. Project page: \

Malfunctioning neurons in the brain sometimes operate synchronously, reportedly causing many neurological diseases, e.g. Parkinson's. Suppression and control of this collective synchronous activity are therefore of great importance for neuroscience, and can only rely on limited engineering trials due to the need to experiment with live human brains. We present the first Reinforcement Learning gym framework that emulates this collective behavior of neurons and allows us to find suppression parameters for the environment of synthetic degenerate models of neurons. We successfully suppress synchrony via RL for three pathological signaling regimes, characterize the framework's stability to noise, and further remove the unwanted oscillations by engaging multiple PPO agents., Comment: 7 pages + 1 references, 7 figures. arXiv admin note: text overlap with arXiv:1909.12154

Playing text-based games requires skills in processing natural language and sequential decision making. Achieving human-level performance on text-based games remains an open challenge, and prior research has largely relied on hand-crafted structured representations and heuristics. In this work, we investigate how an agent can plan and generalize in text-based games using graph-structured representations learned end-to-end from raw text. We propose a novel graph-aided transformer agent (GATA) that infers and updates latent belief graphs during planning to enable effective action selection by capturing the underlying game dynamics. GATA is trained using a combination of reinforcement and self-supervised learning. Our work demonstrates that the learned graph-based representations help agents converge to better policies than their text-only counterparts and facilitate effective generalization across game configurations. Experiments on 500+ unique games from the TextWorld suite show that our best agent outperforms text-based baselines by an average of 24.2%., Comment: Bug fixed in Table 1

In this paper, the objective is to assess the histomechanical effects of collagen proteolysis in arteries under loading conditions reproducing in vivo environment. Thirteen segments of common porcine carotid arteries (8 proximal and 5 distal) were immersed in a bath of bacterial collagenase and tested with a pulsatile tension/inflation machine. Diameter, pressure and axial load were monitored throughout the tests and used to derive the stress-stretch curves and to determine the secant circumferential stiffness. Results were analyzed separately for proximal and distal segments, before and after 1, 2 and 3 hours of enzymatic degradation. A histological analysis was performed to relate the arterial microstructure to its mechanical behavior under collagen proteolysis. Control (before enzymatic degradation) and treated populations (after 1, 2 or 3 hours of enzymatic degradation) were found statistically incomparable, and histology confirmed the alteration of the fibrous structure of collagen bundles induced by the collagenase treatment. A decrease of the secant circumferential stiffness of the arterial wall was noticed mostly at the beginning of the treatment, and was less pronounced after 1 hour. These results constitute an important set of enzymatically damaged arteries that can be used to validate biomechanical computational models correlating structure and properties of blood vessels.

We are interested in learning how to update Knowledge Graphs (KG) from text. In this preliminary work, we propose a novel Sequence-to-Sequence (Seq2Seq) architecture to generate elementary KG operations. Furthermore, we introduce a new dataset for KG extraction built upon text-based game transitions (over 300k data points). We conduct experiments and discuss the results., Comment: NeurIPS 2019, Graph Representation Learning (GRL) Workshop