Search

In the era of increasing privacy concerns and demand for personalized experiences, traditional Reinforcement Learning with Human Feedback (RLHF) frameworks face significant challenges due to their reliance on centralized data. We introduce Federated Reinforcement Learning with Human Feedback (FedRLHF), a novel framework that decentralizes the RLHF process. FedRLHF enables collaborative policy learning across multiple clients without necessitating the sharing of raw data or human feedback, thereby ensuring robust privacy preservation. Leveraging federated reinforcement learning, each client integrates human feedback locally into their reward functions and updates their policies through personalized RLHF processes. We establish rigorous theoretical foundations for FedRLHF, providing convergence guarantees, and deriving sample complexity bounds that scale efficiently with the number of clients. Empirical evaluations on the MovieLens and IMDb datasets demonstrate that FedRLHF not only preserves user privacy but also achieves performance on par with centralized RLHF, while enhancing personalization across diverse client environments., Comment: Accepted to AAMAS 2025. This preprint represents the full version of the paper, including all proofs, experimental details, and additional discussions

TbV$_6$Sn$_6$ is a topological metal where ferromagnetic Tb ions with strong uniaxial magnetic anisotropy interact with V kagome layers. Inelastic neutron scattering measurements show that the Tb ions adopt an Ising doublet ground state. Here, we consider whether a transverse magnetic field can drive TbV$_6$Sn$_6$ towards a quantum critical point, providing a rare example of transverse-field Ising criticality in a metallic compound. High-field magnetization measurements suggest that this quantum criticality is avoided and reveal a first-order-like spin-reorientation transition at 25.6 T due to an excited-state level crossing. Theoretical analysis shows that small changes in the local Hamiltonian can restore the quantum criticality for some in-plane field directions, suggesting that TbV$_6$Sn$_6$ is close to a novel quantum tricritical point induced by in-plane magnetic anisotropy.

Many software engineering maintenance tasks require linking a commit that induced a bug with the commit that later fixed that bug. Several existing SZZ algorithms provide a way to identify the potential commit that induced a bug when given a fixing commit as input. Prior work introduced the notion of a "work item", a logical grouping of commits that could be a single unit of work. Our key insight in this work is to recognize that a bug-inducing commit and the fix(es) for that bug together represent a "work item." It is not currently understood how these work items, which are logical groups of revisions addressing a single issue or feature, could impact the performance of algorithms such as SZZ. In this paper, we propose a heuristic that, given an input commit, uses information about changed methods to identify related commits that form a work item with the input commit. We hypothesize that given such a work item identifying heuristic, we can identify bug-inducing commits more accurately than existing SZZ approaches. We then build a new variant of SZZ that we call Work Item Aware SZZ (WIA-SZZ), that leverages our work item detecting heuristic to first suggest bug-inducing commits. If our heuristic fails to find any candidates, we then fall back to baseline variants of SZZ. We conduct a manual evaluation to assess the accuracy of our heuristic to identify work items. Our evaluation reveals the heuristic is 64% accurate in finding work items, but most importantly it is able to find many bug-inducing commits. We then evaluate our approach on 821 repositories that have been previously used to study the performance of SZZ, comparing our work against six SZZ variants. That evaluation shows an improvement in F1 scores ranging from 2% to 9%, or when looking only at the subset of cases that found work item improved 3% to 14%.

Statically typed languages offer numerous benefits to developers, such as improved code quality and reduced runtime errors, but they also require the overhead of manual type annotations. To mitigate this burden, language designers have started incorporating support for type inference, where the compiler infers the type of a variable based on its declaration/usage context. As a result, type annotations are optional in certain contexts, and developers are empowered to use type inference in these situations. However, the usage patterns of type annotations in languages that support type inference are unclear. These patterns can help provide evidence for further research in program comprehension, in language design, and for education. We conduct a large-scale empirical study using Boa, a tool for mining software repositories, to investigate when and where developers use type inference in 498,963 Kotlin projects. We choose Kotlin because it is the default language for Android development, one of the largest software marketplaces. Additionally, Kotlin has supported declaration-site optional type annotations from its initial release. Our findings reveal that type inference is frequently employed for local variables and variables initialized with method calls declared outside the file are more likely to use type inference. These results have significant implications for language designers, providing valuable insight into where to allow type inference and how to optimize type inference algorithms for maximum efficiency, ultimately improving the development experience for developers.

Symbolic Regression remains an NP-Hard problem, with extensive research focusing on AI models for this task. Transformer models have shown promise in Symbolic Regression, but performance suffers with smaller datasets. We propose applying k-fold cross-validation to a transformer-based symbolic regression model trained on a significantly reduced dataset (15,000 data points, down from 500,000). This technique partitions the training data into multiple subsets (folds), iteratively training on some while validating on others. Our aim is to provide an estimate of model generalization and mitigate overfitting issues associated with smaller datasets. Results show that this process improves the model's output consistency and generalization by a relative improvement in validation loss of 53.31%. Potentially enabling more efficient and accessible symbolic regression in resource-constrained environments.

We explore how light-driven optical phonons can be used to drive magnetic exchange interactions into interesting physical regimes by developing a general theory of spin-phonon pumping in magnetic insulators with non-equilibrium optical phonon distributions, focusing on the diabatic regime where phonon frequencies are much larger than the magnetic interactions. We present several applications of spin-phonon pumping two-dimensional nearest-neighbor Heisenberg, XYZ and Kitaev models to examine what kind of further neighbor interactions and chiral fields can be generated, and how anisotropic couplings can be enhanced, showing that experimentally accessible non-equilibrium phonon distributions can generically drive significant frustration and realize a variety of spin liquid regimes. This effect is described for both direct and superexchange mechanisms, and we derive simple geometric rules for which phonon modes are "spin-phonon'' active and for which magnetic interactions. Spin-phonon pumping provides an intriguing possibility for preferentially pumping specific magnetic interaction terms. In addition to generating further neighbor interactions, such pumping can lead to increased magnetic anisotropy for initially weakly anisotropic models, and selectively pumping the Kitaev-Heisenberg model can suppress undesirable Heisenberg terms while enhancing Kitaev interactions., Comment: 16 pages, 10 figures

Social conventions are the foundation for social and economic life. As legions of AI agents increasingly interact with each other and with humans, their ability to form shared conventions will determine how effectively they will coordinate behaviors, integrate into society and influence it. Here, we investigate the dynamics of conventions within populations of Large Language Model (LLM) agents using simulated interactions. First, we show that globally accepted social conventions can spontaneously arise from local interactions between communicating LLMs. Second, we demonstrate how strong collective biases can emerge during this process, even when individual agents appear to be unbiased. Third, we examine how minority groups of committed LLMs can drive social change by establishing new social conventions. We show that once these minority groups reach a critical size, they can consistently overturn established behaviors. In all cases, contrasting the experimental results with predictions from a minimal multi-agent model allows us to isolate the specific role of LLM agents. Our results clarify how AI systems can autonomously develop norms without explicit programming and have implications for designing AI systems that align with human values and societal goals.

Imitation learning (IL) has shown great potential in various applications, such as robot control. However, traditional IL methods are usually designed to learn only one specific type of behavior since demonstrations typically correspond to a single expert. In this work, we introduce the first generic framework for Quality Diversity Imitation Learning (QD-IL), which enables the agent to learn a broad range of skills from limited demonstrations. Our framework integrates the principles of quality diversity with adversarial imitation learning (AIL) methods, and can potentially improve any inverse reinforcement learning (IRL) method. Empirically, our framework significantly improves the QD performance of GAIL and VAIL on the challenging continuous control tasks derived from Mujoco environments. Moreover, our method even achieves 2x expert performance in the most challenging Humanoid environment., Comment: 22 pages, conference paper

The discovery of novel quantum materials within ternary phase spaces containing antagonistic pair such as Fe with Bi, Pb, In, and Ag, presents significant challenges yet holds great potential. In this work, we investigate the stabilization of these immiscible pairs through the integration of Cerium (Ce), an abundant rare-earth and cost-effective element. By employing a machine learning (ML)-guided framework, particularly crystal graph convolutional neural networks (CGCNN), combined with first-principles calculations, we efficiently explore the composition/structure space and predict 9 stable and 37 metastable Ce-Fe-X (X=Bi, Pb, In and Ag) ternary compounds. Our findings include the identification of multiple new stable and metastable phases, which are evaluated for their structural and energetic properties. These discoveries not only contribute to the advancement of quantum materials but also offer viable alternatives to critical rare earth elements, underscoring the importance of Ce-based intermetallic compounds in technological applications.

Density functional theory (DFT), including its extensions designed to treat strongly correlated localized electron systems such as DFT+$U$ and DFT+dynamical mean field theory, has proven exceedingly useful in studying the magnetic properties of solids. However, materials with rare earths ($R$) have remained a notable exception. The most vital rare-earth magnetic properties, such as magnetocrystalline anisotropy (MA), have been notoriously elusive due to the ubiquitous self-interaction error present in nearly all available DFT flavors. In this work, we show explicitly how the orbital dependence of self-interaction error may contradict Hund's rules and plague MA calculations, and how analyzing DFT metastable states that respect Hund's rules can alleviate the problem. We systematically investigate and discuss several rare-earth-containing families, $R$Co$_5$, $R_2$Fe$_{14}$B, $R$Fe$_{12}$, and $R$Mn$_6$Sn$_6$, to benchmark the MA calculations in DFT+$U$. For all compounds we investigated, we found that our methodology reproduces the magnetic easy-axis, easy-plane, and non-trivial easy-cone anisotropies in full agreement with low-temperature experimental measurements. Besides the fully-numerical ab initio approach, we further illustrate an efficient semi-analytical perturbation method that treats the crystal field as a perturbation in the limit of large spin-orbit coupling. This approach evaluates the rare-earth anisotropy by assessing the dependence of crystal-field energy on spin-quantization axis rotation using $4f$ crystal-field levels obtained from non-spin-orbit calculations. Our analytical method provides a quantitative microscopic understanding of the factors that control MA and can be used for predicting new high-MA materials., Comment: 11 pages, 4 figures

Magnetic anisotropy is a key feature of rare earth materials from permanent magnets to heavy fermions. We explore the complex interplay of Kondo physics and anisotropy in a minimal impurity model using numerical renormalization group. While anisotropy suppresses Kondo physics, Kondo physics enhances the anisotropy. Importantly, we find distinct renormalization of the magnetic anisotropy measured via dynamical spin response (inelastic neutron scattering) versus electronic excitations in the impurity spectral function (resonant inelastic x-rays and scanning tunneling spectroscopy). The two measurement types have different responses and dependences on the temperature and Kondo scales., Comment: 6 pages, 4 figures, plus supplementary material included

techniques can be considered as an attractive alternative to the often-cumbersome traditional manufacturing routes. With the use of high-power lasers, localized hot zones that are necessary for glass making can be obtained rapidly. For instance, Laser-Powder-Deposition enables rapid fabrication of short, high gain fibers used in e.g., distributed feedback fiber lasers (DFFL). DFFLs offer sought after performance suitable for a broad range of applications in modern photonics i.e., superior stability and narrower, single-frequency linewidth compared to conventional fiber lasers. Tunable, narrow laser sources with output in eye-safe spectrum are desired for sensing, signal multiplexing, LIDAR systems, quantum applications etc. In this work we present DFFL obtained using Laser-Powder-Deposition made Er-doped silica fiber. Milliwatt level, narrow line lasing (< 704 kHz, equipment limited) was obtained using a phaseshifted grating written in 16 mm long fiber. The backward slope efficiency was as high as 24% when pumping at 976 nm. The results presented in this work showcase new possibilities in fiber fabrication that were unlocked through laser-assisted additive manufacturing. This fiber laser sets the stage for the future of rapid fabrication of advanced fiber devices through unconventional manufacturing routes.

Food insecurity, defined as the lack of physical or economic access to safe, nutritious and sufficient food, remains one of the main challenges of the 2030 Agenda for Sustainable Development. Food insecurity is a complex phenomenon, resulting from the interplay of environmental, socio-demographic, and political events. Previous work has investigated the nexus between climate change, conflict, migration and food security at the household level, however these relations are still largely unexplored at national scales. In this context, during the Complexity72h workshop, held at the Universidad Carlos III de Madrid in June 2024, we explored the co-evolution of international migration flows and food insecurity at the national scale, accounting for remittances, as well as for changes in the economic, conflict, and climate situation. To this aim, we gathered data from several publicly available sources (Food and Agriculture Organization, World Bank, and UN Department of Economic and Social Affairs) and analyzed the association between food insecurity and migration, migration and remittances, and remittances and food insecurity. We then propose a framework linking together these associations to model the co-evolution of food insecurity and international migrations.

Reinforcement learning (RL) has gained popularity in the realm of recommender systems due to its ability to optimize long-term rewards and guide users in discovering relevant content. However, the successful implementation of RL in recommender systems is challenging because of several factors, including the limited availability of online data for training on-policy methods. This scarcity requires expensive human interaction for online model training. Furthermore, the development of effective evaluation frameworks that accurately reflect the quality of models remains a fundamental challenge in recommender systems. To address these challenges, we propose a comprehensive framework for synthetic environments that simulate human behavior by harnessing the capabilities of large language models (LLMs). We complement our framework with in-depth ablation studies and demonstrate its effectiveness with experiments on movie and book recommendations. Using LLMs as synthetic users, this work introduces a modular and novel framework to train RL-based recommender systems. The software, including the RL environment, is publicly available on GitHub.

Symbolic Regression (SR) is a widely studied field of research that aims to infer symbolic expressions from data. A popular approach for SR is the Sparse Identification of Nonlinear Dynamical Systems (SINDy) framework, which uses sparse regression to identify governing equations from data. This study introduces an enhanced method, Nested SINDy, that aims to increase the expressivity of the SINDy approach thanks to a nested structure. Indeed, traditional symbolic regression and system identification methods often fail with complex systems that cannot be easily described analytically. Nested SINDy builds on the SINDy framework by introducing additional layers before and after the core SINDy layer. This allows the method to identify symbolic representations for a wider range of systems, including those with compositions and products of functions. We demonstrate the ability of the Nested SINDy approach to accurately find symbolic expressions for simple systems, such as basic trigonometric functions, and sparse (false but accurate) analytical representations for more complex systems. Our results highlight Nested SINDy's potential as a tool for symbolic regression, surpassing the traditional SINDy approach in terms of expressivity. However, we also note the challenges in the optimization process for Nested SINDy and suggest future research directions, including the designing of a more robust methodology for the optimization process. This study proves that Nested SINDy can effectively discover symbolic representations of dynamical systems from data, offering new opportunities for understanding complex systems through data-driven methods.

In this study, we delve into Federated Reinforcement Learning (FedRL) in the context of value-based agents operating across diverse Markov Decision Processes (MDPs). Existing FedRL methods typically aggregate agents' learning by averaging the value functions across them to improve their performance. However, this aggregation strategy is suboptimal in heterogeneous environments where agents converge to diverse optimal value functions. To address this problem, we introduce the Convergence-AwarE SAmpling with scReening (CAESAR) aggregation scheme designed to enhance the learning of individual agents across varied MDPs. CAESAR is an aggregation strategy used by the server that combines convergence-aware sampling with a screening mechanism. By exploiting the fact that agents learning in identical MDPs are converging to the same optimal value function, CAESAR enables the selective assimilation of knowledge from more proficient counterparts, thereby significantly enhancing the overall learning efficiency. We empirically validate our hypothesis and demonstrate the effectiveness of CAESAR in enhancing the learning efficiency of agents, using both a custom-built GridWorld environment and the classical FrozenLake-v1 task, each presenting varying levels of environmental heterogeneity.

Mining software repositories is a useful technique for researchers and practitioners to see what software developers actually do when developing software. Tools like Boa provide users with the ability to easily mine these open-source software repositories at a very large scale, with datasets containing hundreds of thousands of projects. The trade-off is that users must use the provided infrastructure, query language, runtime, and datasets and this might not fit all analysis needs. In this work, we present Boidae: a family of Boa installations controlled and customized by users. Boidae uses automation tools such as Ansible and Docker to facilitate the deployment of a customized Boa installation. In particular, Boidae allows the creation of custom datasets generated from any set of Git repositories, with helper scripts to aid in finding and cloning repositories from GitHub and SourceForge. In this paper, we briefly describe the architecture of Boidae and how researchers can utilize the infrastructure to generate custom datasets. Boidae's scripts and all infrastructure it builds upon are open-sourced. A video demonstration of Boidae's installation and extension is available at https://go.unl.edu/boidae.

In Federated Reinforcement Learning (FRL), agents aim to collaboratively learn a common task, while each agent is acting in its local environment without exchanging raw trajectories. Existing approaches for FRL either (a) do not provide any fault-tolerance guarantees (against misbehaving agents), or (b) rely on a trusted central agent (a single point of failure) for aggregating updates. We provide the first decentralized Byzantine fault-tolerant FRL method. Towards this end, we first propose a new centralized Byzantine fault-tolerant policy gradient (PG) algorithm that improves over existing methods by relying only on assumptions standard for non-fault-tolerant PG. Then, as our main contribution, we show how a combination of robust aggregation and Byzantine-resilient agreement methods can be leveraged in order to eliminate the need for a trusted central entity. Since our results represent the first sample complexity analysis for Byzantine fault-tolerant decentralized federated non-convex optimization, our technical contributions may be of independent interest. Finally, we corroborate our theoretical results experimentally for common RL environments, demonstrating the speed-up of decentralized federations w.r.t. the number of participating agents and resilience against various Byzantine attacks., Comment: Accepted at AAMAS'24

CePd3S4 exhibits interplay between ferromagnetism (FM), quadrupolar order, and the Kondo effect at low temperatures with a FM transition temperature that is much higher than the value expected from the de Gennes scaling of the heavier RPd3S4 compounds. In this work, we investigated the electrical transport and magnetic properties of CePd3S4 under pressure up through 12 GPa so as to better understand the interplay between electronic and magnetic phases in this material. Our findings show that the low pressure FM state is suddenly replaced by a new magnetically ordered phase that is most likely antiferromagnetic that spans from ~ 7 GPa to ~ 11 GPa. Whereas this could be described as an example of avoided quantum criticality, given that clear changes in resistance and Hall data are detected near 6.3 GPa for all temperatures below 300 K, it is also possible that the change in ground state is a response to a pressure induced change in structure. The lack of any discernible change in the pressure dependence of the room temperature unit cell parameter/volume across this whole pressure range suggests that this change in structure is either more subtle than could be detected by our measurements (i.e. development of weak, new wave vector) or the transition is electronic (such as a Lifshitz transition)., Comment: 16 pages, 16 figures, 74 references