Search

Electron-phonon coupling (EPC) is key for understanding many properties of materials such as superconductivity and electric resistivity. Although first principles density-functional-theory (DFT) based EPC calculations are used widely, their efficacy is limited by the accuracy and efficiency of the underlying exchange-correlation functionals. These limitations become exacerbated in complex $d$- and $f$-electron materials, where beyond-DFT approaches and empirical corrections, such as the Hubbard $U$, are commonly invoked. Here, using the examples of CoO and NiO, we show how the efficient r2scan density functional correctly captures strong EPC effects in transition-metal oxides without requiring the introduction of empirical parameters. We also demonstrate the ability of r2scan to accurately model phonon-mediated superconducting properties of the main group compounds (e.g., MgB$_2$), with improved electronic bands and phonon dispersions over those of traditional density functionals. Our study provides a pathway for extending the scope of accurate first principles modeling of electron-phonon interactions to encompass complex $d$-electron materials., Comment: 9 pages, 3 figures, 1 table

The safe and effective deployment of Large Language Models (LLMs) involves a critical step called alignment, which ensures that the model's responses are in accordance with human preferences. Prevalent alignment techniques, such as DPO, PPO and their variants, align LLMs by changing the pre-trained model weights during a phase called post-training. While predominant, these post-training methods add substantial complexity before LLMs can be deployed. Inference-time alignment methods avoid the complex post-training step and instead bias the generation towards responses that are aligned with human preferences. The best-known inference-time alignment method, called Best-of-N, is as effective as the state-of-the-art post-training procedures. Unfortunately, Best-of-N requires vastly more resources at inference time than standard decoding strategies, which makes it computationally not viable. In this work, we introduce Speculative Rejection, a computationally-viable inference-time alignment algorithm. It generates high-scoring responses according to a given reward model, like Best-of-N does, while being between 16 to 32 times more computationally efficient., Comment: NeurIPS 2024

Understanding molecular adsorption on surfaces underpins many problems in chemistry and materials science. Accurately and efficiently describing the adsorption has been a challenging task for first-principles methods as the process can involve both short-range chemical bond formations and long-range physical interactions, e.g., the van der Waals (vdW) interaction. Density functional theory presents an appealing choice for modeling adsorption reactions, though calculations with many exchange correlation density functional approximations struggle to accurately describe both chemical and physical molecular adsorptions. Here, we propose an efficient density functional approximation that is accurate for both chemical and physical adsorption by concurrently optimizing its semilocal component and the long-range vdW correction against the prototypical adsorption CO/Pt(111) and Ar$_2$ binding energy curve. The resulting functional opens the door to accurate and efficient modeling of general molecular adsorption.

In this work, we address the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences and associated model capabilities (e.g., copyrighted data or harmful content generation) while preserving essential model utilities, without the need for retraining from scratch. Despite the growing need for LLM unlearning, a principled optimization framework remains lacking. To this end, we revisit the state-of-the-art approach, negative preference optimization (NPO), and identify the issue of reference model bias, which could undermine NPO's effectiveness, particularly when unlearning forget data of varying difficulty. Given that, we propose a simple yet effective unlearning optimization framework, called SimNPO, showing that 'simplicity' in removing the reliance on a reference model (through the lens of simple preference optimization) benefits unlearning. We also provide deeper insights into SimNPO's advantages, supported by analysis using mixtures of Markov chains. Furthermore, we present extensive experiments validating SimNPO's superiority over existing unlearning baselines in benchmarks like TOFU and MUSE, and robustness against relearning attacks. Codes are available at https://github.com/OPTML-Group/Unlearn-Simple.

Certifying entanglement for unknown quantum states experimentally is a fundamental problem in quantum computing and quantum physics. Because of being easy to implement, a most popular approach for this problem in modern quantum experiments is detecting target quantum states with fidelity-based entanglement witnesses. Specifically, if the fidelity between a target state and an entangled pure state exceeds a certain value, the target state can be guaranteed to be entangled. Recently, however, it has been realized that there exist so-called unfaithful quantum states, which can be entangled, but their entanglement cannot be certified by any fidelity-based entanglement witnesses. In this paper, by specific examples we show that if one makes a slight modification to fidelity-based entanglement witnesses by combining multiple fidelities together, it is still possible to certify entanglement for unfaithful quantum states with this popular technique. Particularly, we will analyze the mathematical structure of the modified entanglement witnesses, and propose an algorithm that can search for the optimal designs for them., Comment: 12 pages, 4 figures. Comments are welcome

This paper proposes the ProxFly, a residual deep Reinforcement Learning (RL)-based controller for close proximity quadcopter flight. Specifically, we design a residual module on top of a cascaded controller (denoted as basic controller) to generate high-level control commands, which compensate for external disturbances and thrust loss caused by downwash effects from other quadcopters. First, our method takes only the ego state and controllers' commands as inputs and does not rely on any communication between quadcopters, thereby reducing the bandwidth requirement. Through domain randomization, our method relaxes the requirement for accurate system identification and fine-tuned controller parameters, allowing it to adapt to changing system models. Meanwhile, our method not only reduces the proportion of unexplainable signals from the black box in control commands but also enables the RL training to skip the time-consuming exploration from scratch via guidance from the basic controller. We validate the effectiveness of the residual module in the simulation with different proximities. Moreover, we conduct the real close proximity flight test to compare ProxFly with the basic controller and an advanced model-based controller with complex aerodynamic compensation. Finally, we show that ProxFly can be used for challenging quadcopter in-air docking, where two quadcopters fly in extreme proximity, and strong airflow significantly disrupts flight. However, our method can stabilize the quadcopter in this case and accomplish docking. The resources are available at https://github.com/ruiqizhang99/ProxFly., Comment: 7 pages, 5 figures

Reinforcement Learning (RL) is a potent tool for sequential decision-making and has achieved performance surpassing human capabilities across many challenging real-world tasks. As the extension of RL in the multi-agent system domain, multi-agent RL (MARL) not only need to learn the control policy but also requires consideration regarding interactions with all other agents in the environment, mutual influences among different system components, and the distribution of computational resources. This augments the complexity of algorithmic design and poses higher requirements on computational resources. Simultaneously, simulators are crucial to obtain realistic data, which is the fundamentals of RL. In this paper, we first propose a series of metrics of simulators and summarize the features of existing benchmarks. Second, to ease comprehension, we recall the foundational knowledge and then synthesize the recently advanced studies of MARL-related autonomous driving and intelligent transportation systems. Specifically, we examine their environmental modeling, state representation, perception units, and algorithm design. Conclusively, we discuss open challenges as well as prospects and opportunities. We hope this paper can help the researchers integrate MARL technologies and trigger more insightful ideas toward the intelligent and autonomous driving., Comment: 23 pages, 6 figures and 2 tables. Submitted to IEEE Journal

Large Language Models (LLMs) often memorize sensitive, private, or copyrighted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the pre-trained model while preserving the model's utilities on other tasks. Several practical methods have recently been proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss of undesirable data. However, on certain unlearning tasks, these methods either fail to effectively unlearn the target data or suffer from catastrophic collapse -- a drastic degradation of the model's utilities. In this paper, we propose Negative Preference Optimization (NPO), a simple alignment-inspired method that could efficiently and effectively unlearn a target dataset. We theoretically show that the progression toward catastrophic collapse by minimizing the NPO loss is exponentially slower than GA. Through experiments on synthetic data and the benchmark TOFU dataset, we demonstrate that NPO-based methods achieve a better balance between unlearning the undesirable data and maintaining the model's utilities. We also observe that NPO-based methods generate more sensible outputs than GA-based methods, whose outputs are often gibberish. Remarkably, on TOFU, NPO-based methods are the first to achieve reasonable unlearning results in forgetting 50% (or more) of the training data, whereas existing methods already struggle with forgetting 10% of training data.

What can an agent learn in a stochastic Multi-Armed Bandit (MAB) problem from a dataset that contains just a single sample for each arm? Surprisingly, in this work, we demonstrate that even in such a data-starved setting it may still be possible to find a policy competitive with the optimal one. This paves the way to reliable decision-making in settings where critical decisions must be made by relying only on a handful of samples. Our analysis reveals that \emph{stochastic policies can be substantially better} than deterministic ones for offline decision-making. Focusing on offline multi-armed bandits, we design an algorithm called Trust Region of Uncertainty for Stochastic policy enhancemenT (TRUST) which is quite different from the predominant value-based lower confidence bound approach. Its design is enabled by localization laws, critical radii, and relative pessimism. We prove that its sample complexity is comparable to that of LCB on minimax problems while being substantially lower on problems with very few samples. Finally, we consider an application to offline reinforcement learning in the special case where the logging policies are known., Comment: 22 pages

We study the \emph{in-context learning} (ICL) ability of a \emph{Linear Transformer Block} (LTB) that combines a linear attention component and a linear multi-layer perceptron (MLP) component. For ICL of linear regression with a Gaussian prior and a \emph{non-zero mean}, we show that LTB can achieve nearly Bayes optimal ICL risk. In contrast, using only linear attention must incur an irreducible additive approximation error. Furthermore, we establish a correspondence between LTB and one-step gradient descent estimators with learnable initialization ($\mathsf{GD}\text{-}\mathbf{\beta}$), in the sense that every $\mathsf{GD}\text{-}\mathbf{\beta}$ estimator can be implemented by an LTB estimator and every optimal LTB estimator that minimizes the in-class ICL risk is effectively a $\mathsf{GD}\text{-}\mathbf{\beta}$ estimator. Finally, we show that $\mathsf{GD}\text{-}\mathbf{\beta}$ estimators can be efficiently optimized with gradient flow, despite a non-convex training objective. Our results reveal that LTB achieves ICL by implementing $\mathsf{GD}\text{-}\mathbf{\beta}$, and they highlight the role of MLP layers in reducing approximation error., Comment: 39 pages

Recent advancements in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet their reliance on extensive manual labeling to provide procedural feedback remains a significant impediment. To address this challenge, in this paper, we propose a novel self-supervised framework AutoPRM that efficiently enhances the fine-tuning of LLMs for intricate reasoning challenges. Specifically, AutoPRM first decomposes complex problems into more manageable subquestions with a controllable granularity switch, then sequentially apply reinforcement learning to iteratively improve the subquestion solver. Additionally, we propose context-guided-decoding to avoid reward tampering and guide the subquestion solver towards the solution of the holistic problem. Extensive experiments show that AutoPRM significantly improves performance on mathematical and commonsense reasoning tasks over SOTA. More encouragingly, AutoPRM can be easily integrated with other orthogonal reasoning pipelines., Comment: 17 pages, 4 figures, 11 tables

The widely-used Kohn-Sham implementation of density functional theory (DFT) maps a system of interacting electrons onto an auxiliary non-interacting one and is presumably inaccurate for strongly correlated materials. We present a concrete benchmark for DFT by examining the electronic ground state phase diagram of a strongly interacting chain with uneven, non-integer nuclear charges. The interplay between charge imbalance and Coulomb repulsion yields two competing phases, a band insulator and a Mott insulator. By including infinitesimal lattice distortions, DFT stabilizes the intermediate spontaneously dimerized insulator phase that results from this competition. We assess the phase diagram by mapping the bond length and nuclear charge ratio of the chain to the ionic Hubbard model and performing highly accurate density matrix renormalization group calculations. Our comparative study provides foundational insight into the utility of symmetry-broken DFT as a predictive tool for elucidating phase diagrams with competing orders.

The promotion of large-scale applications of reinforcement learning (RL) requires efficient training computation. While existing parallel RL frameworks encompass a variety of RL algorithms and parallelization techniques, the excessively burdensome communication frameworks hinder the attainment of the hardware's limit for final throughput and training effects on a single desktop. In this paper, we propose Spreeze, a lightweight parallel framework for RL that efficiently utilizes a single desktop hardware resource to approach the throughput limit. We asynchronously parallelize the experience sampling, network update, performance evaluation, and visualization operations, and employ multiple efficient data transmission techniques to transfer various types of data between processes. The framework can automatically adjust the parallelization hyperparameters based on the computing ability of the hardware device in order to perform efficient large-batch updates. Based on the characteristics of the "Actor-Critic" RL algorithm, our framework uses dual GPUs to independently update the network of actors and critics in order to further improve throughput. Simulation results show that our framework can achieve up to 15,000Hz experience sampling and 370,000Hz network update frame rate using only a personal desktop computer, which is an order of magnitude higher than other mainstream parallel RL frameworks, resulting in a 73% reduction of training time. Our work on fully utilizing the hardware resources of a single desktop computer is fundamental to enabling efficient large-scale distributed RL training., Comment: 11 pages, 8 figures, submitted to IEEE Transactions Journal

Flat bands coupled with magnetism and topological orders near or at the Fermi level are well known to drive exotic correlation physics and unconventional superconductivity. Here, based on first-principles modeling combined with an in-depth symmetry analysis, we reveal the presence of topological flat bands involving low-energy Ni-$3d_{z^2}$ states in the recently discovered superconductor LaNiO$_{2}$. Our analysis demonstrates that LaNiO$_2$ is an Axion insulator with $\mathbb{Z}_{4} = 2$ and that it supports topological crystalline insulating states protected by the glide mirror symmetries. The topological flat bands in LaNiO$_{2}$ are also shown to host odd-parity superconductivity. Our study indicates that the nickelates would provide an interesting materials platform for exploring the interplay of flat bands, topological states, and superconductivity., Comment: 8, 3 figures, 1 table

The recent development of the accurate and efficient semilocal density functionals on the third rung of Jacob's ladder of density functional theory such as the revised regularized strongly constrained and appropriately normed (r2SCAN) density functional could enable the rapid and highly reliable prediction of the elasticity and temperature dependence of thermophysical parameters of refractory elements and their intermetallic compounds using quasi-harmonic approximation (QHA). Here, we present a comparative evaluation of the equilibrium cell volumes, cohesive energy, mechanical moduli, and thermophysical properties (Debye temperature and thermal expansion coefficient) for 22 transition metals using semilocal density functionals, including local density approximation (LDA), the Perdew-Burke-Ernzerhof (PBE) and PBEsol generalized gradient approximations (GGA), and the r2SCAN meta-GGA. PBEsol and r2SCAN deliver the same level of accuracies for structural, mechanical and thermophysical properties. Otherwise, PBE and r2SCAN perform better than LDA and PBEsol for calculating cohesive energies of transition metals. Among the tested density functionals, r2SCAN provides an overall well-balanced performance for reliably computing the cell volumes, cohesive energies, mechanical properties, and thermophysical properties of various 3d, 4d, and 5d transition metals using QHA. Therefore, we recommend that r2SCAN could be employed as a workhorse method to evaluate the thermophysical properties of transition metal compounds and alloys in the high throughput workflows.

This paper proposes a technique for efficiently modeling dynamic humans by explicifying the implicit neural fields via a Neural Explicit Surface (NES). Implicit neural fields have advantages over traditional explicit representations in modeling dynamic 3D content from sparse observations and effectively representing complex geometries and appearances. Implicit neural fields defined in 3D space, however, are expensive to render due to the need for dense sampling during volumetric rendering. Moreover, their memory efficiency can be further optimized when modeling sparse 3D space. To overcome these issues, the paper proposes utilizing Neural Explicit Surface (NES) to explicitly represent implicit neural fields, facilitating memory and computational efficiency. To achieve this, the paper creates a fully differentiable conversion between the implicit neural fields and the explicit rendering interface of NES, leveraging the strengths of both implicit and explicit approaches. This conversion enables effective training of the hybrid representation using implicit methods and efficient rendering by integrating the explicit rendering interface with a newly proposed rasterization-based neural renderer that only incurs a texture color query once for the initial ray interaction with the explicit surface, resulting in improved inference efficiency. NES describes dynamic human geometries with pose-dependent neural implicit surface deformation fields and their dynamic neural textures both in 2D space, which is a more memory-efficient alternative to traditional 3D methods, reducing redundancy and computational load. The comprehensive experiments show that NES performs similarly to previous 3D approaches, with greatly improved rendering speed and reduced memory cost., Comment: 8 pages, accepted by ACMMM 2023

In some applications of reinforcement learning, a dataset of pre-collected experience is already available but it is also possible to acquire some additional online data to help improve the quality of the policy. However, it may be preferable to gather additional data with a single, non-reactive exploration policy and avoid the engineering costs associated with switching policies. In this paper we propose an algorithm with provable guarantees that can leverage an offline dataset to design a single non-reactive policy for exploration. We theoretically analyze the algorithm and measure the quality of the final policy as a function of the local coverage of the original dataset and the amount of additional data collected., Comment: 43 pages

Attention-based neural networks such as transformers have demonstrated a remarkable ability to exhibit in-context learning (ICL): Given a short prompt sequence of tokens from an unseen task, they can formulate relevant per-token and next-token predictions without any parameter updates. By embedding a sequence of labeled training data and unlabeled test data as a prompt, this allows for transformers to behave like supervised learning algorithms. Indeed, recent work has shown that when training transformer architectures over random instances of linear regression problems, these models' predictions mimic those of ordinary least squares. Towards understanding the mechanisms underlying this phenomenon, we investigate the dynamics of ICL in transformers with a single linear self-attention layer trained by gradient flow on linear regression tasks. We show that despite non-convexity, gradient flow with a suitable random initialization finds a global minimum of the objective function. At this global minimum, when given a test prompt of labeled examples from a new prediction task, the transformer achieves prediction error competitive with the best linear predictor over the test prompt distribution. We additionally characterize the robustness of the trained transformer to a variety of distribution shifts and show that although a number of shifts are tolerated, shifts in the covariate distribution of the prompts are not. Motivated by this, we consider a generalized ICL setting where the covariate distributions can vary across prompts. We show that although gradient flow succeeds at finding a global minimum in this setting, the trained transformer is still brittle under mild covariate shifts. We complement this finding with experiments on large, nonlinear transformer architectures which we show are more robust under covariate shifts., Comment: 50 pages, revised definition 3.2 and corollary 4.3

Evidence is growing that a second dome of high-$T_\mathrm{c}$ superconductivity can be accessed in the cuprates by increasing the doping beyond the first dome. Here we use \emph{ab initio} methods without invoking any free parameters, such as the Hubbard $U$, to reveal that pressure could turn YBa$_2$Cu$_3$O$_7$ into an ideal candidate for second-dome-superconductivity, displaying the predicted signature of strongly hybridized $d_{x^2-y^2}$ and $d_{z^2}$ orbitals. Notably, pressure is found to induce a phase transition replacing the antiferromagnetic phases with an orbitally-degenerate $d$--$d$ phase. Our study suggests that the origin of the second dome is correlated with the oxygen-hole fraction in the CuO$_2$ planes and the collapse of the pseudogap phase., Comment: 7 pages, 4 figures

Understanding many-electron phenomena with competing near-degenerate electronic states is of fundamental importance to chemistry and condensed matter physics. One of the most significant challenges for exploring such many-electron phenomena is the necessity for large system sizes in order to realize competing states, far beyond those practical for first-principles methods. Here, we show how allowing non-integer nuclear charges expands the space of computationally tractable electron systems that host competing electronic states. The emergence of competing electronic states from non-integer nuclear charges is exemplified in the simple 2-electron H_{2} molecule and used to examine the microscopic structure of doped quasi-1D cuprate chains, showing how non-integer nuclear charges can open a window for first-principles calculations of difficult many-electron phenomena., Comment: 20 pages, 3 figures

Recent advances in two-dimensional (2D) magnetism have heightened interest in layered magnetic materials due to their potential for spintronics. In particular, layered semiconducting antiferromagnets exhibit intriguing low-dimensional semiconducting behavior with both charge and spin as carrier controls. However, synthesis of these compounds is challenging and remains rare. Here, we conducted firstprinciples based high-throughput search to screen potentially stable mixed metal phosphorous trichalcogenides (MM'P2X6, where M and M' are transition metals and X is a chalcogenide) that have a wide range of tunable bandgaps and interesting magnetic properties. Among the potential candidates, we successfully synthesized a stable semiconducting layered magnetic material, CdFeP2Se6, that exhibits a short-range antiferromagnetic order at TN = 21 K with an indirect band gap of 2.23 eV. Our work suggests that highthroughput screening assisted synthesis be an effective method for layered magnetic materials discovery., Comment: 14 pages, 3 figures

The recently discovered infinite-layer nickelates show great promise in helping to disentangle the various cooperative mechanisms responsible for high-temperature superconductivity. However, lack of antiferromagnetic order in the pristine nickelates presents a challenge for connecting the physics of the cuprates and nickelates. Here, by using a quantum many-body Green's function-based approach to treat the electronic and magnetic structures, we unveil the presence of many two- and three-dimensional magnetic stripe instabilities that are shown to persist across the phase diagram of LaNiO$_2$. Our analysis indicates that the magnetic properties of the infinite-layer nickelates are closer to those of the doped cuprates which host inhomogeneous ground states rather than the undoped cuprates. The computed magnon spectrum in LaNiO$_2$ is found to contain an admixture of contributions from localized and itinerant carriers. The theoretically obtained magnon dispersion is in accord with the results of the corresponding RIXS experiments. Our study gives insight into the origin of inhomogeneity in the infinite-layer nickelates and their relationship with the cuprates., Comment: 11 pages, 4 figures

We propose Neural Deformable Fields (NDF), a new representation for dynamic human digitization from a multi-view video. Recent works proposed to represent a dynamic human body with shared canonical neural radiance fields which links to the observation space with deformation fields estimations. However, the learned canonical representation is static and the current design of the deformation fields is not able to represent large movements or detailed geometry changes. In this paper, we propose to learn a neural deformable field wrapped around a fitted parametric body model to represent the dynamic human. The NDF is spatially aligned by the underlying reference surface. A neural network is then learned to map pose to the dynamics of NDF. The proposed NDF representation can synthesize the digitized performer with novel views and novel poses with a detailed and reasonable dynamic appearance. Experiments show that our method significantly outperforms recent human synthesis methods., Comment: 16 pages, 7 figures. Accepted by ECCV 2022

Understanding similarities and differences between the cuprate and nickelate superconductors is drawing intense current interest. Competing charge orders have been observed recently in the $undoped$ infinite-layer nickelates in sharp contrast to the $undoped$ cuprates which exhibit robust antiferromagnetic insulating ground states. The microscopic mechanisms driving these differences remain unclear. Here, using in-depth first-principles and many-body theory based modeling, we show that the parent compound of the nickelate family, LaNiO$_2$, hosts a charge density wave (CDW) ground state with the predicted wavevectors in accord with the corresponding experimental findings. The CDW ground state is shown to be connected to a multi-orbital Peierls distortion. Our study points to the key role of electron-phonon coupling effects in the infinite-layer nickelates., Comment: 8 pages, 4 figures

Weyl electrons are intensely studied due to novel charge transport phenomena such as chiral anomaly, Fermi arcs, and photogalvanic effect. Recent theoretical works suggest that Weyl electrons can also participate in magnetic interactions, and the Weyl-mediated indirect exchange coupling between local moments is proposed as a new mechanism of spiral magnetism that involves chiral electrons. Despite reports of incommensurate and non-collinear magnetic ordering in Weyl semimetals, an actual spiral order has remained hitherto undetected. Here, we present evidence of Weyl-mediated spiral magnetism in SmAlSi from neutron diffraction, transport, and thermodynamic data. We show that the spiral order in SmAlSi results from the nesting between topologically non-trivial Fermi pockets and weak magnetocrystalline anisotropy, unlike related materials (Ce,Pr,Nd)AlSi, where a strong anisotropy prevents the spins from freely rotating. We map the magnetic phase diagram of SmAlSi and reveal an A-phase where topological magnetic excitations may exist. This is corroborated by the observation of a topological Hall effect within the A-phase., Comment: 7 pages, 4 figures

Off-Policy Evaluation (OPE) serves as one of the cornerstones in Reinforcement Learning (RL). Fitted Q Evaluation (FQE) with various function approximators, especially deep neural networks, has gained practical success. While statistical analysis has proved FQE to be minimax-optimal with tabular, linear and several nonparametric function families, its practical performance with more general function approximator is less theoretically understood. We focus on FQE with general differentiable function approximators, making our theory applicable to neural function approximations. We approach this problem using the Z-estimation theory and establish the following results: The FQE estimation error is asymptotically normal with explicit variance determined jointly by the tangent space of the function class at the ground truth, the reward structure, and the distribution shift due to off-policy learning; The finite-sample FQE error bound is dominated by the same variance term, and it can also be bounded by function class-dependent divergence, which measures how the off-policy distribution shift intertwines with the function approximator. In addition, we study bootstrapping FQE estimators for error distribution inference and estimating confidence intervals, accompanied by a Cramer-Rao lower bound that matches our upper bounds. The Z-estimation analysis provides a generalizable theoretical framework for studying off-policy estimation in RL and provides sharp statistical theory for FQE with differentiable function approximators., Comment: 39 pages

Policy gradient (PG) estimation becomes a challenge when we are not allowed to sample with the target policy but only have access to a dataset generated by some unknown behavior policy. Conventional methods for off-policy PG estimation often suffer from either significant bias or exponentially large variance. In this paper, we propose the double Fitted PG estimation (FPG) algorithm. FPG can work with an arbitrary policy parameterization, assuming access to a Bellman-complete value function class. In the case of linear value function approximation, we provide a tight finite-sample upper bound on policy gradient estimation error, that is governed by the amount of distribution mismatch measured in feature space. We also establish the asymptotic normality of FPG estimation error with a precise covariance characterization, which is further shown to be statistically optimal with a matching Cramer-Rao lower bound. Empirically, we evaluate the performance of FPG on both policy gradient estimation and policy optimization, using either softmax tabular or ReLU policy networks. Under various metrics, our results show that FPG significantly outperforms existing off-policy PG estimation methods based on importance sampling and variance reduction techniques.

Out-of-plane spin polarization {\sigma}_z has attracted increasing interests of researchers recently, due to its potential in high-density and low-power spintronic devices. Noncollinear antiferromagnet (AFM), which has unique 120{\deg} triangular spin configuration, has been discovered to possess {\sigma}_z. However, the physical origin of {\sigma}_z in noncollinear AFM is still not clear, and the external magnetic field-free switching of perpendicular magnetic layer using the corresponding {\sigma}_z has not been reported yet. Here, we use the cluster magnetic octupole in antiperovskite AFM Mn3SnN to demonstrate the generation of {\sigma}_z. {\sigma}_z is induced by the precession of carrier spins when currents flow through the cluster magnetic octupole, which also relies on the direction of the cluster magnetic octupole in conjunction with the applied current. With the aid of {\sigma}_z, current induced spin-orbit torque (SOT) switching of adjacent perpendicular ferromagnet is realized without external magnetic field. Our findings present a new perspective to the generation of out-of-plane spin polarizations via noncollinear AFM spin structure, and provide a potential path to realize ultrafast high-density applications., Comment: 23 pages, 4 figures, accepted by Nat. Commun