Search

Reinforcement Learning from Human Feedback (RLHF) aligns language models to human preferences by employing a singular reward model derived from preference data. However, such an approach overlooks the rich diversity of human preferences inherent in data collected from multiple users. In this work, we first derive an impossibility result of alignment with single reward RLHF, thereby highlighting its insufficiency in representing diverse human preferences. To provide an equitable solution to the problem, we learn a mixture of preference distributions via an expectation-maximization algorithm and propose a MaxMin alignment objective for policy learning inspired by the Egalitarian principle in social choice theory to better represent diverse human preferences. We elucidate the connection of our proposed approach to distributionally robust optimization and general utility RL, thereby highlighting the generality and robustness of our proposed solution. We present comprehensive experimental results on small-scale (GPT-2) and large-scale language models (with Tulu2-7B) and show the efficacy of the proposed approach in the presence of diversity among human preferences. Our algorithm achieves an average improvement of more than 16% in win-rates over conventional RLHF algorithms and improves the win-rate (accuracy) for minority groups by over 33% without compromising the performance of majority groups, showcasing the robustness and fairness of our approach. We remark that our findings in this work are not only limited to language models but also extend to reinforcement learning in general.

While modern biotechnologies allow synthesizing new proteins and function measurements at scale, efficiently exploring a protein sequence space and engineering it remains a daunting task due to the vast sequence space of any given protein. Protein engineering is typically conducted through an iterative process of adding mutations to the wild-type or lead sequences, recombination of mutations, and running new rounds of screening. To enhance the efficiency of such a process, we propose a tree search-based bandit learning method, which expands a tree starting from the initial sequence with the guidance of a bandit machine learning model. Under simplified assumptions and a Gaussian Process prior, we provide theoretical analysis and a Bayesian regret bound, demonstrating that the combination of local search and bandit learning method can efficiently discover a near-optimal design. The full algorithm is compatible with a suite of randomized tree search heuristics, machine learning models, pre-trained embeddings, and bandit techniques. We test various instances of the algorithm across benchmark protein datasets using simulated screens. Experiment results demonstrate that the algorithm is both sample-efficient and able to find top designs using reasonably small mutation counts., Comment: AAAI 2024

We show that for a minimal system $(X,T)$, the set of saturated points along cubes with respect to its maximal $\infty$-step pro-nilfactor $X_\infty$ has a full measure. As an application, it is shown that if a minimal system $(X,T)$ has no non-trivial $(k+1)$-tuples with arbitrarily long finite IP-independence sets, then it has only at most $k$ ergodic measures and is an almost $k'$ to one extension of $X_\infty$ for some $k'\leqslant k$. Particularly, for $k=1$ we prove that $(X,T)$ is uniquely ergodic (even regular with respect to $X_\infty$), which answers a conjecture stated in [3]., Comment: arXiv admin note: text overlap with arXiv:2202.08782; text overlap with arXiv:1105.3584, arXiv:1007.0189 by other authors

While transferring a pretrained language model, common approaches conventionally attach their task-specific classifiers to the top layer and adapt all the pretrained layers. We investigate whether one could make a task-specific selection on which subset of the layers to adapt and where to place the classifier. The goal is to reduce the computation cost of transfer learning methods (e.g. fine-tuning or adapter-tuning) without sacrificing its performance. We propose to select layers based on the variability of their hidden states given a task-specific corpus. We say a layer is already "well-specialized" in a task if the within-class variability of its hidden states is low relative to the between-class variability. Our variability metric is cheap to compute and doesn't need any training or hyperparameter tuning. It is robust to data imbalance and data scarcity. Extensive experiments on the GLUE benchmark demonstrate that selecting layers based on our metric can yield significantly stronger performance than using the same number of top layers and often match the performance of fine-tuning or adapter-tuning the entire language model., Comment: EMNLP 2022 camera-ready

Inspired by the recent work of Glasner, Huang, Shao, Weiss and Ye, we prove that the maximal $\infty$-step pro-nilfactor $X_\infty$ of a minimal system $(X,T)$ is the topological characteristic factor along polynomials in a certain sense. Namely, we show that by an almost one to one modification of $\pi:X\to X_\infty$, the induced open extension $\pi^*:X^*\to X_\infty^*$ has the following property: for any $d\in \mathbb{N}$, any open subsets $V_0,V_1,\ldots,V_d$ of $X^*$ with $\bigcap_{i=0}^d \pi^*(V_i)\neq \emptyset$ and any distinct non-constant integer polynomials $p_i$ with $p_i(0)=0$ for $i=1,\ldots,d$, there exists some $n\in \mathbb{Z}$ such that $V_0\cap T^{-p_1(n)}V_1\cap \ldots \cap T^{-p_d(n)}V_d \neq \emptyset$. where an integer polynomial is the polynomial with rational coefficients taking integer values on the integers. As an application, the following result is obtained: for a totally minimal system $(X,T)$ and integer polynomials $p_1,\ldots,p_d$, if every non-trivial integer combination of $p_1,\ldots,p_d$ is not constant, then there is a dense $G_\delta$ subset $\Omega$ of $ X$ such that the set \[ \{(T^{p_1(n)}x,\ldots, T^{p_d(n)}x):n\in \mathbb{Z}\} \] is dense in $X^d$ for every $x\in \Omega$., Comment: arXiv admin note: text overlap with arXiv:2006.12385, arXiv:1407.1179 by other authors

In this paper, it is shown that for a minimal system $(X,T)$ and $d,k\in \mathbb{N}$, if $(x,x_i)$ is regionally proximal of order $d$ for $1\leq i\leq k$, then $(x,x_1,\ldots,x_k)$ is $(k+1)$-regionally proximal of order $d$. Meanwhile, we introduce the notion of $\mathrm{IN}^{[d]}$-pair: for a dynamical system $(X,T)$ and $d\in \mathbb{N}$, a pair $(x_0,x_1)\in X\times X$ is called an $\mathrm{IN}^{[d]}$-pair if for any $k\in \mathbb{N}$ and any neighborhoods $U_0 ,U_1 $ of $x_0$ and $x_1$ respectively, there exist integers $p_j^{(i)},1\leq i\leq k,$ $1\leq j\leq d$ such that $$ \bigcup_{i=1}^k\{ p_1^{(i)}\epsilon(1)+\ldots+p_d^{(i)} \epsilon(d):\epsilon(j)\in \{0,1\},1\leq j\leq d\}\backslash \{0\}\subset \mathrm{Ind}(U_0,U_1), $$ where $\mathrm{Ind}(U_0,U_1)$ denotes the collection of all independence sets for $(U_0,U_1)$. It turns out that for a minimal system, if it dose not contain any nontrivial $\mathrm{IN}^{[d]}$-pair, then it is an almost one-to-one extension of its maximal factor of order $d$., Comment: arXiv admin note: text overlap with arXiv:1911.05435. text overlap with arXiv:1105.3584, arXiv:1007.0189 by other authors

For a topological dynamical system $(X, T)$, $l\in\mathbb{N}$ and $x\in X$, let $N_l(X)$ and $L_x^l(X)$ be the orbit closures of the diagonal point $(x,x,\ldots,x)$ ($l $ times) under the actions $\mathcal{G}_{l}$ and $\tau_l $ respectively, where $\mathcal{G}_{l}$ is generated by $T\times T\times \ldots \times T$ ($l $ times) and $\tau_l=T\times T^2\times \ldots \times T^l$. In this paper, we show that for a minimal system $(X,T)$ and $l\in \mathbb{N}$, the maximal $d$-step pro-nilfactor of $(N_l(X),\mathcal{G}_{l})$ is $(N_l(X_d),\mathcal{G}_{l})$, where $\pi_d:X\to X/\mathbf{RP}^{[d]}=X_d,d\in \mathbb{N}$ is the factor map and $\mathbf{RP}^{[d]}$ is the regionally proximal relation of order $d$. Meanwhile, when $(X,T)$ is a minimal nilsystem, we also calculate the pro-nilfactors of $(L_x^l(X),\tau_l)$ for almost every $x$ w.r.t. the Haar measure. In particular, there exists a minimal $2$-step nilsystem $(Y,T)$ and a countable set $\Omega\subset Y$ such that for $y\in Y\backslash \Omega$ the maximal equicontinuous factor of $(L_y^2(Y),\tau_2)$ is not $(L_{\pi_1(y)}^2(Y_{1}),\tau_2)$.

In this paper, it is shown that for $d\in\mathbb{N}$, a minimal system $(X,T)$ is a $d$-step pro-nilsystem if its enveloping semigroup is a $d$-step top-nilpotent group, answering an open question by Donoso. Thus, combining the previous result of Donoso, it turns out that a minimal system $(X,T)$ is a $d$-step pro-nilsystem if and only if its enveloping semigroup is a $d$-step top-nilpotent group.

In this paper, it is shown that if a dynamical system is null and distal, then it is equicontinuous. It turns out that a null system with closed proximal relation is mean equicontinuous. As a direct application, it follows that a null dynamical system with dense minimal points is also mean equicontinuous. Meanwhile, a distal system with trivial $\text{Ind}_{fip}$-pairs, and a non-trivial regionally proximal relation of order $\infty$ is constructed., Comment: arXiv admin note: text overlap with arXiv:1312.7663 by other authors

In this note, it is shown that several results concerning mean equicontinuity proved before for minimal systems are actually held for general topological dynamical systems. Particularly, it turns out that a dynamical system is mean equicontinuous if and only if it is equicontinuous in the mean if and only if it is Banach (or Weyl) mean equicontinuous if and only if its regionally proximal relation is equal to the Banach proximal relation. Meanwhile, a relation is introduced such that the smallest closed invariant equivalence relation containing this relation induces the maximal mean equicontinuous factor for any system., Comment: arXiv admin note: substantial text overlap with arXiv:1312.7663 by other authors