Search

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them. Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.

In this paper, we study format biases in reinforcement learning from human feedback (RLHF). We observe that many widely-used preference models, including human evaluators, GPT-4, and top-ranking models on the RewardBench benchmark, exhibit strong biases towards specific format patterns, such as lists, links, bold text, and emojis. Furthermore, large language models (LLMs) can exploit these biases to achieve higher rankings on popular benchmarks like AlpacaEval and LMSYS Chatbot Arena. One notable example of this is verbosity bias, where current preference models favor longer responses that appear more comprehensive, even when their quality is equal to or lower than shorter, competing responses. However, format biases beyond verbosity remain largely underexplored in the literature. In this work, we extend the study of biases in preference learning beyond the commonly recognized length bias, offering a comprehensive analysis of a wider range of format biases. Additionally, we show that with a small amount of biased data (less than 1%), we can inject significant bias into the reward model. Moreover, these format biases can also be easily exploited by downstream alignment algorithms, such as best-of-n sampling and online iterative DPO, as it is usually easier to manipulate the format than to improve the quality of responses. Our findings emphasize the need to disentangle format and content both for designing alignment algorithms and evaluating models., Comment: Working in progress

Emojis have become an integral part of digital communication, enriching text by conveying emotions, tone, and intent. Existing emoji recommendation methods are primarily evaluated based on their ability to match the exact emoji a user chooses in the original text. However, they ignore the essence of users' behavior on social media in that each text can correspond to multiple reasonable emojis. To better assess a model's ability to align with such real-world emoji usage, we propose a new semantics preserving evaluation framework for emoji recommendation, which measures a model's ability to recommend emojis that maintain the semantic consistency with the user's text. To evaluate how well a model preserves semantics, we assess whether the predicted affective state, demographic profile, and attitudinal stance of the user remain unchanged. If these attributes are preserved, we consider the recommended emojis to have maintained the original semantics. The advanced abilities of Large Language Models (LLMs) in understanding and generating nuanced, contextually relevant output make them well-suited for handling the complexities of semantics preserving emoji recommendation. To this end, we construct a comprehensive benchmark to systematically assess the performance of six proprietary and open-source LLMs using different prompting techniques on our task. Our experiments demonstrate that GPT-4o outperforms other LLMs, achieving a semantics preservation score of 79.23%. Additionally, we conduct case studies to analyze model biases in downstream classification tasks and evaluate the diversity of the recommended emojis.

Recent studies in text-to-image customization show great success in generating personalized object variants given several images of a subject. While existing methods focus more on preserving the identity of the subject, they often fall short of controlling the spatial relationship between objects. In this work, we introduce GroundingBooth, a framework that achieves zero-shot instance-level spatial grounding on both foreground subjects and background objects in the text-to-image customization task. Our proposed text-image grounding module and masked cross-attention layer allow us to generate personalized images with both accurate layout alignment and identity preservation while maintaining text-image coherence. With such layout control, our model inherently enables the customization of multiple subjects at once. Our model is evaluated on both layout-guided image synthesis and reference-based customization tasks, showing strong results compared to existing methods. Our work is the first work to achieve a joint grounding of both subject-driven foreground generation and text-driven background generation.

Recent studies have shown that large language models' (LLMs) mathematical problem-solving capabilities can be enhanced by integrating external tools, such as code interpreters, and employing multi-turn Chain-of-Thought (CoT) reasoning. While current methods focus on synthetic data generation and Supervised Fine-Tuning (SFT), this paper studies the complementary direct preference learning approach to further improve model performance. However, existing direct preference learning algorithms are originally designed for the single-turn chat task, and do not fully address the complexities of multi-turn reasoning and external tool integration required for tool-integrated mathematical reasoning tasks. To fill in this gap, we introduce a multi-turn direct preference learning framework, tailored for this context, that leverages feedback from code interpreters and optimizes trajectory-level preferences. This framework includes multi-turn DPO and multi-turn KTO as specific implementations. The effectiveness of our framework is validated through training of various language models using an augmented prompt set from the GSM8K and MATH datasets. Our results demonstrate substantial improvements: a supervised fine-tuned Gemma-1.1-it-7B model's performance increased from 77.5% to 83.9% on GSM8K and from 46.1% to 51.2% on MATH. Similarly, a Gemma-2-it-9B model improved from 84.1% to 86.3% on GSM8K and from 51.0% to 54.5% on MATH., Comment: A multi-turn direct preference learning framework for tool-integrated reasoning tasks

This perspective outlines several linear optical effects featured by molecular polaritons arising in the collective strong light-matter coupling regime, focusing on the limit when the number of molecules per photon mode is large. We show that, under these circumstances, molecular absorption within a cavity can be understood as the overlap between the polariton transmission and bare molecular absorption spectra, suggesting that polaritons act in part as optical filters. This framework demystifies and provides a straightforward explanation for a large class of theoretical models of polaritonic phenomena, highlighting that similar effects might be achievable outside a cavity with shaped laser pulses. With a few modifications, this simple conceptual picture can also be adapted to understand the incoherent nonlinear response of polaritonic systems. However, we note that there are experimental observations in the collective regime that exhibit phenomena that go beyond this treatment. Our analysis underscores the importance of the notion that the field still needs to establish a clear distinction between polaritonic phenomena that can be fully explained through classical optics and those that require a more advanced theoretical framework. The linear optics approach presented here is exact when the number of molecules tends to infinity and is quite accurate for a large, but finite, number of molecules. We highlight the limitations of this treatment when the rates of the single-molecule processes that facilitate dark-state-to-polariton relaxation cannot be neglected and in systems under strong coupling with few molecules. Further exploration in these areas is needed to uncover novel polaritonic phenomena.

Accurate text segmentation results are crucial for text-related generative tasks, such as text image generation, text editing, text removal, and text style transfer. Recently, some scene text segmentation methods have made significant progress in segmenting regular text. However, these methods perform poorly in scenarios containing artistic text. Therefore, this paper focuses on the more challenging task of artistic text segmentation and constructs a real artistic text segmentation dataset. One challenge of the task is that the local stroke shapes of artistic text are changeable with diversity and complexity. We propose a decoder with the layer-wise momentum query to prevent the model from ignoring stroke regions of special shapes. Another challenge is the complexity of the global topological structure. We further design a skeleton-assisted head to guide the model to focus on the global structure. Additionally, to enhance the generalization performance of the text segmentation model, we propose a strategy for training data synthesis, based on the large multi-modal model and the diffusion model. Experimental results show that our proposed method and synthetic dataset can significantly enhance the performance of artistic text segmentation and achieve state-of-the-art results on other public datasets., Comment: Accepted by ECCV 2024

The ability to isolate a quantum system from its environment is of fundamental interest and importance in optical quantum science and technology. Here we propose an experimentally feasible scheme for beating environment-induced dissipation in an open two-level system coupled to a parametrically driven cavity. The mechanism relies on a novel cooperation between light-matter coupling enhancement and frequent measurements. We demonstrate that, in the presence of the cooperation, the system dynamics can be completely dominated by the effective system-cavity interaction and the dissipative effects from the system-environment coupling can be surprisingly ignored. This work provides a generic method of dissipation suppression in a variety of quantum mechanical platforms, including natural atoms and superconducting circuits.

Quantum teleportation is a fundamental ingredient for quantum information science and technology. In particular, the ability to perform quantum teleportation between quantum systems of different natures and encoding types is crucial for building complex systems, such as distributed quantum internet. Here we propose a scheme to teleport a continuous variable optical qubit, encoded in an optical qumode by means of a superposed coherent state, onto a discrete variable solid-state qubit, associated with a single nitrogen-vacancy center spin in diamond, via a hybrid entanglement. By using a newly developed method for Bell-state measurement, which relies only on light homodyne detection and spin polarization measurement, near-deterministic and -perfect quantum teleportation can be achieved for large coherent-state amplitude input. Taking noise effects into account, we find that the average teleportation fidelity can still exceed the classical limit, enabling substantial teleportation distances under realistic experimental conditions.

Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically starts by training a reward model (RM) using human preference data. Conventional RMs are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, as humans cannot intuitively understand why an RM thinks a response is good or not. As RMs act as human preference proxies, we believe they should be human-interpretable to ensure that their internal decision processes are consistent with human preferences and to prevent reward hacking in LLM alignment. To build RMs with interpretable preferences, we propose a two-stage approach: i) train an Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional absolute-rating data, each dimension corresponding to a human-interpretable objective (e.g., honesty, verbosity, safety); ii) employ a Mixture-of-Experts (MoE) strategy with a gating network that automatically selects the most suitable reward objectives based on the context. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow MLP on top of the ArmoRM. Our trained model, ArmoRM-Llama3-8B, obtains state-of-the-art performance on RewardBench, a benchmark evaluating RMs for language modeling. Notably, the performance of our model surpasses the LLM-as-a-judge method with GPT-4 judges by a margin, and approaches the performance of the much larger Nemotron-4 340B reward model., Comment: Technical report v1. Code and model are released at https://github.com/RLHFlow/RLHF-Reward-Modeling/

We theoretically study diverse exceptional points (EPs) in an experimentally feasible magno-optomechanics consisting of an optomechanical subsystem coupled to a magnomechanical subsystem via physically direct contact. By adiabatically eliminating both the cavity and the Kittel mode, dissipative and parity-time symmetric exceptional points can be observed. When only the cavity mode is eliminated, a second (third) -order pseudo-Hermitian EP emerges for nondegenerate (degenerate) mechanical modes. The distinct dynamical behavior of two mechanical modes around these EPs are further studied. Our proposal provides a promising way to engineer diverse EPs and quantify non-Hermitian phase transition with exceptional dynamical behavior in magno-optomechanics., Comment: 6 pages,5 figures

Data-driven deep learning has emerged as the new paradigm to model complex physical space-time systems. These data-driven methods learn patterns by optimizing statistical metrics and tend to overlook the adherence to physical laws, unlike traditional model-driven numerical methods. Thus, they often generate predictions that are not physically realistic. On the other hand, by sampling a large amount of high quality predictions from a data-driven model, some predictions will be more physically plausible than the others and closer to what will happen in the future. Based on this observation, we propose \emph{Beam search by Vector Quantization} (BeamVQ) to enhance the physical alignment of data-driven space-time forecasting models. The key of BeamVQ is to train model on self-generated samples filtered with physics-aware metrics. To be flexibly support different backbone architectures, BeamVQ leverages a code bank to transform any encoder-decoder model to the continuous state space into discrete codes. Afterwards, it iteratively employs beam search to sample high-quality sequences, retains those with the highest physics-aware scores, and trains model on the new dataset. Comprehensive experiments show that BeamVQ not only gave an average statistical skill score boost for more than 32% for ten backbones on five datasets, but also significantly enhances physics-aware metrics.

We study the dynamical evolution of cold atoms in crossed optical dipole trap theoretically and experimentally. The atomic transport process is accompanied by two competitive kinds of physical mechanics, atomic loading and atomic loss. The loading process normally is negligible in the evaporative cooling experiment on the ground, while it is significant in the preparation of ultra-cold atoms in the space station. Normally, the atomic loading process is much weaker than the atomic loss process, and the atomic number in the center region of the trap decreases monotonically, as reported in previous research. However, when the atomic loading process is comparable to the atomic loss process, the atomic number in the center region of the trap will initially increase to a maximum value and then slowly decrease, and we have observed the phenomenon first. The increase of atomic number in the center region of the trap shows the presence of the loading process, and this will be significant especially under microgravity conditions. We build a theoretical model to analyze the competitive relationship, which coincides with the experimental results well. Furthermore, we have also given the predicted evolutionary behaviors under different conditions. This research provides a solid foundation for further understanding of the atomic transport process in traps. The analysis of loading process is of significant importance for the preparation of ultra-cold atoms in a crossed optical dipole trap under microgravity conditions.

We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature. However, existing open-source RLHF projects are still largely confined to the offline learning setting. In this technical report, we aim to fill in this gap and provide a detailed recipe that is easy to reproduce for online iterative RLHF. In particular, since online human feedback is usually infeasible for open-source communities with limited resources, we start by constructing preference models using a diverse set of open-source datasets and use the constructed proxy preference model to approximate human feedback. Then, we discuss the theoretical insights and algorithmic principles behind online iterative RLHF, followed by a detailed practical implementation. Our trained LLM, LLaMA-3-8B-SFR-Iterative-DPO-R, achieves impressive performance on LLM chatbot benchmarks, including AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning (SFT) and iterative RLHF can obtain state-of-the-art performance with fully open-source datasets. Further, we have made our models, curated datasets, and comprehensive step-by-step code guidebooks publicly available. Please refer to https://github.com/RLHFlow/RLHF-Reward-Modeling and https://github.com/RLHFlow/Online-RLHF for more detailed information.

Controlling microstructure in fusion-based metal additive manufacturing (AM) remains a challenge due to numerous parameters directly impacting solidification conditions. Multiprincipal element alloys (MPEAs) offer a vast compositional design space for microstructural engineering due to their chemical complexity and exceptional properties. Here, we establish a novel alloy design paradigm in MPEAs for AM using the FeMnCoCr system. By exploiting the decreasing phase stability with increasing Mn content, we achieve notable grain refinement and breakdown of columnar grain growth. We combine thermodynamic modeling, operando synchrotron X-ray diffraction, multiscale microstructural characterization, and mechanical testing to gain insight into the solidification physics and its ramifications on the resulting microstructure. This work paves way for tailoring grain sizes through targeted manipulation of phase stability, thereby advancing microstructure control in AM.

In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards -- a challenging scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of state-of-the-art closed-source large language models (LLMs), its open-source implementation is still largely sub-optimal, as widely reported by numerous research studies. To address these issues, we introduce a framework that models RLHF problems as a Markov decision process (MDP), enabling the capture of fine-grained token-wise information. Furthermore, we provide theoretical insights that demonstrate the superiority of our MDP framework over the previous sentence-level bandit formulation. Under this framework, we introduce an algorithm, dubbed as Reinforced Token Optimization (\texttt{RTO}), which learns the token-wise reward function from preference data and performs policy optimization based on this learned token-wise reward signal. Theoretically, \texttt{RTO} is proven to have the capability of finding the near-optimal policy sample-efficiently. For its practical implementation, \texttt{RTO} innovatively integrates Direct Preference Optimization (DPO) and PPO. DPO, originally derived from sparse sentence rewards, surprisingly provides us with a token-wise characterization of response quality, which is seamlessly incorporated into our subsequent PPO training stage. Extensive real-world alignment experiments verify the effectiveness of the proposed approach.

Cavity optomechanics, providing an inherently nonlinear interaction between photons and phonons, have shown enomerous potential in generating macroscopic quantum entanglement. Here we propose to realize diverse bipartite and tripartite entanglement in cavity-magnon optomechanics. By introducing magnons to standard cavity optomechanics, not only tunable optomechanical entanglement and magnon-magnon entanglement can be achieved, but also flexible tripartite entanglement including magnon-photon-phonon entanglement, magnon-magnon-photon and -phonon entanglement can be generated. Moreover, optimal bipartite and tripartite entanglement can be achieved by tuning parameters. We further show that all entanglement can be enhanced via engineering the magnon-photon coupling,and is proven to be robust against the bath temperature within the survival temperature. Besides, we find that the optomechanical entanglement can be protected or restored by bad magnons with large decay rate, while other entanglement is severely reduced. The results indicate that our proposal provides a novel avenue to explore and control tunable macroscopic quantum effects in hybrid cavity-magnon optomechanics., Comment: 10 pages, 8 figures

Effective editing of personal content holds a pivotal role in enabling individuals to express their creativity, weaving captivating narratives within their visual stories, and elevate the overall quality and impact of their visual content. Therefore, in this work, we introduce SwapAnything, a novel framework that can swap any objects in an image with personalized concepts given by the reference, while keeping the context unchanged. Compared with existing methods for personalized subject swapping, SwapAnything has three unique advantages: (1) precise control of arbitrary objects and parts rather than the main subject, (2) more faithful preservation of context pixels, (3) better adaptation of the personalized concept to the image. First, we propose targeted variable swapping to apply region control over latent feature maps and swap masked variables for faithful context preservation and initial semantic concept swapping. Then, we introduce appearance adaptation, to seamlessly adapt the semantic concept into the original image in terms of target location, shape, style, and content during the image generation process. Extensive results on both human and automatic evaluation demonstrate significant improvements of our approach over baseline methods on personalized swapping. Furthermore, SwapAnything shows its precise and faithful swapping abilities across single object, multiple objects, partial object, and cross-domain swapping tasks. SwapAnything also achieves great performance on text-based swapping and tasks beyond swapping such as object insertion., Comment: 18 pages, 16 figures, 3 tables

Generative object compositing emerges as a promising new avenue for compositional image editing. However, the requirement of object identity preservation poses a significant challenge, limiting practical usage of most existing methods. In response, this paper introduces IMPRINT, a novel diffusion-based generative model trained with a two-stage learning framework that decouples learning of identity preservation from that of compositing. The first stage is targeted for context-agnostic, identity-preserving pretraining of the object encoder, enabling the encoder to learn an embedding that is both view-invariant and conducive to enhanced detail preservation. The subsequent stage leverages this representation to learn seamless harmonization of the object composited to the background. In addition, IMPRINT incorporates a shape-guidance mechanism offering user-directed control over the compositing process. Extensive experiments demonstrate that IMPRINT significantly outperforms existing methods and various baselines on identity preservation and composition quality.

Multimodal Large Language Models (MLLMs) excel in generating responses based on visual inputs. However, they often suffer from a bias towards generating responses similar to their pretraining corpus, overshadowing the importance of visual information. We treat this bias as a "preference" for pretraining statistics, which hinders the model's grounding in visual input. To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself. Specifically, we propose the following two strategies: 1) using distorted image inputs to the MLLM for eliciting responses that contain signified pretraining bias; 2) leveraging text-based LLM to explicitly inject erroneous but common elements into the original response. Those undesirable responses are paired with original annotated responses from the datasets to construct the preference dataset, which is subsequently utilized to perform preference learning. Our approach effectively suppresses pretrained LLM bias, enabling enhanced grounding in visual inputs. Extensive experimentation demonstrates significant performance improvements across multiple benchmarks, advancing the state-of-the-art in multimodal conversational systems.

Superradiant phase transitions play a fundamental role in understanding the mechanism of collective light-matter interaction at the quantum level. Here we investigate multiple superradiant phases and phase transitions with different symmetry-breaking patterns in a two-mode V-type Dicke model. Interestingly, we show that there exists a quadruple point where one normal phase, one global symmetry-breaking superradiant phase and two local symmetry-breaking superradiant phases meet. Such a global phase results from the phase competition between two local superradiant phases and can not occur in the standard $\Lambda$- and $\Xi$-type three-level configurations in quantum optics. Moreover, we exhibit a sequential first-order quantum phase transition from one local to the global again to the other local superradiant phase. Our study opens up a perspective of exploring multi-level quantum critical phenomena with global symmetry breaking.

Exploring intermixing and interplay between different frequency-mixing processes has always been one of the interesting subjects at the interface of nonlinear optics with quantum optics. Here we investigate coherent competition and control between three-wave mixing (TWM) and four-wave mixing (FWM) in a cyclic three-level superconducting quantum system. In the weak control-field regime, strong competition leads to an alternating oscillation between TWM and FWM signals and this oscillation is a signature of strong energy exchange between these two nonlinear processes. In particular, such oscillation is absent from conventional multi-wave mixing in atomic systems. Surprisingly, synchronous TWM and FWM processes are demonstrated in the strong control-field regime and, at the same time, their efficiencies can be as high as 40% and 45%, respectively. Our study shows that these competitive behaviors between TWM and FWM can be manipulated by tuning the control-field intensity.

Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO)., Comment: The code and model are released at https://github.com/Haoxiang-Wang/directional-preference-alignment

Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks, facilitating the synthesis of visual content in an unconditional or input-conditional manner. The core idea behind them is learning to reverse the process of gradually adding noise to images, allowing them to generate high-quality samples from a complex distribution. In this survey, we provide an exhaustive overview of existing methods using diffusion models for image editing, covering both theoretical and practical aspects in the field. We delve into a thorough analysis and categorization of these works from multiple perspectives, including learning strategies, user-input conditions, and the array of specific editing tasks that can be accomplished. In addition, we pay special attention to image inpainting and outpainting, and explore both earlier traditional context-driven and current multimodal conditional methods, offering a comprehensive analysis of their methodologies. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval, featuring an innovative metric, LMM Score. Finally, we address current limitations and envision some potential directions for future research. The accompanying repository is released at https://github.com/SiatMMLab/Awesome-Diffusion-Model-Based-Image-Editing-Methods.

Phonon lasers, as mechanical analogues of optical lasers, are unique tools for not only fundamental studies of phononics but also diverse applications such as acoustic imaging and force sensing. Very recently, by levitating a micro-size sphere in an optical tweezer, higher-order mechanical harmonics were observed in the phonon-lasing regime, as the first step towards nonlinear levitated optomechanics [Nat. Phys. 19, 414 (2023)]. However, both the lasing strengths and the quality factors of the observed harmonics are typically very low, thus severely hindering their applications. Here we show that, by applying a simple but powerful electronic control to such a levitated micro-sphere, three orders of magnitude enhancement are achievable in the brightness of the phonon lasers, including both the fundamental mode and all its higher-order harmonics. Also, giant improvements of their linewidth and frequency stability are realized in such an electro-optomechanical system, together with further improved higher-order phonon coherence. These results, as a significant step forward for enhancing and controlling micro-object phonon lasers, can be readily used for a wide range of applications involving nonlinear phonon lasers, such as acoustic frequency comb, ultra-sound sensing, atmospherical monitoring, and even bio-medical diagnosis of levitated micro-size objects., Comment: 15 pages, 4 figures

We study Reinforcement Learning from Human Feedback (RLHF) under a general preference oracle. In particular, we do not assume that there exists a reward function and the preference signal is drawn from the Bradley-Terry model as most of the prior works do. We consider a standard mathematical formulation, the reverse-KL regularized minimax game between two LLMs for RLHF under general preference oracle. The learning objective of this formulation is to find a policy so that it is consistently preferred by the KL-regularized preference oracle over any competing LLMs. We show that this framework is strictly more general than the reward-based one, and propose sample-efficient algorithms for both the offline learning from a pre-collected preference dataset and online learning where we can query the preference oracle along the way of training. Empirical studies verify the effectiveness of the proposed framework., Comment: RLHF, Preference Learning, Alignment for LLMs

We propose an approach to produce spin squeezed states of a large number of nitrogen-vacancy centers in diamond nanostructures coupled to an optical cavity. Unlike the previous squeezing method proposed by Bennett et al. [Phys. Rev. Lett. 110, 156402 (2013)], which is limited by phonon number fluctuations due to the existence of phonon-spin entanglement, our proposal can completely erase the entanglement between spins and hybrid phonon-photon mode mediating the effective spin-spin interaction, and thus achieves unitary one-axis-twisting interactions between nitrogen-vacancy centres, yielding a squeezing scaling $J^{-2/3}$, where J is the total angular momentum. We found that, under certain conditions, our method has the potential to enhance the spin-spin nonlinear interactions. We also proposed a scheme utilizing repeatedly applying the one-axis-twisting evolution to two orthogonal spin directions, which enables the transformation of the one-axis-twisting interactions into two-axis-twisting type, and therefore leads to the spin squeezing with Heisenberg-limited scaling $J^{-1}$. Taking into account the noise effects of spin dephasing and relaxtion, we found that the proposed approaches are robust against imperfections.

At leading order in QCD coupling constant, we compute the energy loss per traveling distance of a heavy quark $dE/dz$ from elastic scattering off thermal quarks and gluons at a temperature $T$, including the thermal perturbative description of soft scatterings ($-t<-t^{\ast}$) and a perturbative QCD-based calculation for hard collisions ($-t>-t^{\ast}$). Within this soft-hard factorization model, we find that the full results of $dE/dz$ behaves a mild sensitivity to the intermediate cutoff $t^{\ast}$, supporting the validity of the soft-hard approach within the temperature region of interest. We re-derive the analytic formula for $dE/dz$ in the high-energy approximation, $E_{1}\gg m^{2}_{1}/T$, where $E_{1}$ is the injected heavy quark energy and $m_{1}$ is its mass. It is realized that the soft logarithmic contribution, $dE/dz\propto ln(-t^{\ast}/m^{2}_{D})$, arises from the $t$-channel scattering off thermal partons, while the hard logarithmic term, $dE/dz\propto ln[E_{1}T/(-t^{\ast})]$, stems from the $t$-channel scattering off thermal partons, and the one $dE/dz\propto ln(E_{1}T/m^{2}_{1})$ comes from the $s$- and $u$-channel scattering off gluons. The sum of these contributions cancels the $t^{\ast}$-dependence as observed in the full result. The mass hierarchy is observed $dE/dz(charm)>dE/dz(bottom)$. Our full results are crucial for a better description of heavy quark transport in QCD medium, in particular at low and moderate energy. We also calculate the energy loss by imposing the Einstein's relationship. The related results appear to be systematically larger than that without imposing the Einstein's relationship., Comment: 16 pages, 8 figures

Nonreciprocal devices, allowing to manipulate one-way signals, are crucial to quantum information processing and quantum network. Here we propose a nonlinear cavity-magnon system, consisting of a microwave cavity coupled to one or two yttrium-iron-garnet (YIG) spheres supporting magnons with Kerr nonlinearity, to investigate nonreciprocal unconventional photon blockade. The nonreciprocity originates from the direction-dependent Kerr effect, distinctly different from previous proposals with spinning cavities and dissipative couplings. For a single sphere case, nonreciprocal unconventional photon blockade can be realized by manipulating the nonreciprocal destructive interference between two active paths, via vary the Kerr coefficient from positive to negative, or vice versa. By optimizing the system parameters, the perfect and well-tuned nonreciprocal unconventional photon blockade can be predicted. For the case of two spheres with opposite Kerr effects, only reciprocal unconventional photon blockade can be observed when two cavity-magnon coupling strengths Kerr strengths are symmetric. However, when coupling strengths or Kerr strengths become asymmetric, nonreciprocal unconventional photon blockade appears. This implies that two-sphere nonlinear cavity-magnon systems can be used to switch the transition between reciprocal and nonreciprocal unconventional photon blockades. Our study offers a potential platform for investigating nonreciprocal photon blockade effect in nonlinear cavity magnonics., Comment: 9 pages,8 figures. Accepted by Advanced Quantum Technologies

The Einstein-Maxwell-scalar (EMS) theory with a quartic coupling function features three branches of fundamental black hole (BH) solutions, labeled as cold, hot, and bald black holes. The static bald black holes (the Reissner-Nordstr\"om BH) exhibit an intriguing nonlinear instability beyond the spontaneous scalarization. We study the rotating scalarized black hole solutions in the EMS model with a quartic coupling function through the spectral method numerically. The domain of existence for the scalarized BHs is presented in the spin-charge region. We found that the rotating solutions for both the two scalarized branches possess similar thermodynamic behavior compared to the static case while varying the electric charge. The BH spin enlarges the thermodynamic differences between the cold and hot branches. The profile of the metric function and the scalar field for the scalarized BHs is depicted, which demonstrates that the scalar field concentrates more on the equatorial plane in contrast to the axisymmetric region as the spin increases., Comment: 22 pages, 7 figures

This paper studies the alignment process of generative models with Reinforcement Learning from Human Feedback (RLHF). We first identify the primary challenges of existing popular methods like offline PPO and offline DPO as lacking in strategical exploration of the environment. Then, to understand the mathematical principle of RLHF, we consider a standard mathematical formulation, the reverse-KL regularized contextual bandit for RLHF. Despite its widespread practical application, a rigorous theoretical analysis of this formulation remains open. We investigate its behavior in three distinct settings -- offline, online, and hybrid -- and propose efficient algorithms with finite-sample theoretical guarantees. Moving towards practical applications, our framework, with a robust approximation of the information-theoretical policy improvement oracle, naturally gives rise to several novel RLHF algorithms. This includes an iterative version of the Direct Preference Optimization (DPO) algorithm for online settings, and a multi-step rejection sampling strategy for offline scenarios. Our empirical evaluations on real-world alignment experiment of large language model demonstrate that these proposed methods significantly surpass existing strong baselines, such as DPO and Rejection Sampling Optimization (RSO), showcasing the connections between solid theoretical foundations and their potent practical implementations., Comment: 53 pages; theoretical study and algorithmic design of iterative RLHF and DPO