1. DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging
- Author
-
Lin, Tzu-Han, Li, Chen-An, Lee, Hung-yi, and Chen, Yun-Nung
- Subjects
Computer Science - Computation and Language - Abstract
Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the \textbf{Do}main knowled\textbf{ge} merged \textbf{R}eward \textbf{M}odel (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment., Comment: Preprint. Code will be released after the review results
- Published
- 2024