Search

Your search keyword '"Zhu, David"' showing total 1,758 results
Start Over Author "Zhu, David"
1,758 results on '"Zhu, David"'

1. Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation

Author

Chen, Yifang and Zhu, David

Subjects
Computer Science - Computation and Language, Computer Science - Artificial Intelligence
Abstract

Recent advances in large language model (LLM) training have highlighted the need for diverse, high-quality instruction data. Recently, many works are exploring synthetic data generation using LLMs. However, they primarily focus on prompt engineering with standard supervised instruction-finetuned models, which contains a fundamental limitation: these models are optimized for general question-answering/problem-solving rather than data generation. We propose a paradigm shift named \textbf{NOMAD} by investigating how to specifically train models for data generation, demonstrating that this task differs significantly from training a classical LM. We identify two key factors: no-prompt-masked training and proper training set size selection. Our method, NOMAD, shows substantial improvements over baselines, achieving >4\% gains in TriviaQA and >2\% in GSM8K with limited training data. Finally, we offer new insights by interpreting synthetic data through the lenses of "relevance" and "novelty".

Published
2024

2. Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning

Author

Xu, Yinglun, Zhu, David, Gumaste, Rohan, and Singh, Gagandeep

Subjects
Computer Science - Machine Learning
Abstract

Offline reinforcement learning has become one of the most practical RL settings. However, most existing works on offline RL focus on the standard setting with scalar reward feedback. It remains unknown how to universally transfer the existing rich understanding of offline RL from the reward-based to the preference-based setting. In this work, we propose a general framework to bridge this gap. Our key insight is transforming preference feedback to scalar rewards via binary reward labeling (BRL), and then any reward-based offline RL algorithms can be applied to the dataset with the reward labels. The information loss during the feedback signal transition is minimized with binary reward labeling in the practical learning scenarios. We theoretically show the connection between several recent PBRL techniques and our framework combined with specific offline RL algorithms. By combining reward labeling with different algorithms, our framework can lead to new and potentially more efficient offline PBRL algorithms. We empirically test our framework on preference datasets based on the standard D4RL benchmark. When combined with a variety of efficient reward-based offline RL algorithms, the learning result achieved under our framework is comparable to training the same algorithm on the dataset with actual rewards in many cases and better than the recent PBRL baselines in most cases.

Published
2024

3. Two-Step Offline Preference-Based Reinforcement Learning with Constrained Actions

Author

Xu, Yinglun, Suresh, Tarun, Gumaste, Rohan, Zhu, David, Li, Ruirui, Wang, Zhengyang, Jiang, Haoming, Tang, Xianfeng, Yin, Qingyu, Cheng, Monica Xiao, Zeng, Qi, Zhang, Chao, and Singh, Gagandeep

Subjects
Computer Science - Machine Learning, Computer Science - Artificial Intelligence
Abstract

Preference-based reinforcement learning (PBRL) in the offline setting has succeeded greatly in industrial applications such as chatbots. A two-step learning framework where one applies a reinforcement learning step after a reward modeling step has been widely adopted for the problem. However, such a method faces challenges from the risk of reward hacking and the complexity of reinforcement learning. To overcome the challenge, our insight is that both challenges come from the state-actions not supported in the dataset. Such state-actions are unreliable and increase the complexity of the reinforcement learning problem at the second step. Based on the insight, we develop a novel two-step learning method called PRC: preference-based reinforcement learning with constrained actions. The high-level idea is to limit the reinforcement learning agent to optimize over a constrained action space that excludes the out-of-distribution state-actions. We empirically verify that our method has high learning efficiency on various datasets in robotic control environments.

Published
2023

11. Transport of Wipes in Sewers

Author

Peng, Xinzai, Zhu, David Z., Zhang, Wenming, di Prisco, Marco, Series Editor, Chen, Sheng-Hong, Series Editor, Vayas, Ioannis, Series Editor, Kumar Shukla, Sanjay, Series Editor, Sharma, Anuj, Series Editor, Kumar, Nagesh, Series Editor, Wang, Chien Ming, Series Editor, Cui, Zhen-Dong, Series Editor, Lu, Xinzheng, Series Editor, Desjardins, Serge, editor, Poitras, Gérard J., editor, and Ng, Kelvin Tsun Wai, editor

Published
2024
Full Text
View/download PDF

Catalog

Books, media, physical & digital resources
See catalog results