1. SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
- Author
-
Wadden, David, Shi, Kejian, Morrison, Jacob, Naik, Aakanksha, Singh, Shruti, Barzilay, Nitzan, Lo, Kyle, Hope, Tom, Soldaini, Luca, Shen, Shannon Zejiang, Downey, Doug, Hajishirzi, Hannaneh, and Cohan, Arman
- Subjects
Computer Science - Computation and Language ,Computer Science - Artificial Intelligence - Abstract
We present SciRIFF (Scientific Resource for Instruction-Following and Finetuning), a dataset of 137K instruction-following demonstrations for 54 tasks covering five essential scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification. SciRIFF demonstrations are notable for their long input contexts, detailed task specifications, and complex structured outputs. While instruction-following resources are available in specific domains such as clinical medicine and chemistry, SciRIFF is the first dataset focused on extracting and synthesizing information from research literature across a wide range of scientific fields. To demonstrate the utility of SciRIFF, we develop a sample-efficient strategy to adapt a general instruction-following model for science by performing additional finetuning on a mix of general-domain and SciRIFF demonstrations. In evaluations on nine held-out scientific tasks, our model -- called SciTulu -- improves over a strong LLM baseline by 28.1% and 6.5% at the 7B and 70B scales respectively, while maintaining general instruction-following performance within 2% of the baseline. We are optimistic that SciRIFF will facilitate the development and evaluation of LLMs to help researchers navigate the ever-growing body of scientific literature. We release our dataset, model checkpoints, and data processing and evaluation code to enable further research., Comment: Submitted to NeurIPS Datasets and Benchmarks 2024
- Published
- 2024