Back to Search
Start Over
SCITUNE: Aligning Large Language Models with Scientific Multimodal Instructions
- Publication Year :
- 2023
-
Abstract
- Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving the LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present SciTune as a tuning framework to improve the ability of LLMs to follow scientific multimodal instructions. To test our methodology, we use a human-generated scientific instruction tuning dataset and train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. In comparison to the models that are finetuned with machine generated data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark.<br />Comment: Preprint. Work in progress
Details
- Database :
- arXiv
- Publication Type :
- Report
- Accession number :
- edsarx.2307.01139
- Document Type :
- Working Paper