1. TeleChat Technical Report
- Author
-
He, Zhongjiang, Wang, Zihan, Liu, Xinzhang, Liu, Shixuan, Yao, Yitong, Huang, Yuyao, Li, Xuelong, Li, Yongxiang, Che, Zhonghao, Zhang, Zhaoxi, Wang, Yan, Wang, Xin, Pu, Luwen, Xu, Huinan, Fang, Ruiyu, Zhao, Yu, Zhang, Jie, Huang, Xiaomeng, Lu, Zhilong, Peng, Jiaxin, Zheng, Wenjun, Wang, Shiquan, Yang, Bingkai, he, Xuewei, Jiang, Zhuoru, Xie, Qiyi, Zhang, Yanhan, Li, Zhongqiu, Shi, Lingling, Fu, Weiwei, Zhang, Yin, Huang, Zilu, Xiong, Sishi, Zhang, Yuxiang, Wang, Chao, and Song, Shuangyong
- Subjects
Computer Science - Computation and Language ,Computer Science - Artificial Intelligence ,I.2.7 - Abstract
In this technical report, we present TeleChat, a collection of large language models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It includes pretrained language models as well as fine-tuned chat models that is aligned with human preferences. TeleChat is initially pretrained on an extensive corpus containing a diverse collection of texts from both English and Chinese languages, including trillions of tokens. Subsequently, the model undergoes fine-tuning to align with human preferences, following a detailed methodology that we describe. We evaluate the performance of TeleChat on various tasks, including language understanding, mathematics, reasoning, code generation, and knowledge-based question answering. Our findings indicate that TeleChat achieves comparable performance to other open-source models of similar size across a wide range of public benchmarks. To support future research and applications utilizing LLMs, we release the fine-tuned model checkpoints of TeleChat's 7B and 12B variant, along with code and a portion of our pretraining data, to the public community., Comment: 28 pages, 2 figures
- Published
- 2024