Ensemble Distilling Pretrained Language Models for Machine Translation Quality Estimation

Machine translation quality estimation (Quality Estimation, QE) aims to evaluate the quality of machine translation automatically without golden reference. QE can be implemented on different granularities, thus to give an estimation for different aspects of machines translation output. In this paper, we propose an effective method to utilize pretrained language models to improve the performance of QE. Our model combines two popular pretrained models, which are Bert and XLM, to create a very strong baseline for both sentence-level and word-level QE. We also propose a simple yet effective strategy, ensemble distillation, to further improve the accuracy of QE system. Ensemble distillation can integrate different knowledge from multiple models into one model, and strengthen each single model by a large margin. We evaluate our system on CCMT2019 Chinese-English and English-Chinese QE dataset, which contains word-level and sentence-level subtasks. Experiment results show our model surpasses previous models to a large extend, demonstrating the effectiveness of our proposed method.