1. CNIMA: A Universal Evaluation Framework and Automated Approach for Assessing Second Language Dialogues
- Author
-
Gao, Rena, Wu, Jingxuan, Roever, Carsten, Wu, Xuetong, Wu, Jing, Lv, Long, and Lau, Jey Han
- Subjects
Computer Science - Computation and Language - Abstract
We develop CNIMA (Chinese Non-Native Interactivity Measurement and Automation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We annotate CNIMA using an evaluation framework -- originally introduced for English-as-a-second-language dialogues -- that assesses micro-level features (e.g.\ backchannels) and macro-level interactivity labels (e.g.\ topic management) and test the framework's transferability from English to Chinese. We found the framework robust across languages and revealed universal and language-specific relationships between micro-level and macro-level features. Next, we propose an approach to automate the evaluation and find strong performance, creating a new tool for automated second language assessment. Our system can be adapted to other languages easily as it uses large language models and as such does not require large-scale annotated training data.
- Published
- 2024