Search

We argue that dialect identification should be treated as a multi-label classification problem rather than the single-class setting prevalent in existing collections and evaluations. In order to avoid extensive human re-labelling of the data, we propose an analysis of ambiguous near-duplicates in an existing collection covering four variants of French. We show how this analysis helps us provide multiple labels for a significant subset of the original data, therefore enriching the annotation with minimal human intervention. The resulting data can then be used to train dialect identifiers in a multi-label setting. Experimental results show that on the enriched dataset, the multi-label classifier produces similar accuracy to the single-label classifier on test cases that are unambiguous (single label), but it increases the macro-averaged F1- score by 0.225 absolute (71% relative gain) on ambiguous texts with multiple labels. On the original data, gains on the ambiguous test cases are smaller but still considerable (+0.077 absolute, 20% relative gain), and accuracy on nonambiguous test cases is again similar in this case. This supports our thesis that modelling dialect identification as a multi-label problem potentially has a positive impact., Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), May 5-6, 2023, Dubrovnik, Croatia

We describe the systems developed by the National Research Council Canada for the French Cross-Domain Dialect Identification shared task at the 2022 VarDial evaluation campaign. We evaluated two different approaches to this task: SVM and probabilistic classifiers exploiting n-grams as features, and trained from scratch on the data provided; and a pre-trained French language model, CamemBERT, that we fine-tuned on the dialect identification task. The latter method turned out to improve the macro-F1 score on the test set from 0.344 to 0.430 (25% increase), which indicates that transfer learning can be helpful for dialect identification., Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, October 2022, Gyeongju, Republic of Korea

While recent studies have been dedicated to cleaning very noisy parallel corpora to improve Machine Translation training, we focus in this work on filtering a large and mostly clean Translation Memory. This problem of practical interest has not received much consideration from the community, in contrast with, for example, filtering large web-mined parallel corpora. We experiment with an extensive, multi-domain proprietary Translation Memory and compare five approaches involving deep-, feature-, and heuristic-based solutions. We propose two ways of evaluating this task, manual annotation and resulting Machine Translation quality. We report significant gains over a state-of-the-art, off-the-shelf cleaning system, using two MT engines., 15th biennial conference of the Association for Machine Translation in the Americas (AMTM), September 12-16, 2022, Orlando, USA

We describe the systems developed by the National Research Council Canada for the Uralic language identification shared task at the 2021 VarDial evaluation campaign. We evaluated two different approaches to this task: a probabilistic classifier exploiting only character 5-grams as features, and a character-based neural network pre-trained through self-supervision, then fine-tuned on the language identification task. The former method turned out to perform better, which casts doubt on the usefulness of deep learning methods for language identification, where they have yet to convincingly and consistently outperform simpler and less costly classification algorithms exploiting n-gram features., 8th VarDial Workshop on NLP for Similar Languages, Varieties and Dialects, April 20th, 2021, Held Virtually

Messages posted on social media such as Twitter and Instagram are a rich and promising source of information on real-life events. However, due to the high volume and the noisy nature of posts on social media, the messages reporting events are usually overwhelmed by unrelated daily chatter. To detect unspecified events, many topic modeling and wavelet signal processing methods have been proposed. In this paper, we propose an improved method, BCCED, using a burstiness index and co-occurrence clustering for event detection, that builds on the Event Detection with Clustering of Wavelet-based Signals (EDCoW) method of Weng et al. [2011]. We compare their performance with two topic modeling methods on two social media datasets. Experiments show that BCCED outperforms these alternatives for unspecified event detection from social media., Presented at the 7th International Workshop on Natural Language Processing for Social Media (SocialNLP @ IJCAI 2019), Aug. 12th, 2019, Macau, China

We introduce the challenging task of detecting changes from an online conversation. Our goal is to detect significant changes in, for example, sentiment or topic in a stream of messages that are part of an ongoing conversation. Our approach relies on first applying linguistic preprocessing or collecting simple statistics on the messages in the conversation in order to build a time series. Change point detection algorithms are then applied to identify the location of significant changes in the distribution of the underlying time series. We present a collection of sport events on which we can evaluate the performance of our change detection method. Our experiments, using several change point detection algorithms and several types of time series, show that it is possible to detect salient changes in an on-line conversation with relatively high accuracy., 11th International Conference on Language Resources and Evaluation, LREC 2018, May 7-12, 2018, Miyazaki, Japan

Detecting changes within an unfolding event in real time from news articles or social media enables to react promptly to serious issues in public safety, public health or natural disasters. In this study, we use on-line Latent Dirichlet Allocation (LDA) to model shifts in topics, and apply on-line change point detection (CPD) algorithms to detect when significant changes happen. We describe an on-line Bayesian change point detection algorithm that we use to detect topic changes from on-line LDA output. Extensive experiments on social media data and news articles show the benefits of on-line LDA versus standard LDA, and of on-line change point detection compared to off-line algorithms. This yields F-scores up to 56% on the detection of significant real-life changes from these document streams., The 27th International Conference on Computational Linguistics, August 20-26, 2018, Santa Fe, New Mexico, United States

Roughly a decade ago appeared the Additive Factor Model (AFM), a cognitive diagnostic model that was subsequently implemented by PSLC-Datashop and successfully used by researchers since then. While powerful, this model is not always simple to apprehend for a novice user. This paper aims at addressing this concern, by sharing our understanding of the model and the way we implemented it while conducting research on Q-matrices evaluation. Situating AFM as a member of the multidimensional Item Response Theory models, this paper provides readers with several considerations regarding the behavior of the model and the meaning of its parameters, as well as details on the technical implementation. Finally a use case is presented showing how the model can be used to monitor, understand and improve learning.

The goal of keyphrase extraction is to automatically identify the most salient phrases from documents. The technique has a wide range of applications such as rendering a quick glimpse of a document, or extracting key content for further use. While previous work often assumes keyphrases are a static property of a given documents, in many applications, the appropriate set of keyphrases that should be extracted depends on the set of documents that are being considered together. In particular, good keyphrases should not only accurately describe the content of a document, but also reveal what discriminates it from the other documents. In this paper, we study this problem of extracting discriminative keyphrases. In particularly, we propose to use the hierarchical semantic structure between candidate keyphrases to promote keyphrases that have the right level of specificity to clearly distinguish the target document from others. We show that such knowledge can be used to construct better discriminative keyphrase extraction systems that do not assume a static, fixed set of keyphrases for a document. We show how this helps identify key expertise of authors from their papers, as well as competencies covered by online courses within different domains., The 26th International Conference on Computational Linguistics, December 11-17, 2016, Osaka, Japan

We describe the system built by the National Research Council (NRC) Canada for the 2015 shared task on Discriminating between similar languages. The NRC system uses various statistical classifiers trained on character and word ngram features. Predictions rely on a two-stage process: we first predict the language group, then discriminate between languages or variants within the group. This year, we focused on two issues: 1) the ngram generation process, and 2) the handling of the anonymized (“blinded”) Named Entities. Despite the slightly harder experimental conditions this year, our systems achieved an average accuracy of 95.24% (closed task) and 95.65% (open task), ending up second or (close) third on the closed task, and first on the open task., LT4VarDial - Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, September 10th, 2015, Hissar, Bulgaria

This paper reports on findings resulting from our research conducted for the 2014 Business Process Intelligence (BPI) Challenge. The goal of this year's challenge was to identify predictive patterns in the data, provided by Rabobank's ICT operations unit, that could be used to determine whether service calls and incidents were likely to increase or decrease as a result of any change implemented in their systems. We took two approaches: a macro approach where conclusions were drawn via analysis of aggregated data from the entire set, and a micro approach where sequence classification was used to uncover predictive patterns in the data for individual configuration items. Each approach addresses the challenge questions from different levels, and offers insight on the findings from each point of view that can be applied to improve company business processes and decision making in the future., Fourth International Business Process Intelligence Challenge (BPIC’14), September 7th-11th 2014, Haifa, Israel