Search

Many research questions in the field of applied linguistics are answered by manually analyzing data collections or corpora: collections of spoken, written and/or visual communicative messages. In this kind of quantitative content analysis, the coding of subjective language data often leads to disagreement among raters. In this paper, we discuss causes of and solutions to disagreement problems in the analysis of discourse. We discuss crucial factors determining the quality and outcome of corpus analyses, and focus on the sometimes tense relation between reliability and validity. We evaluate formal assessments of intercoder reliability. We suggest a number of ways to improve the intercoder reliability, such as the precise specification of the variables and their coding categories and carving up the coding process into smaller substeps. The paper ends with a reflection on challenges for future work in discourse analysis, with special attention to big data and multimodal discourse.

Chatbots worden steeds vaker ingezet in de klantenservice, maar zijn verre van foutloos. Wanneer chatbots fouten maken, zijn er verschillende reparatiestrategieën om het onbegrip te communiceren. Dit artikel geeft een overzicht van de literatuur over dit onderwerp, en presenteert twee experimentele studies waaruit blijkt dat chatbots onbegrip beter met een tegemoetkomende reparatiestrategie kunnen communiceren dan met een defensieve strategie.

How can chatbots undo their mistakes? Repair strategies for misunderstandings in a chatbot conversation This paper presents a taxonomy of chatbot repair strategies identif ied in the f ields of communication science and dialogue systems. Subsequently, two experimental studies investigated the effects of miscommunication in chatbot conversations. In both experiments, participants engaged f irst in a successful and then a (manipulated) unsuccessful chatbot conversation. The unsuccessful chatbot responded to the misunderstanding with either defensive or accommodative repair strategies. The f indings of both studies reveal the negative impact of an unsuccessful chatbot conversation on brand attitude compared to a successful chatbot conversation. However, if the chatbot uses an accommodative repair strategy, brand attitude is rated higher than if the chatbot uses a defensive repair strategy. Also, study 2 showed that the chatbot’s communicated expertise (being an expert vs. in training) does not impact this relation. Thus, the way in which misunderstanding is communicated by the chatbot is more important than the expertise exhibited by the chatbot.

This thesis evaluates the effectiveness of various topic tracking and trend identification methods in detecting trending topics in open-ended healthcare policy questionnaire responses. Identifying these trending topics is crucial for providing decision-makers with insights into public concerns and important policy areas. Two primary model types are used in this study. First, topic tracking models, including Length Weighted Topic Chains (LWTC) and Single Pass Clustering (SPC), are utilized to identify topics and generate a timeseries of their occurrences. Second, trend identification models leverage these timeseries to detect trending topics through methods such as Most Occurring Topics Selection, Moving Average (MA), and AutoRegressive Integrated Moving Average (ARIMA), or by analyzing word frequency using Bursty Term Extraction (BTE). The evaluation of topic tracking methods is conducted via the word intrusion task, while trend identification models are assessed using precision and F1-scores across two labeled timelines. The findings indicate that SPC beats LWTC in topic modeling but lacks generalizability, whereas LWTC demonstrates consistent performance. For trend identification, the Moving Average method emerges as the most effective, achieving the highest precision and F1-score combination, followed by ARIMA, Most Occurring Topics Selection, and BTE. A notable result is that the combination of LWTC with the Moving Average method yields the best overall performance for identifying trending topics in open-ended healthcare policy questionnaire responses. This combination achieves both a high precision and F1-scores, making it the most robust approach in this context. However, the thesis also reveals a disconnect between topic modeling quality and trend detection effectiveness, suggesting that higher interpretability does not necessarily translate to superior trend identification. This thesis highlights several limitations, including the subjectivity in topic label

In the Netherlands, government letters are sometimes written with such intricate language that the public might find them difficult to understand, resulting in a need for simplification of these letters. Because of the growth in Natural Language Processing (NLP) tasks with the transformer architecture, this study explores the possibility of lowering the difficulty of the letters from Rijksdienst voor Ondernemend Nederland (RVO) with transformers without changing the semantic meaning of the letters. To this end, this research has performed several steps, including finding the characteristics that contributed to the difficulty by using the tool LiNT, conducting a small exploratory research to identify the best approach for text simplification and performing an analysis of the original and the simplified letters. This research unveiled that the characteristic “unknown words” was the main contributor to the difficulty of the letters. For more challenging letters, the characteristics “length of clauses” and “grammatical dependencies” also affected the difficulty. From the exploratory research, it followed that Fietje-2 outperformed other models with the approach “few-shot prompting”. After performing this approach on all letters from RVO, the analysis of the original and simplified letters showed that a majority of the letters simplified slightly. The model simplified the letters by decreasing scores for the characteristics “unknown words” and “length of clauses”. These adjustments were mostly attributed to the examples given in the few-shot prompting. However, this was also the weakness of the model, as it was responsible for the minor increase in difficulty as well for a small portion of the letters. Thus, future research can focus on curating more appropriate examples for this approach.

Text simplification aims to make text more readable by reducing the linguistic complexity. This study explores the use of sequence-to-sequence transformer models to simplify Dutch governmental letters, enhancing their readability for individuals with low literacy levels. Various models, including T5-Small, BART-Base, mT5-Small, mBART-Large-50, T5-Base-Dutch, UL2-Small-Dutch and UL2-Small-Dutch-Simplification, were trained on datasets comprising complex and simplified Dutch sentences. These models were evaluated using quantitative metrics such as the Flesch-Kincaid Grade Level, BLEU score and SARI score, complemented by a qualitative analysis. The best-performing model was applied to a dataset of letters provided by the Rijksdienst voor Ondernemend Nederland to produce simplified versions. The study demonstrates that while the models slightly improve readability as indicated by Flesch-Kincaid scores, qualitative analysis reveals significant issues with content preservation and coherence. This highlights the need for further refinement to achieve the desired readability improvement, while maintaining accuracy in Dutch text simplification.

In conversation genres like instruction, clarification questions asked by a user may either relate to the task at hand or to common-sense knowledge about the task domain, whereas most conversational agents focus on only one of these types. To learn more about the best approach and feasibility of integrating both types of questions, we experimented with different approaches for modelling and distinguishing between task-specific and common sense questions in the context of a cooking assistant. We subsequently integrated the best ones in a conversational agent, which we tested in a study with six users cooking a recipe. Even though the three elements functioned well on their own and all participants completed the recipe, question-answering accuracy was relatively low (66%). We conclude with a discussion of the aspects that need to be improved upon to cope with the diverse information need in task-based conversational agents.

We developed a system to detect positive expectation, disappointment, and satisfaction in tweets that refer to events automatically discovered in the Twitter stream. The emotional content shared on Twitter when referring to public events can provide insights into the presumed and experienced quality of the event. We expected to find a connection between positive expectation and disappointment, a succession that is referred to as anticipointment. The application of computational approaches makes it possible to detect the presence and strength of this hypothetical relation for a large number of events. We extracted events from a longitudinal dataset of Dutch Twitter posts, and modeled classifiers to detect emotion in the tweets related to those events by means of hashtag-labeled training data. After classifying all tweets before and after the events in our dataset, we summarized the collective emotions for over 3000 events as the percentage of tweets classified as positive expectation (in anticipation), disappointment and satisfaction (in hindsight). Only a weak correlation of around 0.2 was found between positive expectation and disappointment, while a higher correlation of 0.6 was found between positive expectation and satisfaction. The most anticipointing events were events with a clear loss, such as a canceled event or when the favored sports team had lost. We conclude that senders of Twitter posts might be more inclined to share satisfaction than disappointment after a much anticipated event.

We developed a machine-learning-based method to detect video game players that harass teammates or opponents in chat earlier in the conversation. This real-time technology would allow gaming companies to intervene during games, such as issue warnings or muting or banning a player. In a proof-of-concept experiment on League of Legends data we compute and visualize evaluation metrics for a machine learning classifier as conversations unfold, and observe that the optimal precision and recall of detecting toxic players at each moment in the conversation depends on the confidence threshold of the classifier: the threshold should start low, and increase as the conversation unfolds. How fast this sliding threshold should increase depends on the training set size.

We investigate what distinguishes reported dreams from other personal narratives. The continuity hypothesis, stemming from psychological dream analysis work, states that most dreams refer to a person’s daily life and personal concerns, similar to other personal narratives such as diary entries. Differences between the two texts may reveal the linguistic markers of dream text, which could be the basis for new dream analysis work and for the automatic detection of dream descriptions. We used three text analytics methods: text classification, topic modeling, and text coherence analysis, and applied these methods to a balanced set of texts representing dreams, diary entries, and other personal stories. We observed that dream texts could be distinguished from other personal narratives nearly perfectly, mostly based on the presence of uncertainty markers and descriptions of scenes. Important markers for non-dream narratives are specific time expressions. Dream texts also exhibit a lower discourse coherence than other personal narratives.