Evaluating information presentation strategies for spoken dialogue systems

A common task for spoken dialogue systems (SDS) is to help users select a suitable option (e.g., flight, hotel, restaurant) from the set of options available. When the number of options is small, they can simply be presented sequentially. However, as the number of options increases, the system must have strategies for helping users browse the space of available options. In this thesis, I compare two approaches to information presentation in SDS: (1) the summarize and refine (SR) approach (Polifroni et al., 2003; Polifroni, 2008) in which the summaries are generated by clustering the options based on attributes that lead to the smallest number of clusters, and (2) the user-model based summarize and refine (UMSR) approach (Demberg, 2005; Demberg and Moore, 2006) which employs a user model to cluster options based on attributes that are relevant to the user and uses coherence markers (e.g., connectives, discourse cues, adverbials) to highlight the trade-offs among the presented items. Prior work has shown that users prefer approaches to information presentation that take the user’s preferences into account (e.g., Komatani et al., 2003;Walker et al., 2004; Demberg and Moore, 2006). However, due to the complexity of building a working end-to-end SDS, these studies employed an ”overhearer” evaluation methodology, in which participants read or listened to pre-prepared dialogues, thus limiting evaluation criteria to users’ perceptions (e.g., informativeness, overview of options, and so on). In order to examine whether users prefer presentations based on UMSR when they were actively interacting with a dialogue system, and to measure the effectiveness and efficiency of the two approaches, I compared them in a Wizard-of-Oz experiment. I found that in terms of both task success and dialogue efficiency the UMSR approach was superior to the SR approach. In addition, I found that users also preferred presentations based on UMSR in the interactive mode. SDS are typically developed for situations in which the user’s hands and eyes are busy. I hypothesized that the benefits of pointing out relationships among options (i.e., trade-offs) in information presentation messages outweighs the costs of processing more complex sentences. To test this hypothesis, I performed two dual task experiments comparing the two approaches to information presentation in terms of their effect on cognitive load. Again, participants performed better with presentations based on the UMSR algorithm in terms of both dialogue efficiency and task success, and I found no detrimental effect on performance of the primary task. Finally, I hypothesized that one of the main reasons why UMSR is more efficient is because it uses coherence markers to highlight relations (e.g., trade-offs) between options and attributes. To test this hypothesis, I performed an eye-tracking experiment in which participants read presentations with and without these linguistic devices, and answered evaluation and comparison questions to measure differences in item recall. In addition, I used reading times to examine comprehension differences between the two information presentation strategies. I found that the linguistic devices used in UMSR indeed facilitated item recall, with no penalty in terms of comprehension cost. Thus, in this thesis I showed that an approach to information presentation that employs a user model and uses linguistic devices such as coherence markers to highlight trade-offs among the presented items improves information browsing. User studies demonstrated that this finding also applies to situations where users are performing another demanding task simultaneously.