Search

The recent surge of interest in explainability in artificial intelligence (XAI) is propelled by not only technological advancements in machine learning but also by regulatory initiatives to foster transparency in algorithmic decision making. In this article, we revise the current concept of explainability and identify three limitations: passive explainee, narrow view on the social process, and undifferentiated assessment of explainee’s understanding. In order to overcome these limitations, we present explanation as a social practice in which explainer and explainee co-construct understanding on the microlevel. We view the co-construction on a microlevel as embedded into a macrolevel, yielding expectations concerning, e.g., social roles or partner models: typically, the role of the explainer is to provide an explanation and to adapt it to the current level of explainee’s understanding; the explainee, in turn, is expected to provide cues that direct the explainer. Building on explanations being a social practice, we present a conceptual framework that aims to guide future research in XAI. The framework relies on the key concepts of monitoring and scaffolding to capture the development of interaction. We relate our conceptual framework and our new perspective on explaining to transparency and autonomy as objectives considered for XAI.

Speech Synthesis Evaluation : State-of-the-Art Assessment and Suggestion for a Novel Research Program

This paper presents an approach to voice conversion, whichdoes neither require parallel data nor speaker or phone labels fortraining. It can convert between speakers which are not in thetraining set by employing the previously proposed concept of afactorized hierarchical variational autoencoder. Here, linguisticand speaker induced variations are separated upon the notionthat content induced variations change at a much shorter timescale, i.e., at the segment level, than speaker induced variations,which vary at the longer utterance level. In this contribution wepropose to employ convolutional instead of recurrent networklayers in the encoder and decoder blocks, which is shown toachieve better phone recognition accuracy on the latent segmentvariables at frame-level due to their better temporal resolution.For voice conversion the mean of the utterance variables is re-placed with the respective estimated mean of the target speaker.The resulting log-mel spectra of the decoder output are used aslocal conditions of a WaveNet which is utilized for synthesis ofthe speech waveforms. Experiments show both good disentan-glement properties of the latent space variables, and good voiceconversion performance.

Human verbal communication is a complex phenomenon involving dynamics that normally result in the alignment of participants on several modalities, and across various linguistic domains. We examined here whether such dynamics occur also for paralinguistic events, in particular, in the case of laughter. Using a conversational corpus containing dyadic interactions in three languages (French, German and Mandarin Chinese), we investigated three measures of alignment: convergence, synchrony and agreement. Support for convergence and synchrony was found in all three languages, although the level of support varied with the language, while the agreement in laughter type was found to be significant for the German data. The implications of these findings towards a better understanding of the role of laughter in human communication are discussed.

Synthetic speech can be used to express uncertainty in dialogue systems by means of hesitation. If a phrase like “Next to the green tree” is uttered in a hesitant way, that is, containing lengthening, silences, and fillers, the listener can infer that the speaker is not certain about the concepts referred to. However, we do not know anything about the referential domain of the uncertainty; if only a particular word in this sentence would be uttered hesitantly, e.g. “the greee:n tree”, the listener could infer that the uncertainty refers to the color in the statement, but not to the object. In this study, we show that the domain of the uncertainty is controllable. We conducted an experiment in which color words in sentences like “search for the green tree” were lengthened in two different positions: word onsets or final consonants, and participants were asked to rate the uncertainty regarding color and object. The results show that initial lengthening is predominantly associated with uncertainty about the word itself, whereas final lengthening is primarily associated with the following object. These findings enable dialogue system developers to finely control the attitudinal display of uncertainty, adding nuances beyond the lexical content to message delivery. QC 20191014

In this paper, we explore the possibility to gather perceptual impressions of prosodic prominence by exploiting the strong prosody-gesture link, i.e., by having listeners transform a perceptual impression into a motor movement, namely drumming, for two domains of prominence: word-level and syllable-level. A feasibility study reveals that such a procedure is indeed easily and speedily mastered by naive listeners, but more difficult for word-level prominences. We furthermore examine whether drummed annotations are comparable to those gathered with more established annotation protocols based on cumulative naive impressions and fine-grained expert ratings. These comparisons reveal high correspondences across all prominence annotation protocols, thus corroborating the general usefulness of the gestural approach. The analyses also reveal that all annotation protocols are strongly driven by structural linguistic considerations. We then use Random Forest Models to investigate the relative impact of signal and structural cues to prominence annotations. We find that expert ratings of prosodic prominence are guided comparatively more by structural concerns than those of naive annotators, that word-level annotations are influenced more by structural linguistic cues than syllable-level ones, and that "drummed" annotations are driven least by structural cues. Lastly, we isolate two main listener strategies among our group of "drummers", namely those integrating structural and signal cues to prominence, and those being guided predominantly by signal cues. (C) 2019 Elsevier Ltd. All rights reserved.

Conversational spoken dialogue systems that interact with the user rather than merely reading the text can be equipped with hesitations to manage dialogue flow and user attention. Based on a series of empirical studies, we elaborated a hesitation synthesis strategy for dialogue systems, which inserts hesitations of a scalable extent wherever needed in the ongoing utterance. Previously, evaluations of hesitation systems have shown that synthesis quality is affected negatively by hesitations, but that they result in improvements of interaction quality. We argue that due to its conversational nature, hesitation synthesis needs interactive evaluation rather than traditional mean opinion score (MOS)-based questionnaires. To validate this claim, we dually evaluate our system’s speech synthesis component, on the one hand, linked to the dialogue system evaluation, and on the other hand, in a traditional MOS way. We are thus able to analyze and discuss differences that arise due to the evaluation methodology. Our results suggest that MOS scales are not sufficient to assess speech synthesis quality, leading to implications for future research that are discussed in this paper. Furthermore, our results indicate that synthetic hesitations are able to increase task performance and that an elaborated hesitation strategy is necessary to avoid likability issues.

In this paper we present the implementation of a robot, that dynamically hesitates, based on the attention of the human interaction partner. To this end, we outline requirements for a real-time interaction scenario, describe the realization of a disfluency insertion strategy, and present observations from the first tests of the system.

Conversational spoken dialogue systems that interact with the user rather than merely reading text can be equipped with hesitations to manage the dialogue flow and the users' attention. Based on a series of empirical studies, we built an elaborated hesitation synthesis strategy for dialogue systems that inserts hesitations of scalable extent wherever needed in the ongoing utterance. So far, evaluations of hesitating systems have shown that synthesis quality is affected negatively by hesitations, but that there is improvement in interaction quality. We argue that due to its conversational nature, hesitation synthesis needs interactive evaluation rather than traditional MOS-based questionnaires. To prove this point, we dually evaluate our system’s speech synthesis component: on the one hand, linked to the dialogue system evaluation, on the other hand, in the traditional MOS way. This way we are able to analyze and discuss differences that arise due to the evaluation methodology. Our results suggest that MOS scales are not sufficient to assess speech synthesis quality, which has implications for future research that are discussed in this paper. Furthermore, our results indicate that hesitations work well to increase task performance and that an elaborated strategy is necessary to avoid likability issues.

Many studies emphasize the importance of infant-directed speech: stronger articulated, higher-quality speech helps infants to better distinguish different speech sounds. This effect has been widely investigated in terms of the infant's perceptual capabilities, but few studies examined whether infant-directed speech has an effect on articulatory learning. In earlier studies, we developed a model that learns articulatory control for a 3D vocal tract model via goal babbling. Exploration is organized in the space of outcomes. This so called goal space is generated from a set of ambient speech sounds. Similarly to how speech from the environment shapes infant's speech perception, the data from which the goal space is learned shapes the later learning process: it determines which sounds the model is able to discriminate, and thus, which sounds it can eventually learn to produce. We investigate how speech sound quality in early learning affects the model's capability to learn new vowel sounds. The model is trained either on hyperarticulated (tense) or on hypoarticulated (lax) vowels. Then we retrain the model with vowels from the other set. Results show that new vowels can be acquired although they were not included in early learning. There is, however, an effect of learning order, showing that models first trained on the stronger articulated tense vowels easier accommodate to new vowel sounds later on.

This paper explores the applications of fuzzy logic inference systems as an instrument to perform linguistic analysis in the domain of prosodic prominence. Understanding how acoustic features interact to make a linguistic unit be perceived as more relevant than the surrounding ones is generally needed to study the cognitive processes needed for speech understanding. It also has technological applications in the field of speech recognition and synthesis. We present a first experiment to show how fuzzy inference systems, being characterised by their capability to provide detailed insight about the models obtained through supervised learning can help investigate the complex relationships among acoustic features linked to prominence perception.

Most prominence annotation methods have certain drawbacks. Simple binary scales may be too coarse to capture fine-grained prominence differences, and multi-level annotation schemes have been shown to be time-consuming and difficult to use for non-expert annotators. This study proposes a novel method for fine-grained and fast prominence annotation by exploiting the prosody-gesture link. On a sentence-by-sentence basis, native German participants were instructed to listen to audio recordings and reiterate them by beating on an electronic drum pad either once per syllable (experiment 1) or once per word (experiment 2), modulating the strength of each beat according to how strongly the syllable or word stood out in the sentence. The velocity profiles of MIDI outputs were then interpreted as correlates of perceived prominence and compared with fine-grained prominence ratings by three expert annotators. While wordlevel drumming showed high correlations to conventional ratings for some of the subjects, inexperienced participants often had considerable difficulty performing the task. Syllable-level drumming, on the other hand, proved to be a time-efficient and intuitive method for experienced and naive subjects alike. Especially by pooling velocity results from several participants to create mean values, it was possible to maintain high levels of correlation with expert prominence ratings.

In recent years there has been a substantial debate about the need for increasingly spontaneous, conversational corpora of spoken interaction that are not controlled or task directed. In parallel the need has arisen for the recording of multi-modal corpora which are not restricted to the audio domain alone. With a corpus that would fulfill both needs, it would be possible to investigate the natural coupling, not only in turn-taking and voice, but also in the movement of participants. In the following paper we describe the design and recording of such a corpus and we provide some illustrative examples of how such a corpus might be exploited in the study of dynamic interaction. The D64 corpus is a multimodal corpus recorded over two successive days. Each day resulted in approximately 4 h of recordings. In total five participants took part in the recordings of whom two participants were female and three were male. Seven video cameras were used of which at least one was trained on each participant. The Optitrack motion capture kit was used in order to enrich information. The D64 corpus comprises annotations on conversational involvement, speech activity and pauses as well as information of the average degree of change in the movement of participants.

For intervention programs that are applied in natural settings, randomization often is difficult or impossible to achieve. If treated individuals are compared with individuals from a nonrandomized comparison group, the inference of causality can be biased. Similar distributions in the relevant characteristics of the treatment and the comparison groups cannot be expected. To adjust between-group comparisons for preexisting differences, this article proposes a simple matching procedure. This procedure involves pairing of treatment and comparison individuals based on observable characteristics, using Euclidean distance scores. Application of the proposed Euclidean-distance matching (EuM) procedure to data from the Viennese E-Lecturing (VEL) project yields satisfying results. Possible generalizations and applications of the EuM procedure are discussed.

The Active Listening Corpus (ALICO) is a multimodal data set of spontaneous dyadic conversations in German with diverse speech and gestural annotations of both dialogue partners. The annotations consist of short feedback expression transcriptions with corresponding communicative function interpretations as well as segmentations of interpausal units, words, rhythmic prominence intervals and vowel-to-vowel intervals. Additionally, ALICO contains head gesture annotations of both interlocutors. The corpus contributes to research on spontaneous human–human interaction, on functional relations between modalities, and timing variability in dialogue. It also provides data that differentiates between distracted and attentive listeners. We describe the main characteristics of the corpus and briefly present the most important results obtained from analyses in recent years. QC 20160711

We investigate Polysyllabic shortening effects in three prosodic domains, the word, the inter-stress interval (ISI) and the narrow rhythm unit (NRU), in a large corpus of English broadcast speech. Results confirm and extend earlier findings, indicating that these effects are interpretable as artifacts of word-final lengthening. We do, however, find effects compatible with the assumption of eurhythmic principles in speech production.

This stud y was ai med at analyzing empirical duration data for Polish spoken at different tempos using an updated version of the Coupled Oscillator Model of speech timing and rhythm variability (O'Dell and Nieminen, 1999, 2009). We use Bayesian inference on parameters relating to speech rate to investigate how tempo affects timing in Polish. The model parameters found are then compared with parameters obtained for equivalent material in Finnish to shed light on which of the effects represent general speech rate mechanisms and which are specific to Polish. We discuss the model and its predictions in the context of current perspectives on speech timing.

This article gives a survey of a blended learning approach called Vienna E-Lecturing (VEL), implemented in the course Research Methods and Evaluation, which is required by the psychology program at the University of Vienna, Austria. VEL replaces a main lecture and has been designed to teach methodological issues more effectively as well as to strengthen students' learning competences in this field. The program's conceptualization is based on instructional and motivational findings yielding the program's two main teaching principles: (1) networking and (2) optimal instructions. The Internet-based course lasts two semesters and is composed of 10 online learning modules and 11 face-to-face meetings (including tutorials). The modules, which are available successively via a learning platform, systematically instruct students to learn more effectively by cooperating and fulfilling different tasks within small groups. The current article describes the program's principles and theoretical background and outlines the 10 online modules. In addition, some module examples are given for illustration.

An optimization-based model of suprasegmental speech timing is presented.Timing patterns are modeled by trading off economy and clarity-related demands.Global and local weights allow for simulating different prosodic conditions.Simulation experiments demonstrate replication of various timing phenomena. We present a model of suprasegmental speech timing based on the assumption that speech patterns are shaped by global and local adjustments of trade-offs between conflicting demands of minimizing production effort and maximizing perceptual clarity. The model uses an optimization procedure to determine durations of suprasegmental constituents of simulated utterances by minimizing an independently motivated composite cost function. The cost is a function of the constituent durations and encompasses different components that represent independently derived measures of speaker-based production effort, listener-oriented perceptual clarity as well as time conceptualized as a resource shared between both parties, linked to transmission efficiency. The trade-offs between these influences can be globally and locally adjusted by weights assigned to individual cost components within the composite cost function. We show that this approach facilitates modeling a hierarchy of interacting prosodic features of utterances, such as different degrees of prominence or effects of speaking rate and overall requirements of clarity. We outline the theoretical foundations and the architecture of the model and present results of simulation experiments, demonstrating that the model correctly predicts a range of suprasegmental timing phenomena in stress-accent languages that have not been addressed by a unified model. Results underline the model's capacity to account for several empirical observations regarding durational variation in speech.

We implement two theories about the temporal organization of speech in an optimization-based model of speech timing and conduct simulation experiments in order to test whether both theories can account for the phenomenon of foot-level shortening (FLS) observed in English speech corpora. Results suggest that a model that induces compensatory timing relations between syllables and feet predicts empirical results very accurately. However, we also observe that the FLS effect can equally well be explained under the assumption that suprasegmental timing is confined to localized lengthening effects at the heads and edges of prosodic domains. Implications for theories of speech timing are discussed.

Gestures and speech interact. They are linked in language production and perception, with their interaction contributing to felicitous communication. The multifaceted nature of these interactions has attracted considerable attention from the speech and gesture community. This article provides an overview of our current understanding of manual and head gesture form and function, of the principle functional interactions between gesture and speech aiding communication, transporting meaning and producing speech. Furthermore, we present an overview of research on temporal speech-gesture synchrony, including the special role of prosody in speech-gesture alignment. In addition, we provide a summary of tools and data available for gesture analysis, and describe speech-gesture interaction models and simulations in technical systems. This overview also serves as an introduction to a Special Issue covering a wide range of articles on these topics. We provide links to the Special Issue throughout this paper.

We show how our optimization-based model of speech timing reproduces three effects of prosodic prominence on suprasegmental timing patterns in speech: (1), the durational interaction between lexical stress and pitch accent, (2), polysyllabic shortening in pitch-accented words and (3), differential behavior of prominent and non-prominent syllables under speaking rate variation. We review the literature and present model simulations that replicate reported phenomena. Results underline the capacity of our model to provide a unified account of the temporal organization of speech.

Faced with lack of objective and easily applicable criteria for segmentation of speech into dialogue turns, many authors resort instead to units defined in terms of stretches of speech minimally bounded by silence of some predefined duration. There is, however, no consensus concerning silence thresholds employed. While such thresholds can be established on perceptual grounds, in practice a wide range of values is used. As this has a direct impact on the reported frequencies of silences and overlaps, the discrepancies make comparisons of results across different studies difficult. In an attempt to overcome these problems in the present paper we use the Switchboard corpus to evaluate the expected variability in distributions of inter- and intra-speaker intervals when silence boundary thresholds of inter-pausal units are manipulated. Index Terms: dialogue segmentation, inter-pausal units, gaps and overlaps

German demonstrative pronouns, relative pronouns, and definite articles are segmentally identical but differ strongly in the frequency with which they appear. We examined the production of five such particles in a reading task. In a comparison of orthographically identical word pairs belonging to different lexical classes we found small but significant differences in word and vowel duration, prominence, and spectral similarity. Three of the particles in particular tended to be longer and more prominent when they occurred as demonstrative or relative articles than when they were assigned their usual role as definite articles.

The perception of prosodic prominence is influenced by different sources like different acoustic cues, linguistic expectations and context. We use a generalized additive model and a random forest to model the perceived prominence on a corpus of spoken German. Both models are able to explain over 80% of the variance. While the random forests give us some insights on the relative importance of the cues, the general additive model gives us insights on the interaction between different cues to prominence.

We show how incompressibility, a well-described property of some prosodic timing effects, can be accounted for in an optimization-based model of speech timing. Preliminary results of a corpus study are presented, replicating and generalizing previous findings on incompressibility as a function of increasing speaking rate. We then introduce the architecture of our model and present results of simulation experiments that reproduce the results of the corpus analysis. Results suggest that incompressibility can be interpreted as a consequence of tradeoffs between competing requirements of production efficiency and communicative efficacy.

The goal of the present article is to introduce a new concept of a perception-production timing model in human-machine communication. The model implements a low-level cognitive timing and coordination mechanism. The basic element of the model is a dynamic oscillator capable of tracking reoccurring events in time. The organization of the oscillators in a network is being referred to as the Dynamic Perception-Production Oscillation Model (DPPOM). The DPPOM is largely based on findings in psychological and phonetic experiments on timing in speech perception and production. It consists of two sub-systems, a perception sub-system and a production sub-system. The perception sub-system accounts for information clustering in an input sequence of events. The production sub-system accounts for speech production rhythmically entrained to the input sequence. We propose a system architecture integrating both sub-systems, providing a flexible mechanism for perception-production timing in dialogues. The model's functionality was evaluated in two experiments.

•Acoustic signal: Segmentation into speech and speech pauses • The ESMERALDA speech recognizer is used to detect voice activity more robustly than an approach that is solely based on signal energy. •Visual signal: Segmentation into motion peaks •A peak ranges between two local minima in the amount of changed pixels in the visual signal. • The amount of changed pixels is calculated by summing up a motion history image at each time step. • Temporal association: Overlapping speech and visual segments are associated to one acoustic package.

Perceptual prominence is an important indicator of a word’s and syllable’s lexical, syntactic, semantic and pragmatic status in a discourse. Its automatic annotation would be a valuable enrichment of large databases used in unit selection speech synthesis and speech recognition. While much research has been carried out on the interaction between prominence and acoustic factors, little progress has been made in its automatic annotation. Previous approaches to German relied on linguistic features in prominence detection, but a purely acoustic method would be advantageous. We applied an algorithm to German data that had been previously used for English and Italian. Both the algorithm and the data annotation encode prominence as a continuous rather than a categorical parameter. First results are encouraging, but again show that prominence perception relies on linguistic expectancies as well as acoustic patterns. Also, our results further strengthen the view that force accents are a more reliable cue to prominence than pitch accents in German. Index Terms: prosody, prominence, German

In order to gain knowledge about the interaction between top-down expectations of listeners concerning prosodic prominence and its acoustic correlates, two exploratory empirical studies were carried out. First, native and non-native subjects rated prominences of speech read at normal and very fast - prosodically very different - speech. Later, these ratings were compared with introspective prominence ratings of different listeners. First results indicate a major influence of the introspection on prominence ratings, especially if acoustic cues are difficult to interpret, as it is the case in very fast speech. Compared to native subjects, non-natives rely less on their introspection and more on the acoustics.

Previous research identified a differential contribution of information structure and the visibility of facial and contextual information to the acoustic-prosodic expression of pitch accents. However, it is unclear whether pitch accent shapes are affected by these conditions as well. To investigate whether varying context cues have a differentiated impact on pitch accent trajectories produced in conversational interaction, we modified the visibility conditions in a spontaneous dyadic interaction task, i.e. a verbalized version of TicTacToe. Besides varying visibility, the game task allows for measuring the impact of information-structure on pitch accent trajectories, differentiating important and unpredictable game moves. Using GAMMs on four speaker groups (identified by a cluster analysis), we could isolate varying strategies of prosodic adaptation to contextual change. While few speaker groups showed a reaction to the availability of visible context cues (facial prosody or executed game moves), all groups differentiated the verbalization of unpredictable and predictable game moves with a groupspecific trajectory adaptation. The importance of game moves resulted in differentiated adaptations in two out of four speaker groups. The detected strategic trajectory adaptations were characterized by different characteristics of boundary tones, adaptations of the global f0-level, or the shape of the corresponding pitch accent.

We report on the first results of an experiment designed to investigate properties of communicative feedback produced by nonattentive listeners in dialogue. Listeners were found to produce less feedback when distracted by an ancillary task. A decreased number of feedback expressions communicating understanding was a particularly reliable indicator of distractedness. We argue this finding could be used to facilitate recognition of attentional states in dialogue system users. Index Terms: communicative feedback; dialogue; distraction; engagement; attention; dual task

Prominence has been widely studied on the word level and the syllable level. An extensive study comparing the two approaches is missing in the literature. This study investigates how word and syllable prominence relate to each other in German. We find that perceptual ratings based on the word level are more extreme than those based on the syllable level. The correlations between word prominence and acoustic features are greater than the correlations between syllable prominence and acoustic features.