Search

To date, comparisons between sign languages (SL) and co-speech gestures (CSG) still face major challenges.One of them is the difficulty to choose, among all the existing transcription systems (TranSys), a single TranSys that could allow researchers to take into account the peculiar characteristics of SL and CSG. Even if the researchers of these two fields describe the same “gesturing body”, they have developed their own representation models (with their own form and scope) to code the features they are interested in. In fact, all these systems adopt a perceptual point of view , i.e. a visuo-spatial representation of the gesture, and are therefore subject to different insight (e.g., in the division of space {1}). This can lead to a fragmentation of the descriptive modalities, limiting the identification of formal and functional common characteristics between SL and CVG. Doing so, researchers currently ignore a rallying perspective allowing to describe both SL and CSG: the body structure. To fill this gap, we have developed Typannot {2}, a typographic TranSys that aims at representing gestures from an articulatory standpoint, using a descriptive approach rooted in the body: the kinesiological approach {3}.Designed for SL but suitable for CSG too, Typannot is a formal TranSys that reproduces the articulatory characteristics of each segment of the body (fingers, hand, forearm, arm; shoulder, torso and hip; neck, head, mouth and eyes). It describes their possibilities of organization from the body articulatory perspective: the positions and the transformations of the degrees of freedom. In Typannot, each articulator (upper-limb, bust, mouth and eyes segments) is associated with a different OpenType font; every character (in Unicode sense) of a Typannot font contains only one specific information (i.e., the degree of extension or abduction) necessary to describe all the features of a segment. An advanced system of typographic ligature allows the user to see a single “holistic” glyph containing every feature, thus assuring searchability in a still readable way.Another challenge faced by LS and CSG TranSys is the time required to transcribe gestural phenomena, which drastically restricts the size of multimodal corpus and therefore the possibility of generalizations. Here again, Typannot deals with this issue by facilitating and automating the work. On one hand, the input interface has been designed to help transcribers produce transcription easily; on the other hand, the articulatory model used in this TranSys can easily be associated with Motion Capture (MoCap) data. This correspondence makes it possible to envisage a (semi)automatic transcription of body structure, drastically minimizing the transcription time.Typannot has been used within the ANR-LexiKHuM project (2021-2025) {4} to transcribe the body postures present in the multimodal and parallel corpus DEGELS1 {5}, consisting of conversations in LSF (≃ 30 min) and in spoken French (≃ 30 min). One of the objectives of LexiKHuM is to identify the epistemic markers common to SL and to CSG, in particular those that are not brachial. Thanks to Typannot, an articulatory description of the upper trunk and of the facial expressions could be implemented out in a coherent manner between the data in SL and CSG. The corpus was transcribed using the ELAN software {6} and the annotation was carried out manually and with the sensor-free MoCap software AlphaPose {7}, allowing a semi-automatic annotation in Typannot characters.In this talk, we will present the kinesiological approach and the Typannot typographic font system; some transcribed data and preliminary results of LexiKHuM will be exposed, also focusing on the time used for manual and semi-automatic transcription.******This presentation is dedicated to the memory of Dominique Boutet (1966-2020), who was co-author of our proposal in 2020 and co-coordinator of the Typannot and the LexiKHuM projects.Bibliography{1} McNeill, D. (1992). Hands and mind: what gestures reveal about thought. Chicago: University of Chicago Press.{2} Danet C., Boutet D., Doan P., Bianchini C.S., Contesse A., Chevrefils L., Rébulard M., Thomas C. & Dauphin J.-F. (2021). Transcribing sign languages with Typannot: a typographic system which retains and displays layers of information. Grapholinguistics and its Applications, 5(2), 1009–1037. https://doi.org/10.36824/2020-graf-dane{3} Boutet D. (2018). Pour une approche kinésiologique de la gestualité: synthèse. Habilitation à diriger des recherches, Université́ de Rouen-Normandie.{4} Catteau F., Morgenstern A. & Bianchini C.S. (2022). From gestures to kinesthetic modality: how to express epistemicity in a haptic device for human-machine interaction. 9th Intl Conf. of Intl. Soc. for Gesture Studies [ISGS 2022].{5} Braffort A. & Boutora L. (2012), DEGELS1: a comparable corpus of French Sign Language and co-speech gestures, Proc. 8th Intl. Conf. on Language Resources and Evaluation [LREC 2012].{6} Wittenburg P., Brugman H., Russel A., Klassmann A. & Sloetjes H. (2006). ELAN: a professional framework for multimodality research. Proc. 5th Intl. Conf. on Language Resources and Evaluation [LREC 2006].{7} Fang H.-S., Xie S., Tai Y.-W. & Lu C. (2018). RMPE: Regional Multi-person Pose Estimation. arXiv:161200137 [cs].

The LexiKHuM project (Human-Machine Kinesthetic Lexicon) [1] studies and develops kinesthetic communication solutions for human-machine interaction. Our objective is to implement a kinesthetic lexicon that will be held on a haptic device for human-machine communication. We therefore explore human gestuality (both co-verbal gesture [CVG] and sign language [SL]) to extract features which can be reinvested in the kinesthetic lexicon, in order to ensure its closeness to natural language. One of the features we would like to introduce in our system is the epistemic dimension of the messages transmitted toward the human being, and thus the degree of certitude of the machine [2].For this purpose, we have gathered the articulatory and phonetic properties of the epistemic gesture in CVG and SL. Previous studies have shown that the head is an epistemic marker and its articulatory characteristics express the different degrees of certitude. For example, in German and Turkish SL, the extent of certitude is expressed by rapid head nods and the incertitude by slow head tilts and nods [3-4]. Similarly, in French, English and Catalan CVG, the degree of certitude is marked by a clear and strong nod of the head, and the incertitude by a tilt [5-6-7-8].We hypothesized that it is possible to identify in these head movements – just broadly described in the literature – some sub-parametric articulatory features which carry epistemic meaning. We thus propose to identify these traits through the study of a corpus of French CVG and SL [LSF], based on an analysis grid of the articulatory characteristics of these movements resulting from previous works [9-10-11-12-13]. Thereafter, we will show how these traits can be used to build a kinesthetic lexicon as close as possible to the natural language. This work is the precursor of a larger study on the identification of epistemic sub-parametric traits which are also present on other articulators.References:[1] LexiKHuM project (2021-2025). https://anr.fr/Projet-ANR-20-CE33-0012[2] Gosselin L. (2015). De l’opposition modus / dictum à la distinction entre modalités extrinsèques et modalités intrinsèques. Bulletin de la Société de Linguistique de Paris, CX-1: 1-50.[3] Herrmann A. (2013).Modal and focus particles in Sign Languages: across-linguistics study. De Gruyter Mouton (Berlin/Boston).[4] Karabüklü S., Bross F., Wilbur R., Hole D. (2018). Modal signs and scope relations in TİD. Proc. FEAST: Formal and Experimental Advances in Sign language Theory, 2: 82-92.[5] Roseano P., González M., Borràs-Comes J., Prieto P. (2016). Communicating epistemic stance: how speech and gesture patterns reflect epistemicity and evidentiality. Discourse Processes, 53: 135-174.[6] Borràs-Comes J., Kiagia E., Prieto P. (2019). Epistemic intonation and epistemic gesture are mutually co-expressive: empirical results from two intonation-gesture matching tasks. Journal of Pragmatics, 150: 39-52.[7] Ferré G. (2012). Functions of three open-palm hand gestures. Multimodal Communication, 1: 5-20.[8] Debras C. (2017). The shrug: forms and meanings of a compound enactment. Gesture, 16(1): 1-34.[9] Puupponen A., Wainio T., Burger B., Jantunen T. (2015). Head movements in Finnish Sign Language on the basis of motion capture data: a study of the form and function of nods, nodding, head thrusts, and head pulls. Sign Language & Linguistics, 18: 41-89.[10] Wilbur R.B., Malaia E. (2018). A new technique for analyzing narrative prosodic effects in sign languages using motion capture technology. in: Hübl A., Steinbach M. [eds], Linguistik Aktuel - Linguistics Today, 247: 15-40.[11] Boutet D. (2018). Pour une approche kinésiologique de la gestualité. HDR, Université de Rouen.[12] Catteau F. (2020). Traduire la poésie en langue des signes : l’empreinte prosodique lors du changement de modalité. PhD dissertation, Université Paris 8.[13] Chevrefils L., Danet C., Doan P., Thomas C., Rébulard M., Contesse A., Dauphin J.-F., Bianchini C.S. (2021). The body between meaning and form: kinesiological analysis and typographical representation of movement in Sign Languages. Languages and Modalities, 1: 49-63.

Notre objectif est de r��fl��chir sur les ph��nom��nes prosodiques dans la traduction vers le fran��ais de la po��sie en langue des signes. Gr��ce �� une m��thodologie exp��rimentale de recueil et d���analyse de corpus, des strat��gies interpr��tatives ont ��t�� identifi��es : un nombre de syllabes en fran��ais plus ��lev�� lorsque le d��bit du po��me est ralenti ; une dur��e moyenne de ces syllabes et une dur��e des pauses plus longue dans les traductions des po��mes au d��bit ralenti.

International audience

International audience; For almost ten years, the marketing of low cost motion capture devices such as the Microsoft Kinect sensor has enabled numerous studies in real-life settings (Mousavi Hondori & Khademi, 2014; Webster & Celik, 2014; Springer & Yogev Seligmann, 2016). Whereas most of this work with elderly people is studying gait and fall risks (see Rougier, Auvinet, Rousseau, Mignotte, & Meunier, 2011 for example), in the present paper we propose to focus on the building of the SignAge corpus dedicated to the study of signing in elderly deaf participants with low cost motion capture devices. Up to now, a (preferably multi-)camera setup was considered a basic requirement in sign language studies, sometimes completed with much more intrusive or expensive equipment such as data gloves, optical motion capture systems (Channon, 2015, p. 132–133). But latest technology advancements allow us to quantify 3D-motions and their time derivatives at a reasonable price. Our newly built SignAge corpus1 of interactions between elderly deaf signers in LSF takes advantage of such advancements. The SignAge corpus combines data acquired by: - 2 digital video cameras, with a plan on the signing interviewee’s upper body and a plan on the whole interaction (similarly to the protocol developed in CorpAGEst, Bolly & Boutet, 2016) - 2 Noitom Perception Neuron body straps, each equipped with 25 IMU (Inertial Measurement Units), recorded with Axis Neuron software - 1 Kinect for Windows v2 (also known as Kinect for Xbox One) depth sensor centered on the interviewee, recorded with Brekel Pro Body v2. Thus, for each participant, our 5 timed data flows are: 2 video streams at 25 fps – synchronised, visualised and annotated in the ELAN software (Sloetjes & Seibert, 2016) –, and BioVision Hierarchy (BVH) files (see Meredith & Maddock, 2001). These BVH files are visualised in Motion Inspector, and the following descriptors are computed in Matlab for the three age groups in our sample: the global quantity of motion (Sarasúa & Guaus, 2014) and the variation of the signing amplitude for each joint of the upper limbs, with the attempt to establish a correlation between age and the articulatory segment involved. For now, after manually post-synchronising our flows thanks to start and end claps realised by both the interviewer and the interviewee during the recording session, we are assessing the quality of our data. Preliminary results show that temporal resolution can be questionable for the Kinect used in conjunction with Brekel, whereas in the Perception Neuron, spatial drift is a problem. Hence, these two devices seem to be complementary: the Kinect gives the absolute movements of the body, whereas the neuron is much precise for relative movements. Furthermore, a post interview feedback with each subject showed a high acceptance of the protocol by elderly signers (see figure below). As a conclusion, knowing the technical limitations it appears that Kinect and Neuron might still be an interesting choice to get usable additional 3D data in aging studies because of their portability and the ease with which participants get accustomed to wearing the body straps.

International audience; Sign languages corpora of LSFB (Meurant 2015) and LSQ (Parisot & Rinfret 2015; Luna 2015) include data from elderly signers. Parisot (2017) has given more details about the analysis of verb agreement in the senior LSQ signers suggesting that “ even if verb agreement is sometimes difficult to perceive in seniors’ productions, it seems to be rather a phonetic difference than a morphosyntactic one”. Our SignAge corpus of interactions between elderly deaf signers in LSF aims to test motion capture systems to complement analysis on the basis of video recordings. Motion capture has the advantage to capture three-dimensional data, including the possibility to change the point of view. It also allows to extract more abstract movement descriptors such as speed or acceleration of the different body parts. Furthermore, it is a step forward to get semi-automatic annotation since manual annotation is time consuming. We could not use a mocap system with many reflective markers and infrared cameras in a mocap studio and we needed a less invasive mocap setting. Then, we conducted complementary tests with Kinect and Inertial Measurement Unit sensors which permit measuring acceleration and relative position differences on xyz axis. Following some of the steps from the methodological protocol developed in CorpAGEst ( http://corpagest.org/ ), we tried to involve relatives or friends as interviewers; the 13 participants they interviewed are over 65 years old (Table 1) and use LSF as their main language in everyday life. Two deaf ‘complices’ conducted 6-min interviews on the themes of the evolution of transport, access to care, culture and new technologies (Fig. 1 and our companion paper for technical details). We explore gestural flow with a kinesiological model based on linguistics and we are currently analysing i) the variation of the signing amplitude per each degree of freedom, ii) the global movement quantity and iii) the way it is correlated with which articulatory segment is concerned, in each of the three age groups in our sample. One of the hypothesis is that the ranges of motion of degrees of freedom are reduced for elderly people compared to younger people. Consequently, the transfer of the movement is much more present than for the younger. The same for finger selection: the ratio of the inertial mass between the hand and the forearm is lower for elderly than for younger people. This fact exerts a motion further on the forearm and a smoother gesture. We expect also two tendencies: the signs which clearly respond to a proximal-distal flow engaging the arm, the forearm and the hand in the movement should show proportionally less movement on the hand than we have for younger people; for the signs which clearly respond to a distal-proximal flow engaging the hand and the forearm, the movement of the forearm should be more developed than the ones of the younger people. We will present some preliminary results and the contrast between elderly and younger people (inside the panel) is reinforced in each interaction when the interviewer is the younger signer.

International audience; CIGALE (Capture et Interaction avec des Gestes Artistiques, Langagiers et Expressifs) is an artistic and dialogical platform that explores the hybridization between human and virtual gestures in real time. This platform allows studying complementary aspects of gestures, from the linguistic field to the arts. CIGALE allows understanding the relationships between symbolic gestures (coverbal ones), expressive gestures (choral conducting and poetry) and full body gestures (mime). Starting from motion capture (1), we have created four databases of gestures. The platform allows to study and analyses gestures, strokes and meaning (2). We propose then to generate new gestures using genetic algorithm that converges to new gestures inspired by the ones of the database (3). In order to visualize the gestures, we can render the motion on different characters by changing body aspect(mesh), using particles effect or trails (4). At least, it’s possible to interact using gestures with the virtual characters that can answer autonomously to the user or to the audience (5).

International audience; Prosodic variation and poetic translation (LSF/French): What happens to prosody with a channel change? The study of spoken language prosody is partially based on the measurement of three parameters: sound duration, intensity and frequency. Sign languages use the visual-gestural channel and mobilize manual and non manual articulators (chest, head, facial features). Our study aims to establish tools that allow to compare, on the prosodic level, the translation of poetic sequences into French and their original versions in French Sign Language (LSF). We collected video and motion capture data-which offers wide avenues for exploration of LSF prosodic features-as well as audio data of French translations-that reveal the strategies of interpreters to convey prosodic variation.; L'étude de la prosodie des langues vocales repose en partie sur la mesure des paramètres de durée, d'intensité et de fréquence sonores. Les langues des signes, quant à elles, empruntent le canal visuo-gestuel et mobilisent des articulateurs manuels et non manuels (buste, tête, éléments du visage). Notre étude a pour objectif d'établir des outils permettant de comparer, au niveau prosodique, la traduction en français de séquences poétiques et la version originale en langue des signes française (LSF). Nous avons recueilli des données vidéo augmentées de capture de mouvement-qui offrent plusieurs pistes d'exploration des paramètres prosodiques pour la LSF-ainsi que des données audio des traductions en français-qui révèlent les stratégies des interprètes pour interpréter la variation prosodique

International audience; Like human communication, gestural communication with a virtual actor (VA) depends on the environment and the type of interaction. This type of exchange tells us about human communication in a paradigm of "rediscovered body" as well as about the asymmetric communication strategies. The corpus designed in the CIGALE project consists of the recording and analysis of gestural interactions between a real actor (RA) and a VA. The perceptual system of this VA prevents the normal course of multimodality. In this gestural interaction model, modality asymmetry imposed by the VA sheds light on relational implications of the choice of gestures the real actor makes. The interaction loop is controlled by the VA which must detect (gestural) activity of the real person, to perform a (defined) action in response. The gesture analysis (carried out in ELAN) of 7 dyads, shows that during the interaction, the RA adapts to the VA stimuli by adopting speeds and amplitudes similar to those of the VA, by focusing on the arm movements and by more or less neutralizing distal movements (hands and fingers). This adjustment corresponds to the limited gestural production of the VA, by a perception / action internal connection based on the Microsoft Kinect sensor (no capture of finger movements or of pronosupination). In the real person this leads to a tendency to empathise, maintaining him/her in the imitative loop initiated by the VA. This mode of gesture exchange based on imitation incites the real actors to incorporate the limitations of the joints of the VA, down to the shape of the gesture. Moreover, empathy exceeds imitation, since the emerging gestural style of the real actors even appears in non-imitative productions. This empathy with the VA fosters arm and forearm movements of the real actors and thus directs the gesture to the movement of these proximal segments. By doing this, it resembles the gestures of children before 17 months and promotes relational gestures initiated by the arms, at the expense of manipulative actions, initiated by the hands. This interaction model sheds new light on gestural exchanges: Would the lack of vocal production and the limitations, including gestural and/or mimicry restriction, foster empathy in the interaction with the VA? Would there be, because of this type of gesture constraint, a relationship to build - as with a child?; A l’instar de la communication entre humains, la communication gestuelle avec un acteur virtuel (AV) dépend de l’environnement et du type d’interaction. Ce type d’échanges nous renseigne sur la communication humaine dans un paradigme du “corps retrouvé” autant que sur les stratégies de communication asymétrique. Le corpus conçu dans le projet CIGALE consiste dans l’enregistrement et l’analyse d’interactions gestuelles entre un acteur réel et un AV. Le système de perception de cet AV empêche le cours normal de la multimodalité. Dans ce dispositif d’interactions gestuelles, l’asymétrie de modalités qu’impose l’AV éclaire les implications relationnelles des choix de gestes de l’acteur réel. Les interactions sont régies par cet AV qui doit percevoir l'activité (gestuelle) de la personne réelle, afin d’exécuter une action (déterminée) en réponse. L’analyse gestuelle menée (sous ELAN) sur 7 dyades, montre que, dans l’interaction, les personnes réelles s’adaptent aux stimuli de l’AV en adoptant des vitesses et des amplitudes analogues à celles de l’AV, en se focalisant sur des mouvements des bras et en neutralisant plus ou moins les mouvements distaux (mains et doigts). Cette adaptation correspond à une production gestuelle limitée de l’AV, par un couplage interne perception/production basé sur un même dispositif, le capteur de mouvements Kinect (absence de captation des mouvements des doigts et de la pronosupination). Ceci entraîne, chez la personne réelle, une tendance à l’empathie, la maintenant dans la boucle imitative initiée par l’AV. Ce mode d’échanges gestuels basé sur l’imitation pousse les acteurs réels à intégrer les limitations des articulations de l’AV, jusque dans la forme de la gestualité. De plus, l’empathie dépasse l’imitation, puisque le style gestuel émergent des acteurs réels apparaît y compris dans les productions non imitatives. Cette empathie avec l’AV favorise les mouvements des bras et des avant-bras des acteurs réels et, ainsi, oriente la gestualité vers le mouvement de ces segments proximaux. Elle s’apparente en cela à la gestuelle des enfants avant 17 mois et promeut les gestes relationnels initiés sur les bras, au détriment des gestes manipulatoires, initiés sur les mains. Ce dispositif apporte un éclairage nouveau aux échanges gestuels : Est-ce l’absence de production vocale et la limitation y compris gestuelle et/ou le mimétisme qui favoriseraient l’empathie dans l’interaction avec l’AV ? Y aurait-il, en raison de ce type de gestualité contrainte, une relation à construire — comme avec un enfant ?

National audience

International audience

International audience; Based on new technologies, the project CIGALE proposes to create an artistic device that questions possible hybridizations between humans and the virtual. The device will enable linguistic and artistic experiments, complementary therefore transdisciplinary. A real platform for non-verbal gestures and symbolic interactions, this robust and affordable device should be able to provide generated gestures whose parameters can be defined and a possible interaction with a real person.

Nous présentons ici une méthode de caractérisation d'unités gestuelles coverbales, en vue d'une exploitation dans une interaction humain-avatar. Nous avons enregistré 12 types de gestes avec un système de capture de mouvement. Nous avons utilisé les signaux de position obtenus afin d'en dégager des unités gestuelles à l'issue d'une segmentation de la partie significative. Pour soutenir notre analyse linguistique des gestes, nous présentons les hypothèses biomécaniques, notre méthode de segmentation, les hypothèses de caractérisation et les résultats obtenus.

This research project focuses on the translation of poetry created in French Sign Language (LSF) into oral French, and, more specifically, on the impact of the prosodic structure of LSF on that of French in the context of poetic translation. In all languages, both vocal and signed, prosody organizes discourse in relation to its informational structure, whether in terms of perception or production. The objective of this study is to determine if, in a context of poetic translation, the prosodic structure of LSF has an impact on the construction of oral French, and more specifically on its prosody.To conduct this study, a new protocol for data collection and analysis was developed, which combines the analysis of gestural and sound signals with the manual annotation of both languages. I collected eleven poems in LSF from five deaf artists and fifty-seven translations of them by nine translators specialized in poetry. In this study, I make an inventory of the prosodic phenomena that participate in the segmentation of the poems in LSF, particularly in relation to hand and bust movements.The poems have therefore been segmented into constituents on several hierarchical levels. In addition, through an analysis of the French translations, I identify eighteen translatory strategies related to prosody. 74% of the prosodic constituents of the translations are identified as being directly related to the prosodic groups of the hierarchical structure of poems in LSF.The results show that, in the context of poetic translation, the prosodic structure of LSF does have an impact on the construction of the French., Ce travail de recherche s’intéresse à la traduction en français oral de la poésie créée en langue des signes française (LSF) et plus particulièrement à l’impact de la structure prosodique de la LSF sur celle du français en contexte de traduction poétique. Dans toutes les langues, qu’elles soient vocales ou signées, la prosodie permet d’organiser le discours en lien avec sa structure informationnelle, que ce soit en perception ou en production.L’objectif de cette recherche est de déterminer si, dans un contexte de traduction poétique, la structure prosodique de la LSF a un impact sur l’élaboration du français oral, et plus particulièrement sur sa prosodie.Pour mener cette étude, un protocole inédit de recueil et d’analyse de données, qui associe analyse des signaux gestuels et sonores à l’annotation manuelle des deux types de discours, a été élaboré. Nous avons recueilli onze œuvres en LSF de cinq artistes sourds et leurs cinquante-sept traductions, proposées par neuf traducteurs experts en poésie.Dans ce travail, nous avons inventorié les phénomènes prosodiques qui participent à la segmentation des œuvres en LSF, notamment en lien avec mouvements des mains et du buste. Les œuvres ont ainsi été segmentées en constituants selon plusieurs niveaux hiérarchiques. De plus, nous avons montré que l’analyse des traductions en français révèlent que dix-huit stratégies traductives en lien avec la prosodie sont utilisées. 74% desconstituants prosodiques des traductions ont été identifiés comme directement liés avec les groupes prosodiques de la structure hiérarchique des œuvres en LSF. Cette étude montre que, dans un contexte de traduction poétique, la structure prosodique de la LSF laisse bien une empreinte dans la construction du français.