Search

Joking draws on complex cognitive abilities: understanding social norms, theory of mind, anticipating others responses and appreciating the violation of others expectations. Playful teasing, which is present in preverbal infants, shares many of these cognitive features. There is some evidence that great apes can tease in structurally similar ways, but no systematic study exists. We developed a coding system to identify playful teasing and applied it to video of zoo-housed great apes. All four species engaged in intentionally provocative behaviour, frequently accompanied by characteristics of play. We found playful teasing to be characterized by attention-getting, one-sidedness, response looking, repetition and elaboration/escalation. It takes place mainly in relaxed contexts, has a wide variety of forms, and differs from play in several ways (e.g. asymmetry, low rates of play signals like the playface and absence of movement-final holds characteristic of intentional gestures). As playful teasing is present in all extant great ape genera, it is likely that the cognitive prerequisites for joking evolved in the hominoid lineage at least 13 million years ago.

Recent cognitive science research suggests that occasional “blindness” to choice manipulations indicates a lack of awareness in choice making. This claim is based on participants’ tendency not to detect choice manipulations and the similarity between their justifications for choices they made and those they were tricked into believing they made. Using a cognitive-semiotic framework, we argue that such conclusions underestimate the embodied, intersubjective nature of human meaning-making. We support this by investigating choice awareness beyond language to include non-verbal behavior. Forty-one participants were asked to choose from pairs of photographs of human faces the one they found most attractive and then to justify their choices, without knowing that for some of the trials they were asked to justify a choice that they had not made. Verbal responses were categorized as (i) non-manipulated, (ii) detected manipulated, and (iii) undetected manipulated trials. Bodily expressions, assessed using five different Categories of Bodily Expression (CBE): Adaptors, Torso, Head, Face and Hand expressions, revealed differences in: (a) duration, (b) rates of occurrence and (c) variety of the CBEs across trials. Thus, even when manipulations were not verbally detected, participants took longer to assess choices, showed increased bodily expressions, and engaged more body parts in undetected manipulations compared to non-manipulated choice trials. This suggests a degree of awareness to the choice manipulation, even if pre-reflective, manifested in participants’ bodily expressions. [ABSTRACT FROM AUTHOR]

Human Activity Recognition (HAR) is a highly promising research area meant to automatically identify and interpret human behaviour using data received from sensors in various contexts. The potential uses of HAR are many, among them health care, sports coaching or monitoring the elderly or disabled. Nonetheless, there are numerous hurdles to be circumvented for HAR's precision and usefulness to be improved. One of the challenges is that there is no uniformity in data collection and annotation making it difficult to compare findings among different studies. Furthermore, more comprehensive datasets are necessary so as to include a wider range of human activities in different contexts while complex activities, which consist of multiple sub‐activities, are still a challenge for recognition systems. Researchers have proposed new frontiers such as multi‐modal sensor data fusion and deep learning approaches for enhancing HAR accuracy while addressing these issues. Also, we are seeing more non‐traditional applications such as robotics and virtual reality/augmented world going forward with their use cases of HAR. This article offers an extensive review on the recent advances in HAR and highlights the major challenges facing this field as well as future opportunities for further researches. [ABSTRACT FROM AUTHOR]

In the period between birth and approximately 9 months, adults introduce infants to triadic (adult–infant–object) communication. The objective of this study is to explore how early triadic interactions arise and develop in moderate–late preterm infants. We observed and analyzed seven preterm infants at 2, 3, and 4 months of age in triadic interactions with a caregiver and a sonorous object. In the first 2 months of the infant's life, the responsibility for the interaction rests with the adult. As time went by, parents used a wider range of semiotic systems to communicate with the child, about and through the object. The results demonstrated the presence of triadic interactions in a preterm population beginning in the third month of life. Interaction is structured with the object, its sound, rhythm, and pauses. The parents' use of pauses was adjusted to the child's availability. These pauses became longer and more communicative over time, which allowed more participation on the part of the infants. Communication is facilitated by the rhythmic components of the triadic interaction (time/pauses/structure of the interaction sequence). [ABSTRACT FROM AUTHOR]

Background: Aphasia, resulting from acquired brain injury, disrupts language processing and usage, significantly impacting individuals' social communication and life participation. Given the limitations of traditional assessments in capturing the nuanced challenges faced by individuals with aphasia, this study seeks to explore the potential benefits of integrating multimodal communication elements into discourse analysis to better capture narrative proficiency in this population. Objective: This study examined how incorporating multimodal communication elements (e.g., physical gestures, writing, drawing) into discourse analysis may affect the narrative outcomes of persons with aphasia compared to those observed using methods that exclude multimodal considerations. Methods: Participants included individuals with chronic aphasia and age-and education-matched healthy controls who completed a storytelling task--the Bear and the Fly story. Macrolinguistic scores were obtained using verbal-only and multimodal scoring approaches. Additionally, the frequency and type of multimodal communication use during storytelling were examined in relation to aphasia characteristics. Statistical analyses included both within-group and between-group comparisons as well as correlational analyses. Results: Individuals with aphasia scored significantly higher in terms of their macrolinguistic abilities when multimodal scoring was considered compared to verbal-only scoring. Within the aphasia group, there were prominent differences noted in macrolinguistic scores for both fluent and nonfluent aphasia. Specifically, both groups scored higher on Main Concepts when multimodal scoring was considered, with the nonfluent group demonstrating significantly higher Main Concept and total macrolinguistic rubric scores in multimodal scoring compared to verbal scoring on the storytelling task. Additionally, aphasia severity showed moderate positive correlations with total macrolinguistic scores, indicating that individuals with less severe aphasia tended to produce higher quality narratives. Lastly, although persons with aphasia used different types of nonverbal modalities (i.e., drawing, writing), the use of meaning-laden gestures was most predominant during storytelling, emphasizing the importance of multimodal elements in communication for individuals with aphasia. Conclusion: Our preliminary study findings underscore the importance of considering multimodal communication in assessing discourse performance among individuals with aphasia. Tailoring assessment approaches based on aphasia subtypes can provide valuable insights into linguistic abilities and inform targeted intervention strategies for improving communication outcomes. [ABSTRACT FROM AUTHOR]

Using a touchscreen mobile device while walking in sunlight glare limits the visual attention. This situational visual impairment could affect the usability of these devices. This work focuses on the user interaction in situational visual impairment regarding touch gestures and no-visual feedbacks. We performed two user studies. In the first study, we modified the user interface of the two most used applications, phone calling and text messaging, using touch gesture and audio feedback while these features were not used before. Then a user study performed to extract the user preferences of modified applications in visual impairment situation. Our findings indicate that participants preferred gestures, but did not clearly indicate a preference for audio feedback. In the second study, we employed the computational evaluation to explore the distinctions between the made gestures by individuals in two different situations: with and without visual impairment. The outcomes show that in situations with visual impairment: (1) the people draw bigger and broader gestures, (2) there is higher error in features related to geometric shape, (3) familiarity of the user with gestures changes the speed and time for drawing them, and (4) there is not a critical distinctive conduct in how to apply strokes. The user interaction in situational visual impairment could be improved using touch gesture without audio feedback. This situation needs a wider area to make the gestures. [ABSTRACT FROM AUTHOR]

One promising method to tackle the question, "In which modality did language evolve?" is by studying the ontogenetic trajectory of signals in human's closest living relatives, including chimpanzees (Pan troglodytes). Concerning gestures, current debates centre on four different hypotheses: "phylogenetic ritualization", "social transmission through imitation", "ontogenetic ritualization", and "social negotiation". These differ in their predictions regarding idiosyncratic gestures, making such occurrences a crucial area of investigation. Here, we describe a novel and potential idiosyncratic behaviour — 'hand-on-eye' — which was initially observed in one mother-infant dyad in a community of chimpanzees living in the wild. We systematically investigated the form, sequential organisation, intentionality, usage, function, and distribution of the behaviour over a five-year period. The results showed that 'hand-on-eye' was nearly exclusively deployed in a single mother-infant dyad, was accompanied by hallmarks of intentionality, and served to initiate or resume joint dorsal travel. Although the behaviour was observed once in each of three other mother-infant dyads, these lacked the same frequency and hallmarks of intentionality. 'Hand-on-eye' thus qualifies as an idiosyncratic gesture. The proposed developmental pathway gives support to both the "ontogenetic ritualization" and "social negotiation" hypotheses. It also stresses the crucial need for longitudinal approaches to tackle developmental processes that are triggered by unique circumstances and unfold over relatively long time windows. [ABSTRACT FROM AUTHOR]

Temporal coordination of communicative behavior is not only located between but also within interaction partners (e.g., gaze and gestures). This intrapersonal synchrony (IaPS) is assumed to constitute interpersonal alignment. Studies show systematic variations in IaPS in individuals with autism, which may affect the degree of interpersonal temporal coordination. In the current study, we reversed the approach and mapped the measured nonverbal behavior of interactants with and without ASD from a previous study onto virtual characters to study the effects of the differential IaPS on observers (N = 68), both with and without ASD (crossed design). During a communication task with both characters, who indicated targets with gaze and delayed pointing gestures, we measured response times, gaze behavior, and post hoc impression formation. Results show that character behavior indicative of ASD resulted in overall enlarged decoding times in observers and this effect was even pronounced in observers with ASD. A classification of observer's gaze types indicated differentiated decoding strategies. Whereas non-autistic observers presented with a rather consistent eyes-focused strategy associated with efficient and fast responses, observers with ASD presented with highly variable decoding strategies. In contrast to communication efficiency, impression formation was not influenced by IaPS. The results underline the importance of timing differences in both production and perception processes during multimodal nonverbal communication in interactants with and without ASD. In essence, the current findings locate the manifestation of reduced reciprocity in autism not merely in the person, but in the interactional dynamics of dyads. [ABSTRACT FROM AUTHOR]

We argue that sign language requires a radical extension of formal semantics. It has long been accepted that sign language employs the same logical machinery as spoken language (occasionally making its abstract components overt), and simultaneously makes extensive use of iconicity. But the articulation between these two modules has only been discussed piecemeal. To capture it, we propose an 'iconological semantics' that combines standard logical semantics with a pictorial semantics in the Greenberg/Abusch tradition. We start by reanalyzing from this perspective earlier data on iconic loci, which are simultaneously variables and simplified depictions of their denotations. We then analyze new data on ASL classifier predicates, constructions that are lexically specified as being iconic. Their behavior argues for a very expressive system, possibly one in which the object language contains viewpoint variables. These can be left free or they may depend on quantifiers, and distinct viewpoint variables can co-occur in a given sentence; this gives rise to an extraordinary interaction between depictions and logical operators. We then sketch an adaptation of pictorial semantics to the dynamic 3D representations used in sign language. Finally, we suggest that iconological semantics might also illuminate the interaction between logical operators and pro-speech gestures in spoken language. In the end, the standard view of language as a discrete compositional system must be revised: it also has a tightly integrated depictive component, and 'textbook semantics' should be revised to capture this fact. [ABSTRACT FROM AUTHOR]

Previous studies on gestures and personality suggest an ambiguous picture of the effects of the various personality dimensions on the different types of speechgestures and adaptors. In foreign language learning an additional variable to take into account is proficiency, which some studies have shown to affect gestures. In this study, we explore how various intrinsic variables, including personality, proficiency, gender and age affect the gestures of 61 Cantonese speakers of English as a second language. Participants were asked to narrate a video cartoon. Their proficiency and the frequency of gestures produced was based on those narrations. A functional categorization of gestures was followed, dividing them into semantic and discursive, and that also noted adaptors and micro-gestures, referred to as "flutters." Personality was self-reported using the 44 question Big-Five inventory. Correlations and a series of generalized linear models were developed to explore the interplay between variables. Agreeableness was found to be positively correlated with semantic gestures; and neuroticism and age were negatively correlated with flutter duration. Contrary to the findings from previous studies, no significant relationships were found between neuroticism and adaptors or semantic gestures, nor between extraversion or openness and semantic gestures. Proficiency and gender had little effect on gestures. As personality has been shown to be an important factor in gesture production in mother tongue speakers, we expected to also see a similar result with foreign language speakers, this not being so suggests that other variables, aside from those tested, should be considered. In particular, the results suggest that emotion and emotional constructs, such as anxiety, self-confidence or empathy, might have a greater impact on gesture production than proficiency or personality, a point which should be taken into account especially in language evaluation contexts or professional contexts with second language speakers. [ABSTRACT FROM AUTHOR]

The aim of this study was to explore differences in gesture use during a free recall task among individuals of different ages and different development of conceptual thinking. Sixty-three children (aged 7–14 years) and 30 adults participated in an experiment in which they first witnessed a live staged event and a week later were interviewed about the event and tested using the Word Meaning Structure Test. Results showed that adults used more gesture types and showed higher gesture use frequency than children. Further, we found that word meaning structure and gesture use – specifically metaphoric and iconic gestures – predicted more accurate free recall in children. Our findings suggest that gestures may function differently during varying developmental stages, with gesture use being related to word meaning structure and the benefits of gesture use in free recall being more evident in children than adults. [ABSTRACT FROM AUTHOR]

The objective of this case study was to identify the role of finger gestures in learning mathematics informally during play, especially in sociocultural settings. A mathematical activity involving addition was qualitatively analyzed at a Japanese preschool. We explored how the process of subjectification and objectification contributed to a mathematical activity at a Japanese preschool and how the role of preschoolers' finger gestures changed during the process of learning mathematics. We utilized Radford's theoretical construct of joint labor and analyzed Japanese preschool children's mathematical behaviors from a sociocultural perspective. The subjects were 15 Japanese preschool children and their teacher. We relied on both Radford's methodology and a microgenetic approach for the analysis. We found that subjectification and objectification proceeded in the scene of the conversations regarding addition; observing joint labor in a classroom activity offered valuable insights into these processes. In the activity, the children actively extended their practice of posing and answering quizzes, and learned how to resolve a conflict with the support of the teacher. Secondly, although the role of finger gestures was originally used to obtain correct answers to quizzes, it was reconstructed to solve the conflict between the children through the teacher's mediation. This showed that, even in an environment where children implicitly learn mathematics, they learn from one another, including the teacher, and that gestures in mathematical communications function well for developing mathematical thinking and skills. [ABSTRACT FROM AUTHOR]

The present paper examines how English native speakers produce scopally ambiguous sentences and how they make use of gestures and prosody for disambiguation. As a case in point, the participants in the present study produced the English negative quantifiers. They appear in two different positions as (1) The election of no candidate was a surprise (a: ‘for those elected, none of them was a surprise’; b: ‘no candidate was elected, and that was a surprise’) and (2) no candidate’s election was a surprise (a: ‘for those elected, none of them was a surprise’; b: # ‘no candidate was elected, and that was a surprise.’ We were able to investigate the gesture production and the prosodic patterns of the positional effects (i.e., a-interpretation is available at two different positions in 1 and 2) and the interpretation effects (i.e., two different interpretations are available in the same position in 1). We discovered that the participants tended to launch more head shakes in the (a) interpretation despites the different positions, but more head nod/beat in the (b) interpretation. While there is not a difference in prosody of no in (a) and (b) interpretation in (1), there are pitch and durational differences between (a) interpretations in (1) and (2). This study points out the abstract similarities across languages such as Catalan and Spanish (Prieto et al. in Lingua 131:136–150, 2013. 10.1016/j.lingua.2013.02.008; Tubau et al. in Linguist Rev 32(1):115–142, 2015. 10.1515/tlr-2014-0016) in the gestural movements, and the meaning is crucial for gesture patterns. We emphasize that gesture patterns disambiguate ambiguous interpretation when prosody cannot do so. [ABSTRACT FROM AUTHOR]

Purpose. The purpose of this research is a comprehensive analysis and optimization of gesture and motion synchronization tools in virtual reality using web technologies. The study focuses on identifying the most effective methods for integrating VR solutions into web space to enhance the naturalness and intuitiveness of user interaction with virtual environments. Special attention is given to exploring the capabilities of modern web platforms such as React XR and React 360 for developing interactive web VR applications that provide precise and smooth gesture and motion control for users. Methodology. The research is based on a comparative analysis of the effectiveness of existing web libraries for creating VR experiences, such as React XR and React 360. Experimental methods are employed to test different approaches to synchronization of gestures and motions, including motion sensors, cameras, and machine learning algorithms. Technical aspects of VR application implementation are examined, including the use of Kalman filter for motion correction and reactive programming for dynamic virtual environment updates. Empirical data is collected through a series of experiments with interactive web VR applications to evaluate the effectiveness of different synchronization methods. Findings. The analysis shows that the use of React XR and React 360 libraries significantly improves the quality and speed of developing web VR applications. React XR, due to its free availability, open-source nature, and active developer community, has proven more effective for rapid integration of VR solutions into web space. The use of the Kalman filter ensures accurate correction of user movements, enabling the creation of more realistic and immersive experiences. Additionally, synchronization methods based on sensors and computer vision algorithms allow precise tracking and smooth transmission of user gestures and movements into the virtual environment. Originality. The research provides a systematic analysis of the effectiveness of using React XR and React 360 libraries for gesture and movement synchronization by defining efficiency criteria such as gesture processing speed, animation smoothness, and movement reproduction accuracy. The proposed synchronization methods, including the application of the Kalman filter and the Wiener filter, demonstrated high accuracy and efficiency in creating interactive and realistic VR experiences. The study also highlights the importance of selecting optimal web technologies to enhance the performance and usability of VR applications. Optimal web technologies in this context are determined by criteria such as data processing speed, animation smoothness, ease of integration, scalability, and developer support. Performance was assessed based on the system's response speed to user gestures and movements, which included data processing latency. Usability was analyzed based on the ease of integrating technologies into existing web applications and developer feedback on the ease of learning and using these libraries. Practical value. The research findings can be useful for developers working on creating interactive web-VR applications in various fields, including education, medicine, and e-commerce. Using React XR and React 360 allows for the rapid integration of VR technologies into the web space, enhancing the intuitiveness and naturalness of user interactions with virtual environments. The application of synchronization methods, such as the Kalman filter, ensures accurate and smooth transmission of user movements into the virtual environment, which is crucial for creating realistic and engaging VR experiences. [ABSTRACT FROM AUTHOR]

The aim of this study is to investigate Norwegian primary school students' multimodal mathematical reasoning when solving combinatorial problems. The data collection took place in four small groups of altogether thirteen 8–9 years old third-graders. Our study shows a variety of approaches used to solve the given combinatorial problems, such as count-all and grouping. These approaches were characterized by the students' use of inscriptions that displayed all combinations and inscriptions that did not display all combinations. Moreover, the students used gestures such as pointing and sliding. The students' multimodal reasoning was characterized by the ways their utterances, inscriptions, and gestures emerged and supplemented each other. The students' pointing gestures mediated the intended mathematical meaning solely when combined with inscriptions displaying all possible combinations. Sliding gestures, on the other hand, did mediate the intended mathematical meaning when their inscriptions were not displaying all possible combinations. In this latter case, the shortcomings of their inscriptions were complemented and compensated by the sliding gestures. The students' multimodal mathematical reasoning made explicit their combinatorial thinking, mediated the intended mathematical meaning, and facilitated their solving of the given combinatorial problems. [ABSTRACT FROM AUTHOR]

The article examines the philosophical and anthropological basis of non-verbal communication and its role in human communication. Non-verbal communication is considered as an integral part of human existence, in the context of which gestures, facial expressions, posture and body movements reflect the existential aspects of the individual, his inner state and worldview. The purpose of this study is to conceptualize the philosophical and anthropological basis of nonverbal communication and its role in human communication. Research methodology. Taking an interdisciplinary approach, the article examines cross-cultural differences, individual characteristics, and social contexts that are important for effective communication. The methods of phenomenological, axiological, hermeneutic, post-structuralist analysis are used, which contribute to deepening the issue of non-verbal communication as a social phenomenon and its role in human communication. According to phenomenological approaches, the perception of nonverbal signals is a subjective experience that is based on individual experiences and contextual conditions. Non-verbal communication can be considered as a text that needs to be interpreted. Understanding non-verbal signs requires knowledge of the context, cultural norms and individual characteristics of the philosophical and anthropological basis of non-verbal communication. Nonverbal communication is analyzed as human body language, which is a complex and multifaceted means of information transmission. This is a favorable asset for understanding a person's personality, which would contribute to a better understanding of others by adopting appropriate gestures, postures or physical expressions. The components of non-verbal communication and its role in human communication are disclosed. The results of the study can be useful for scientists, practitioners and politicians interested in understanding and improving the philosophical and anthropological basis of non-verbal communication and its role in human communication. The study of nonverbal cues helps to understand how different cultures use gestures, facial expressions, space, and body language to communicate. This provides an opportunity to study and compare cultural differences in communicative practices. Nonverbal communication is key to studying psychological aspects of human behavior, such as emotional intelligence, perception, and interaction between individuals [ABSTRACT FROM AUTHOR]

Manual co-speech gestures can facilitate language comprehension, especially in adverse listening conditions. However, we do not know whether gestures influence simultaneous interpreters' language comprehension in adverse listening conditions, and if so, whether this influence is modulated by interpreting experience, or by active simultaneous interpreting (SI). We exposed 24 interpreters and 24 bilinguals without interpreting experience to utterances with semantically related gestures, semantically unrelated gestures, or without gestures while engaging in comprehension (interpreters and bilinguals) or in SI (interpreters only). Tasks were administered in clear and noisy speech. Accuracy and reaction time were measured, and participants' gaze was tracked. During comprehension, semantically related gestures facilitated both groups' processing in noise. Facilitation was not modulated by interpreting experience. However, when interpreting noisy speech, interpreters did not benefit from gestures. This suggests that the comprehension component, and specifically crossmodal information processing, in SI differs from that of other language comprehension. [ABSTRACT FROM AUTHOR]

An intentional transfer of information is central to human communication. When comparing nonhuman primate communication systems to language, a critical challenge is to determine whether a signal is used in intentional, goal-oriented ways. As it is not possible to directly observe psychological states in any species, comparative researchers have inferred intentionality via behavioral markers derived from studies on prelinguistic human children. Recent efforts to increase consistency in nonhuman primate communication studies undervalue the effect of possible sources of bias: some behavioral markers are not generalizable across certain signal types (gestures, vocalizations, and facial expressions), contexts, settings, and species. Despite laudable attempts to operationalize first-order intentionality across signal types, a true "multimodal" approach requires integration across their sensory components (visual-silent, contact, audible), as a signal from a certain type can comprise more than one sensory component. Here we discuss how the study of intentional communication in nonlinguistic systems is hampered by issues of reliability, validity, consistency, and generalizability. We then highlight future research avenues that may help to understand the use of goal-oriented communication by opting, whenever possible, for reliable, valid, and consistent behavioral markers, but also taking into account sampling biases and integrating detailed observations of intraspecific communicative interactions. [ABSTRACT FROM AUTHOR]

Gestures are one of the ways in which mathematical cognition is embodied and have been elevated as a potentially important semiotic device in the teaching of mathematics. As such, a better understanding of gestures used during mathematics instruction (including frequency of use, types of gestures, how they are used, and the possible relationship between gestures and student performance) would inform mathematics education. We aim to understand teachers' gestures in the context of early algebra, particularly in the teaching of the equal sign. Our findings suggest that the equal sign is a relatively rich environment for gestures, which are used in a variety of ways. Participating teachers used gestures frequently to support their teaching about the equal sign. Furthermore, the use of gestures varied depending on the particular conception of the equal sign the instruction aimed to promote. Finally, teacher gesture use in this context is correlated with students' high performance on an early algebra assessment. [ABSTRACT FROM AUTHOR]

The aim of the article is to inventory gestures related to mockery, insult, attracting good luck, or warding off bad luck that a group of informants from Barcelona have performed. The data come from the application of the survey from the Atlas de Gestos, whose task is to collect gestures from the Pan-Hispanic context to describe the gestural repertoire of each territory and conduct comparative studies. The results, interpreted based on social factors (gender, age, and level of instruction), confirm the wide range of gestures for different functions (10 for mockery, 11 for insult and attracting good luck, and 9 for warding off bad luck). Regarding differences based on social variables, the results allow for indicating trends, namely, while the most used gesture to express mockery is sticking out the tongue, young people prefer the gesture of pointing and laughing, and informants over 55 years old only prefer laughing. Concerning insults, the use of peineta is widespread, although differences in behavior between men and women are observed. As for attracting good luck, Barcelona informants opt for crossing fingers, although older generations use gestures with more religious connotations, such as clasping hands. Finally, to ward off bad luck, Barcelona locals mention the gesture of crossing made with the index fingers of each hand. [ABSTRACT FROM AUTHOR]

Copyright of Annual of the General Union of Arab Archaeologists Studies on the Arab World Antiquities is the property of General Union of Arab Archaeologists and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In recent years, indoor or at-home sports have experienced significant growth. However, monotony is a common challenge in these static physical activities. Exergames, a genre of video games that combines physical activity and entertainment, have emerged as an attractive solution. Nevertheless, running on a treadmill and engaging in other activities simultaneously presents additional challenges. The balance and concentration required during running while interacting with a video game demand a special focus on the design of the Exergame. This paper presents a mobile Exergame designed specifically for treadmill running, utilizing interaction with a smartwatch. The game offers natural environments where, through smartwatch technology, it interprets the player's movements, transforming them into running speed and interactive actions by detecting gestures within the game. The main objective is to provide users with a satisfying gaming experience tailored to the characteristics of treadmill running. Particular emphasis has been placed on prioritizing the playful component of this Exergame, recognizing its relevance in the context of treadmill running. To evaluate the achievement of objectives and the proposed hypothesis, a comparative study was conducted between the proposed Exergame and a treadmill running simulator. Participants experienced both experiences and subsequently completed the Game Experience Questionnaire (GEQ), specifically the In-game GEQ version. The results obtained indicate that participants had a better gaming experience in the Exergame than in the simulator. These findings highlight the importance of prioritizing the playful component in Exergames and provide guidelines for future improvements and developments in the field. [ABSTRACT FROM AUTHOR]

Speech neuroprostheses have the potential to restore communication to people living with paralysis, but naturalistic speed and expressivity are elusive1. Here we use high-density surface recordings of the speech cortex in a clinical-trial participant with severe limb and vocal paralysis to achieve high-performance real-time decoding across three complementary speech-related output modalities: text, speech audio and facial-avatar animation. We trained and evaluated deep-learning models using neural data collected as the participant attempted to silently speak sentences. For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%. For speech audio, we demonstrate intelligible and rapid speech synthesis and personalization to the participants pre-injury voice. For facial-avatar animation, we demonstrate the control of virtual orofacial movements for speech and non-speech communicative gestures. The decoders reached high performance with less than two weeks of training. Our findings introduce a multimodal speech-neuroprosthetic approach that has substantial promise to restore full, embodied communication to people living with severe paralysis.

For sign languages, transitional movements of the hands are fully visible and may be used to predict upcoming linguistic input. We investigated whether and how deaf signers and hearing nonsigners use transitional information to detect a target item in a string of either pseudosigns or grooming gestures, as well as whether motor imagery ability was related to this skill. Transitional information between items was either intact (Normal videos), digitally altered such that the hands were selectively blurred (Blurred videos), or edited to only show the frame prior to the transition which was frozen for the entire transition period, removing all transitional information (Static videos). For both pseudosigns and gestures, signers and nonsigners had faster target detection times for Blurred than Static videos, indicating similar use of movement transition cues. For linguistic stimuli (pseudosigns), only signers made use of transitional handshape information, as evidenced by faster target detection times for Normal than Blurred videos. This result indicates that signers can use their linguistic knowledge to interpret transitional handshapes to predict the upcoming signal. Signers and nonsigners did not differ in motor imagery abilities, but only non-signers exhibited evidence of using motor imagery as a prediction strategy. Overall, these results suggest that signers use transitional movement and handshape cues to facilitate sign recognition.

BackgroundAphasia, resulting from acquired brain injury, disrupts language processing and usage, significantly impacting individuals’ social communication and life participation. Given the limitations of traditional assessments in capturing the nuanced challenges faced by individuals with aphasia, this study seeks to explore the potential benefits of integrating multimodal communication elements into discourse analysis to better capture narrative proficiency in this population.ObjectiveThis study examined how incorporating multimodal communication elements (e.g., physical gestures, writing, drawing) into discourse analysis may affect the narrative outcomes of persons with aphasia compared to those observed using methods that exclude multimodal considerations.MethodsParticipants included individuals with chronic aphasia and age-and education-matched healthy controls who completed a storytelling task—the Bear and the Fly story. Macrolinguistic scores were obtained using verbal-only and multimodal scoring approaches. Additionally, the frequency and type of multimodal communication use during storytelling were examined in relation to aphasia characteristics. Statistical analyses included both within-group and between-group comparisons as well as correlational analyses.ResultsIndividuals with aphasia scored significantly higher in terms of their macrolinguistic abilities when multimodal scoring was considered compared to verbal-only scoring. Within the aphasia group, there were prominent differences noted in macrolinguistic scores for both fluent and nonfluent aphasia. Specifically, both groups scored higher on Main Concepts when multimodal scoring was considered, with the nonfluent group demonstrating significantly higher Main Concept and total macrolinguistic rubric scores in multimodal scoring compared to verbal scoring on the storytelling task. Additionally, aphasia severity showed moderate positive correlations with total macrolinguistic scores, indicating that individuals with less severe aphasia tended to produce higher quality narratives. Lastly, although persons with aphasia used different types of nonverbal modalities (i.e., drawing, writing), the use of meaning-laden gestures was most predominant during storytelling, emphasizing the importance of multimodal elements in communication for individuals with aphasia.ConclusionOur preliminary study findings underscore the importance of considering multimodal communication in assessing discourse performance among individuals with aphasia. Tailoring assessment approaches based on aphasia subtypes can provide valuable insights into linguistic abilities and inform targeted intervention strategies for improving communication outcomes.

Previous studies on gestures and personality suggest an ambiguous picture of the effects of the various personality dimensions on the different types of speech-gestures and adaptors. In foreign language learning an additional variable to take into account is proficiency, which some studies have shown to affect gestures. In this study, we explore how various intrinsic variables, including personality, proficiency, gender and age affect the gestures of 61 Cantonese speakers of English as a second language. Participants were asked to narrate a video cartoon. Their proficiency and the frequency of gestures produced was based on those narrations. A functional categorization of gestures was followed, dividing them into semantic and discursive, and that also noted adaptors and micro-gestures, referred to as “flutters.” Personality was self-reported using the 44 question Big-Five inventory. Correlations and a series of generalized linear models were developed to explore the interplay between variables. Agreeableness was found to be positively correlated with semantic gestures; and neuroticism and age were negatively correlated with flutter duration. Contrary to the findings from previous studies, no significant relationships were found between neuroticism and adaptors or semantic gestures, nor between extraversion or openness and semantic gestures. Proficiency and gender had little effect on gestures. As personality has been shown to be an important factor in gesture production in mother tongue speakers, we expected to also see a similar result with foreign language speakers, this not being so suggests that other variables, aside from those tested, should be considered. In particular, the results suggest that emotion and emotional constructs, such as anxiety, self-confidence or empathy, might have a greater impact on gesture production than proficiency or personality, a point which should be taken into account especially in language evaluation contexts or professional contexts with second language speakers.

This study builds on the increasing evidence that the multimodal nature of adult-child interactions and the use of objects play an important role in early linguistic development. Most of these studies analyzed dyadic interactions at home, whereas few research has been conducted in early childhood education and care settings. In this paper, we characterized the multimodal nature of teachers’ communicative bids during classroom-based group interactions in nursery schools. Observational data of circle-time activities was collected from 16 Spanish nursery school classrooms, comprising 16 teachers and 161 children between two and three years of age. We analyzed teachers’ communicative bids (i.e., verbal utterances and verbal-gestural bids) considering the frequency of use of different types of gestures, to whom are they addressed (i.e., the whole group or a single child), the extent to which they involve the use of objects, the classroom layout, and the relationship between the communicative bids and the number of children that participated in each classroom. Teachers’ communication with toddlers is highly multimodal and rely on different types of gestures, although the use of objects in our sample was scarce. Descriptive analysis suggest that certain classroom layouts may favor teachers’ use of some types of gestures over others. In this article, we discuss the implications of both the use of objects and space for understanding how adults shape the linguistic contexts of young children, and the potential opportunities and limitations they pose for classroom interactions.

Background The consensus in scientific literature is that each child undergoes a unique linguistic development path, albeit with shared developmental stages. Some children excel or lag behind their peers in language skills. Consequently, a key challenge in language acquisition research is pinpointing factors influencing individual differences in language development. Methods We observed children longitudinally from 3 to 24 months of life to explore early predictors of vocabulary size. Based on the productive vocabulary size of children at 24 months, 30 children met our sample selection criteria: 10 late talkers and 10 early talkers, and we compared them with 10 typical talkers. We evaluated interactive behaviors at 3, 6, 9 and 12 months, considering vocal production, gaze at mother’s face, and gestural production during mother-child interactions, and we considered mothers’ report of children’s actions and gestures and receptive-vocabulary size at 15 and 18 months. Results Results indicated early precursors of language outcome at 24 months identifiable as early as 3 months in vocal productions, 6 months for gaze at mother’s face and 12 months for gestural productions. Conclusions Our research highlights both theoretical and practical implications. Theoretically, identifying the early indicators of belonging to the group of late or early talkers underscores the significant role of this developmental period for future studies. On a practical note, our findings emphasize the crucial need for early investigations to identify predictors of vocabulary development before the typical age at which lexical delay is identified.

BackgroundAimsMethods & ProceduresOutcomes & ResultsConclusionsResearch on aphasia treatments that directly target conversation skills have been on the rise in the recent past. Despite the ubiquitous nature of gestures in everyday conversations and their possible compensatory and/or supportive functions for adults with aphasia, the conversational treatment outcome measures continue to rely on spoken language alone.We aimed to explore the feasibility of using co-speech hand gestures as a treatment outcome measure in Clyde, an individual with chronic and severe Wernicke’s aphasia.Using a multiple-probe single-case experimental research, we examined data from one participant with aphasia (Clyde) who was a part of a larger study involving 3 people with aphasia (Devanga, 2020). 10-minute conversations between Clyde and clinician, and Clyde and his spouse, were completed during 3 baseline probes, 3 treatment probes and 3 maintenance probes. Clyde also completed a 15-session Collaborative Referencing Intervention (CRI; Devanga etal., 2021) with the clinician. The 18 conversation samples were videorecorded, transcribed, and analysed.The outcomes analysed included overall gesture frequency and type (i.e., referential vs. non-referential) across sessions (baseline – treatment – maintenance) and partner familiarity (clinician – spouse). We also compared the gesture outcomes with the spoken language measure, CIUconv (Correct Information Units). Overall gesture frequency was higher with the clinician than the spouse, and the frequency of referential gestures were higher than the non-referential gestures with both partners. With the onset of treatment, we found that overall gesture frequency decreased and #CIUconv increased with the clinician, however there were minimal variations across phases with the spouse. Additionally, with the onset of treatment, referential gestures decreased, and non-referential gestures increased with both partners.Findings from this exploratory study provide preliminary support to using co-speech hand gestures as treatment outcome measures for aphasia. Clinical implications including comparisons of gestural outcomes with spoken language outcomes and their implementation are discussed. Further research on differences in gesture use among aphasia types and severities and patterns of change in gesture use with aphasia treatment is warranted. [ABSTRACT FROM AUTHOR]

This study builds on the increasing evidence that the multimodal nature of adult-child interactions and the use of objects play an important role in early linguistic development. Most of these studies analyzed dyadic interactions at home, whereas few research has been conducted in early childhood education and care settings. In this paper, we characterized the multimodal nature of teachers' communicative bids during classroom-based group interactions in nursery schools. Observational data of circle-time activities was collected from 16 Spanish nursery school classrooms, comprising 16 teachers and 161 children between two and three years of age. We analyzed teachers' communicative bids (i.e., verbal utterances and verbal-gestural bids) considering the frequency of use of different types of gestures, to whom are they addressed (i.e., the whole group or a single child), the extent to which they involve the use of objects, the classroom layout, and the relationship between the communicative bids and the number of children that participated in each classroom. Teachers' communication with toddlers is highly multimodal and rely on different types of gestures, although the use of objects in our sample was scarce. Descriptive analysis suggest that certain classroom layouts may favor teachers' use of some types of gestures over others. In this article, we discuss the implications of both the use of objects and space for understanding how adults shape the linguistic contexts of young children, and the potential opportunities and limitations they pose for classroom interactions. [ABSTRACT FROM AUTHOR]

This research paper delves into the significant role that gestures play in the context of service recovery tactics, exploring how they affect brand perception, customer happiness, and loyalty development. This study embarks on a journey to uncover their profound influence on forming customer perceptions after instances of service failure by engaging in a thorough exploration of a variety of gestures, including polite interactions, compassionate responses, active listening, and efficient service delivery. The study carefully examines the nuanced ways that various gestures affect emotional regulation, drive customer satisfaction levels, affect retention rates, direct online suggestions, and word-of-mouth referrals by using a meticulously designed approach. Customer satisfaction plays a crucial part in the inquiry which is considered as a key element that shapes repurchase intentions as well as loyalty. The study reveals particular behaviours through an empirical analysis that directly increase customer satisfaction, such as politeness and nice interactions, demonstrating their significant ability to improve the entire service recovery experience. Parallel to this, the research reveals deeper implications, showing how actions like empathy and engaged listening may develop good impressions through other channels even when they don't directly affect consumer satisfaction. This study provides organizations with practical knowledge to modify the service restoration paradigms by illuminating the nuanced subtleties of gesture dynamics in service recovery. Utilizing the consequences of this research, businesses may respond to service lapses with precision, tailoring their actions to the requirements of their customers. This comprehensive understanding equips companies to plan memorable client encounters, build long-lasting bonds, and solidify their positions in a cutthroat industry. [ABSTRACT FROM AUTHOR]

Communication for individuals who are hearing- and speech-impaired, commonly referred to as the deaf and mute community, heavily relies on sign language as their primary mode of expression. This study presents a novel framework leveraging image processing techniques for the detection and recognition of sign language gestures. The developed software offers promising avenues for enhancing comprehension of Sign Language, with potential applications in educational settings, public spaces, and interpersonal interactions. The proposed method streamlines the recognition process of sign language, employing deep learning algorithms for the accurate prediction of signs. The system operates by processing input images containing signs through a convolutional neural network, encompassing stages such as pre-processing, feature extraction, model training, testing, and sign-to-text conversion. Crucially, the system's output provides text-based descriptions of the sign in the input image and notably integrates voice output for enhanced accessibility and communication. This multifaceted approach contributes towards bridging communication barriers between individuals with disabilities and those without, promoting inclusivity and understanding in diverse social contexts. [ABSTRACT FROM AUTHOR]

The purpose of this study is to present the state of the art of recent research on nonverbal communication in L2 classroom interaction, in particular on teachers' and students' gestures, and then focus on a case of gestures in an L2 Italian classroom. A corpus of video-recorded interactions (CLODIS) were analyzed to answer the following research question: How do L2 Italian native teachers behave when addressing international students? Are there differences with what has been observed in other foreign language (L2) teaching contexts? Both previous data-based research on multimodality in L2 classes and the analysis on CLODIS show that teachers select and coordinate multiple semiotic modes as interactional resources to complete various teaching tasks. Furthermore, Italian native teachers use not only the typical pedagogical gestures (both iconic and metaphorical), but also culturally specific emblems that may cause misunderstandings or inappropriate mirroring effects. For these reasons, it is important that L2 teachers develop a good multimodal awareness, especially if they teach their mother tongue to foreign students and if they belong to a "contact culture", as is the case observed in L2 Italian classes. [ABSTRACT FROM AUTHOR]

Sketching is a common way to depict design ideas freely. Augmented reality technology extends the human–computer interaction space from the screen to the real world, providing a new application space for 3D sketching. However, due to the difficulty of depth perception, it is difficult for sketching strokes to maintain good planar and spatial consistency, making drawing 3D sketches in AR challenging. In this article, we propose FMASketch—a method of drawing mid-air sketches similar to building blocks. The auxiliary surfaces can ensure that the strokes drawn by users can always stay on the same plane as drawing in 2D space. Gestures are used to stitch together multiple drawing planes, and the strokes on the planes form a 3D sketch, which is very similar to how 3D modeling is done on the desktop by drawing different views and assembling them. Experiments have verified the effectiveness of the auxiliary surface generation method and gesture operations in this article. The mid-air sketching method is easy to learn and provides a new idea for rapid modeling in AR scenarios. [ABSTRACT FROM AUTHOR]

The objective of this study was at analyzing the action schemes of two young deaf students in combinatorial situations, based on the scheme construct of Theory of Conceptual Fields by Gérard Vergnaud, in Brazilian Sign Language (Libras) and communicative and cognitive potential of gestures. This is a qualitative approach in design of a descriptive case study, based on a micro genetic analysis associated with videography. The deaf students mobilized schemes in space, associating gestures and signs in Libras, expressing the concepts of addition and one- to-one correspondence (finger-sign). The linguistic choices by the Libras interpreter influenced the resolution of each deaf person. Detailed description of the schemes mobilized by the deaf students may broaden the understanding of their stage of development and benefit mathematics teacher in teaching this concept in an inclusive educational context. [ABSTRACT FROM AUTHOR]

Imitation is a social learning mechanism in humans but its relevance in nonhuman primates is controversial. Studies have suggested that some ape species, such chimpanzees and orangutans, may be able to imitate. However, records of true imitation in gorillas are scarce. This study was designed to evaluate the imitation ability of two female western lowland gorillas (Gorilla gorilla gorilla) by using the "do as I do" method, after 15 months of training. For the final evaluation, we tested the gorillas with 52 novel actions classified in four categories (Gestures, Object, Object-Object, and Object-Subject). To assess the difficulty of the novel actions, the lead researcher and two independent observers scored the responses from 0 to 3 based on how well they were done. During the training period, the two gorillas obtained similar success rates for imitation. During the evaluation, they achieved true imitation in 46% of transparent actions and 58% of the opaque actions. We identified "Gestures" as the easiest actions for one gorilla and "Object-Subject" for the other. "Object-Object" were the most difficult actions for both gorillas. Our findings show that female western lowland gorillas can imitate after a training period. [ABSTRACT FROM AUTHOR]

We describe how an instructor integrated embodiment to teach the Fundamental Homomorphism Theorem (FHT) and preliminary concepts in an undergraduate abstract algebra course. The instructor's use of embodiment reduced levels of abstraction for formal definitions, theorems, and proofs. The instructor's simultaneous use of various forms of embodiment primed students for the formalism and symbolism, highlighted and disambiguated students' referents, amplified students' contributions to develop conceptual fluency, and linked students' body form catchments to motivate the FHT. Our results offer practical implications for teaching by illustrating examples of how embodiment can assist in making abstract concepts concrete. [ABSTRACT FROM AUTHOR]

The emergence of symbolic gestures is a solid milestone in early childhood development. Interventions that intentionally promote them have contributed to children's language, cognitive, and socioemotional development. However, these studies have mainly been conducted in the USA with middle-SES families, and such research has considerably decreased in recent years. This study aimed to assess the effects of an intentional intervention for promoting symbolic gestures in the expressive and comprehensive language of Chilean children who were aged 18 months. Sixty-nine highly psychosocially at-risk children were assessed at 5–9 months and then at 18 months. Teachers from half of the nurseries involved in the study received the intervention. The assessment included a report on the children's gestural vocabulary, the CDI, and the language scale of BSID-III. The results showed that the children in the intervention group had a significantly greater gestural vocabulary at 18 months and they performed better in their expressive language than the children in the control group did. Additionally, this study aimed to analyze if this intervention affected children differentially in consideration of their language development (adequate and at-risk). The results showed that children with adequate development improved their language when they received the intervention, but those from the at-risk group did not. The implications of these results for the design of interventions at an early age are discussed while considering children from different sociocultural backgrounds and with different language development. [ABSTRACT FROM AUTHOR]

This exploratory study aims to investigate the co-occurrence of Catalan pragmatic markers that fulfill an interactive function and multimodal cues, such as manual gestures, adaptors, head gestures, and eye gaze. To do so, we utilized spontaneous face-to-face conversations between ten different dyads, all of whom were undergraduate students discussing movies that they had recently seen. We processed the data with descriptive quantitative statistics, specifically employing exploratory multifactorial statistical visualization techniques, including multiple correspondence analysis (MCA) and cluster analysis. The results showed that pragmatic markers generally co-occur with multimodal cues. Although the multimodal system works independently, when pragmatic markers are uttered, both systems tend to harmonize in accordance with general communicative tasks, particularly turn management, alignment, and social affiliation. [ABSTRACT FROM AUTHOR]

Sign language is a complex language that uses hand gestures, body movements, and facial expressions and is majorly used by the deaf community. Sign language recognition (SLR) is a popular research domain as it provides an efficient and reliable solution to bridge the communication gap between people who are hard of hearing and those with good hearing. Recognizing isolated sign language words from video is a challenging research area in computer vision. This paper proposes a hybrid SLR framework that combines a convolutional neural network (CNN) and an attention-based long-short-term memory (LSTM) neural network. We used MobileNetV2 as a backbone model due to its lightweight structure, which reduces the complexity of the model architecture for deriving meaningful features from the video frame sequence. The spatial features are fed to LSTM optimized with an attention mechanism to select the significant gesture cues from the video frames and focus on salient features from the sequential data. The proposed method is evaluated on a benchmark WLASL dataset with 100 classes based on precision, recall, F1-score, and 5-fold cross-validation metrics. Our methodology acquired an average accuracy of 84.65%. The experiment results illustrate that our model performed effectively and computationally efficiently compared to other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper draws on Merleau-Ponty's perspective on intercorporeality to inquire into the role of the body in virtual collaboration. Merleau-Ponty redefines the human body beyond pure physicality, as material-semiotic processes between self and other. From this perspective, meaningful engagements are based on the intertwinement of bodily activities including gestures and displays. Drawing on a rich case of an exclusively virtual collaboration, this paper illuminates how bodily expressivity is rematerialised through digital technology. The findings show how distributed bodies pattern an intercorporeal armature constituted of processes of co-presencing, co-orienting, and co-investing. The paper problematises claims of virtual disembodiment and advances current understanding of how body and technology merge to make virtual collaboration possible. The rich empirical insights are synthesised into four main theoretical propositions that constitute a novel perspective on virtual intercorporeality. This perspective advances our understanding of virtual embodiment and elaborates the role of technology and language in the process of mutual attunement between bodies. [ABSTRACT FROM AUTHOR]

Although adults are known to play an important role in young children's development, little work has focused on the enactive features of scaffolding in informal learning settings, and the embodied dynamics of intergenerational interaction. To address this gap, this paper undertakes a microinteractional analysis to examine intergenerational collaborative interaction in a science museum setting. The paper presents a fine‐grained moment‐by‐moment analysis of video‐recorded interaction of children and their adult carers around science‐themed objects. Taking an enactive cognition perspective, the analysis enables access to subtle shifts in interactants' perception, action, gesture, and movement to examine how young children engage with exhibits, and the role adult action plays in supporting young children's engagement with exhibits and developing ideas about science. Our findings demonstrate that intergenerational "embodied scaffolding" is instrumental in making "enactive potentialities" in the environment more accessible for children, thus deepening and enriching children's engagement with science. Adult action is central to revealing scientific dimensions of objects' interaction and relationships in ways that expose novel types of perception and action opportunities in shaping science experiences and meaning making. This has implications for science education practices since it foregrounds not only "doing" science, through active hands‐on activities, but also speaks to the interconnectedness between senses and the role of the body in thinking. Drawing on the findings, this paper also offers design implications for informal science learning environments. [ABSTRACT FROM AUTHOR]