Search

This study describes data on auditory-visual integration and visually-guided adaptation of auditory distance perception using the ventriloquism effect (VE) and ventriloquism aftereffect (VAE). In an experiment, participants judged egocentric distance of interleaved auditory or auditory-visual stimuli with the auditory component located from 0.7 to 2.04 m in front of listeners in a real reverberant environment. The visual component of auditory-visual stimuli was displaced 30% closer (V-closer), 30% farther (V-farther), or aligned (V-aligned) with respect to the auditory component. The VE and VAE were measured in auditory and auditory-visual trials, respectively. Both effects were approximately independent of target distance when expressed in logarithmic units. The VE strength, defined as a difference of V-misaligned and V-aligned response bias, was approximately 72% of the auditory-visual disparity regardless of the visual-displacement direction, while the VAE was stronger in the V-farther (44%) than the V-closer (31%) condition. The VAE persisted to post-adaptation auditory-only blocks of trials, although it was diminished. The rates of build-up/break-down of the VAE were asymmetrical, with slower adaptation in the V-closer condition. These results suggest that auditory-visual distance integration is independent of the direction of induced shift, while the re-calibration is stronger and faster when evoked by more distant visual stimuli.

The main purpose of this research is to determine the impact of SAVI(Somatic ,Auditory ,Visual and Intellectual approach) in improving vocabulary, listening and creative thinking skills of elementary stage Pupils .Three statistical hypotheses were set to guide this research. The research adopted the one treatment group design to examine the research hypotheses. The participants consisted of (N=40) fourth year primary stage pupils at Dawaa school in Beni suef .The instruments of the study represent in a pre-post vocabulary test ,a pre-post listening test, and a pre-post creative thinking test . The research results revealed that SAVI has an impact on improving vocabulary, Listening and creative thinking skills of elementary stage pupils. This research has implications both for researchers conducting classroom-research and language teachers.

The perceptual experiment reported in this article explored whether the communication of five pairs of timbral intentions (bright/dark, heavy/light, round/sharp, tense/relaxed, and dry/velvety) between pianists and listeners is reliable and the extent to which performers' gestures provide visual cues that influence the perceived timbre. Three pianists played three musical excerpts with 10 different timbral intentions (3 × 10 = 30 music stimuli) and 21 piano students were asked to rate perceived timbral qualities on both unipolar Likert scales and non-verbal sensory scales (shape, size, and brightness) under three modes (vision-alone, audio-alone, and audio-visual). The results revealed that nine of the timbral intentions were reliably communicated between the pianists and the listeners, except for the dark timbre. The communication of tense and relaxed timbres was improved by the visual conditions regardless of who is performing; for the rest, we found the individuality in each pianist's preference for using visual cues. The results also revealed a strong cross-modal association between timbre and shape. This study implies that the communication of piano timbre is not based on acoustic cues alone but relates to a shared understanding of sensorimotor experiences between the performers and the listeners.

Visual cues facilitate speech perception during face-to-face communication, particularly in noisy environments. These visual-driven enhancements arise from both automatic lip-reading behaviors and attentional tuning to auditory-visual signals. However, in crowded settings, such as a cocktail party, how do we accurately bind the correct voice to the correct face, enabling the benefit of visual cues on speech perception processes? Previous research has emphasized that spatial and temporal alignment of the auditory-visual signals determines which voice is integrated with which speaking face. Here, we present a novel illusion demonstrating that when multiple faces and voices are presented in the presence of ambiguous temporal and spatial information as to which pairs of auditory-visual signals should be integrated, our perceptual system relies on identity information extracted from each signal to determine pairings. Data from three experiments demonstrate that expectations about an individual’s voice (based on their identity) can change where individuals perceive that voice to arise from.

The research on the restorative capacity of soundscape has grown rapidly in recent decades. Most literature explores this topic using the single medium of soundscape. Their findings perhaps do not provide a cohesive guideline for practical design because a visitor perceives the landscape using the pattern of multi-sensory input, such as visual, auditory, tactile and olfactory. To fill this gap this study conducted an experiment, in which 20 photographs were combined with five sounds, and Short-version Revised Restoration Scale was used to measure the restorative quality of auditory-visual combinations evaluated by 382 college students. The results indicated that: (1) easy accessibility to waterscape and flat topography normally are the promoters for the restorative quality of silent landscape; (2) a landscape containing natural water and a high coverage of plants matches the visual association of a bird singing, adding the birdsong to this landscape will produce higher restorative potential; (3) introducing flowing water sound to the landscape with less still water is a better choice to improve the restorative capacity; (4) adding wind sound to the landscape possessing high coverage of vegetation increases the restorative quality; (5) adding traditional Chinese music to the landscape with less paved areas will promote the mental stress relief of users, and no results were found for the western music.

This research was motivated by the finding that the learning process between teachers and students occurred in one way directions. It was also initiated by the lack of learning that build students' abilities, skills, and creativity. By assisting the teachers, it was hoped that they would be able to design effective learning in order to make students active in the learning process, one of which was somatic, auditory, visual, and intellectual (SAVI) learning. This research was a school action research involving stages of planning, implementing, observing, and reflecting. This research was conducted to 6 teachers of SDN 009 Teluk Kiambang. The instrument used was the observation sheet. The results showed that in the first cycle the teacher's ability was 65.83 with Sufficient category while in the second cycle, it increased to 79.16, which was in Good category. Based on the results of the study, it was concluded that the ability of teachers at SDN 009 Teluk Kiambang to develop SAVI learning model increased after getting the assistance from the principal as a researcher.

This study aimed at comparing preschool children’s learning of relations between pseudowords and pictures in two teaching conditions: shared storybook reading (implicit learning) and auditory-visual pairing (word pictures) with echoic request (explicit learning). We conducted two studies with children aged 4 to 5 years that were divided into two groups. In the repeated reading condition, we read one non-commercial story twice in a row. In the pairing with echoic condition, the children were exposed to auditory-visual pairing of pseudowords and pictures, and asked to repeat the target word (echoic behavior). Two new word-figure relations were taught in Study 1, and three relations in Study 2. After the presentation of the conditions and 1 week later, exclusion, matching-to-sample, naming, and generalization probes were performed. No differences were found between the two experimental conditions in Study 1. In Study 2, the pairing condition produced better results than the reading condition. We discuss the conditions necessary for learning novel words.

Some classical studies on temporal order judgments (TOJ) suggested a single central process comparing stimulus onsets across modalities. The prevalent current view suggests that there is modality-specific timing estimation followed by a cross-modal stage. If the latter view is correct, TOJ's may vary depending on stimulus modality. Further, if TOJ is based only on onsets, stimulus duration should be irrelevant. To address these issues, we used both unisensory and multisensory stimuli to test whether unisensory duration processing influences cross-modal TOJ's. The stimuli were auditory noise bursts, visual squares, and their cross-modal combinations presented at 10, 40 and 500 ms durations, and various stimulus onset asynchronies. Psychometric functions were measured with an identical task in all conditions: On each trial, two stimuli were presented, one to the left, the other to the right of fixation. The participants judged which one started first. TOJ's were little affected by stimulus duration, implying that they are mainly determined by stimulus onsets. Throughout, the cross-modal just noticeable differences were larger than the unisensory ones. In accordance with the current view, our results suggest that cross-modal TOJ's require a comparison of timing after modality-specific estimations.

Cross-language McGurk Effects are used to investigate the locus of auditory–visual speech integration. Experiment 1 uses the fact that [], as in ‘sing’, is phonotactically legal in word-final position in English and Thai, but in word-initial position only in Thai. English and Thai language participants were tested for ‘n’ perception from auditory [m]/visual [] (A[m]V[]) in word-initial and -final positions. Despite English speakers’ native language bias to label word-initial [] as ‘n’, the incidence of ‘n’ percepts to A[m]V[] was equivalent for English and Thai speakers in final and initial positions. Experiment 2 used the facts that (i) [ð] as in ‘that’ is not present in Japanese, and (ii) English speakers respond more often with ‘tha’ than ‘da’ to A[ba]V[ga], but more often with ‘di’ than ‘thi’ to A[bi]V[gi]. English and three groups of Japanese language participants (Beginner, Intermediate, Advanced English knowledge) were presented with A[ba]V[ga] and A[bi]V[gi] by an English (Experiment 2a) or a Japanese (Experiment 2b) speaker. Despite Japanese participants’ native language bias to perceive ‘d’ more often than ‘th’, the four groups showed a similar phonetic level effect of [a]/[i] vowel context × ‘th’ vs. ‘d’ responses to A[b]V[g] presentations. In Experiment 2b this phonetic level interaction held, but was more one-sided as very few ‘th’ responses were evident, even in Australian English participants. Results are discussed in terms of a phonetic plus postcategorical model, in which incoming auditory and visual information is integrated at a phonetic level, after which there are post-categorical phonemic influences.

The low level of student creativity is a reflection of the unsuccessful learning process. Thus, it needs a special treatment to solve it. The study aims to increase student creativity using the SAVI method for 4th semester students of the Mathematic Education Study Program. This research is a classroom action research. Research with a focus on increasing student creativity using the SAVI method was conducted in semester 4 of the Mathematic Education Study Program, with 44 students. This research was conducted in 3 cycles. After observing and evaluating in 3 cycles with the results mentioned above, it can be concluded that the SAVI learning approach can increase the creativity of students of the UMP FKIP Mathematic Education Study Program. It is proven that in cycle I the average response is 33.71%, in cycle II the average response is 49.3%, and in cycle III the average response is 64.9%. With an increase from cycle I to cycle II of 15.59% and from cycle II to cycle III of 15.6%. From the result it can be said that SAVI was effective to increase students creativity in mathematic.

Insects have the ability to navigate through a complex environment. It has been reported that using multisensory information improves their ability to navigate. To reveal when and how insects utilize multiple sensory information, we developed a multisensory virtual reality system for crickets (Gryllus bimaculatus), which enables us to manipulate auditory and visual stimuli from every direction. We investigated how visual information influenced their phonotaxis towards a conspecific male calling song. Our results showed that crickets exhibit phonotaxis under virtual reality and that a visual target with a sound source reduced the variability of the body angle towards the source. These results suggested the possibility that crickets utilize visual cues to localize sound sources. Further investigation with our system will reveal how crickets effectively utilize multiple sensory information in complex environments.

Simple Summary Comparative investigations on infants’ and dogs’ social and communicative skills revealed striking similarity, which can be attributed to convergent evolutionary and domestication processes. Using a suitable experimental method that allows systematic and direct comparisons of dogs and humans is essential. In the current study, we used non-invasive eye-tracking technology in order to investigate looking behaviour of dogs and human infants in an auditory–visual matching task. We found a similar gazing pattern in the two species when they were presented with pictures and vocalisations of a dog and a female human, that is, both dogs and infants looked longer at the dog portrait during the dog’s bark, while matching human speech with the human face was less obvious. Our results suggested different mechanisms underlying this analogous behaviour and highlighted the importance of future investigations into cross-modal cognition in dogs and humans. Abstract We tested whether dogs and 14–16-month-old infants are able to integrate intersensory information when presented with conspecific and heterospecific faces and vocalisations. The looking behaviour of dogs and infants was recorded with a non-invasive eye-tracking technique while they were concurrently presented with a dog and a female human portrait accompanied with acoustic stimuli of female human speech and a dog’s bark. Dogs showed evidence of both con- and heterospecific intermodal matching, while infants’ looking preferences indicated effective auditory–visual matching only when presented with the audio and visual stimuli of the non-conspecifics. The results of the present study provided further evidence that domestic dogs and human infants have similar socio-cognitive skills and highlighted the importance of comparative examinations on intermodal perception.

This paper proposes a novel approach to assess audiovisual integration for both congruent and incongruent speech stimuli using reaction times (RT). The experiments are based on the McGurk effect, in which a listener is presented with incongruent audiovisual speech signals. A typical example involves the auditory consonant/b/combined with a visually articulated/g/, often yielding a perception of/d/. We quantify the amount of integration relative to the predictions of a parallel independent model as a function of attention and congruency between auditory and visual signals. We assessed RT distributions for congruent and incongruent auditory and visual signals in a within-subjects signal detection paradigm under conditions of divided versus focused attention. Results showed that listeners often received only minimal benefit from congruent auditory visual stimuli, even when such information could have improved performance. Incongruent stimuli adversely affected performance in divided and focused attention conditions. Our findings support a parallel model of auditory-visual integration with interactions between auditory and visual channels.

This study aimed to investigate how individuals with bipolar disorder integrate auditory and visual speech information compared to healthy individuals. Furthermore, we wanted to see whether there were any differences between manic and depressive episode bipolar disorder patients with respect to auditory and visual speech integration. It was hypothesized that the bipolar group&rsquo, s auditory&ndash, visual speech integration would be weaker than that of the control group. Further, it was predicted that those in the manic phase of bipolar disorder would integrate visual speech information more robustly than their depressive phase counterparts. To examine these predictions, a McGurk effect paradigm with an identification task was used with typical auditory&ndash, visual (AV) speech stimuli. Additionally, auditory-only (AO) and visual-only (VO, lip-reading) speech perceptions were also tested. The dependent variable for the AV stimuli was the amount of visual speech influence. The dependent variables for AO and VO stimuli were accurate modality-based responses. Results showed that the disordered and control groups did not differ in AV speech integration and AO speech perception. However, there was a striking difference in favour of the healthy group with respect to the VO stimuli. The results suggest the need for further research whereby both behavioural and physiological data are collected simultaneously. This will help us understand the full dynamics of how auditory and visual speech information are integrated in people with bipolar disorder.

Author(s): Gabouer, Allison; Oghalai, John; Bortfeld, Heather | Abstract: Parent-child dyads in which the child is deaf but the parent is hearing present a unique opportunity to examine parents’ use of non-auditory cues, particularly vision and touch, to establish communicative intent. This study examines the multimodal communication patterns of hearing parents during a free play task with their hearing (N=9) or deaf (N=9) children. Specifically, we coded parents’ use of multimodal cues in the service of establishing joint attention with their children. Dyad types were compared for overall use of multimodal – auditory, visual, and tactile – attention-establishing cues, and for the overall number of successful and failed bids by a parent for a child’s attention. The relationship between multimodal behaviors on the part of the parent were tracked for whether they resulted in successful or failed initiation of joint attention. We focus our interpretation of the results on how hearing parents differentially accommodate their hearing and deaf children to engage them in joint attention. Findings can inform the development of recommendations for hearing parents of deaf children who are candidates for cochlear implantation regarding communication strategies to use prior to a child’s implantation. Moreover, these findings expand our understanding of how joint attention is established between parents and their preverbal children, regardless of children’s hearing status.

We continuously receive the external information from multiple sensors simultaneously. The brain must judge a source event of these sensory informations and integrate them. It is thought that judging the simultaneity of such multisensory stimuli is an important cue when we discriminate whether the stimuli are derived from one event or not. Although previous studies have investigated the correspondence between an auditory-visual (AV) simultaneity perceptions and the neural responses, there are still few studies of this. Electrophysiological studies have reported that ongoing oscillations in human cortex affect perception. Especially, the phase resetting of ongoing oscillations has been examined as it plays an important role in multisensory integration. The aim of this study was to investigate the relationship of phase resetting for the judgment of AV simultaneity judgement tasks. The subjects were successively presented with auditory and visual stimuli with intervals that were controlled as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm{SOA_{50\%}}$$\end{document}SOA50% and they were asked to report whether they perceived them simultaneously or not. We investigated the effects of the phase of ongoing oscillations on simultaneity judgments with AV stimuli with SOAs in which the detection rate of asynchrony was 50 %. It was found that phase resetting at the beta frequency band in the brain area that related to the modality of the following stimulus occurred after preceding stimulus onset only when the subjects perceived AV stimuli as simultaneous. This result suggested that beta phase resetting occurred in areas that are related to the subsequent stimulus, supporting perception multisensory stimuli as simultaneous.

Conditionally automated driving systems may soon be available on the market. Even though these systems exempt drivers from the driving task for extended periods of time, drivers are expected to take back control when the automation issues a so-called take-over request. This study investigated the interaction between take-over request modality and type of non-driving task, regarding the driver's reaction time. It was hypothesized that reaction times are higher when the non-driving task and the take-over request use the same modality. For example, auditory take-over requests were expected to be relatively ineffective in situations in which the driver is making a phone call. 101 participants, divided into three groups, performed one of three non-driving tasks, namely reading (i.e., visual task), calling (auditory task), or watching a video (visual/auditory task). Results showed that auditory and tactile take-over requests yielded overall faster reactions than visual take-over requests. The expected interaction between takeover modality and the dominant modality of the non-driving task was not found. As for self-reported usefulness, auditory and tactile take-over requests yielded higher scores than visual ones. In conclusion, it seems that auditory and tactile stimuli are equally effective as take-over requests, regardless of the non-driving task. Further study into the effects of realistic non-driving tasks is needed to identify which non-driving tasks are detrimental to safety in automated driving.

The human superior temporal sulcus (STS) is responsive to visual and auditory information, including sounds and facial cues during speech recognition. We investigated the functional organization of STS with respect to modality-specific and multimodal speech representations. Twenty younger adult participants were instructed to perform an oddball detection task and were presented with auditory, visual, and audiovisual speech stimuli, as well as auditory and visual nonspeech control stimuli in a block fMRI design. Consistent with a hypothesized anterior-posterior processing gradient in STS, auditory, visual, and audiovisual stimuli produced the largest BOLD effects in anterior, posterior, and middle STS, respectively, based on whole-brain, linear mixed effects, and principle component analyses. Notably, the middle STS exhibited preferential responses to multisensory stimulation, as well as speech compared to nonspeech. Within the mid-posterior and middle STS regions, response preferences changed gradually from visual, to multisensory, to auditory moving posterior to anterior. Post-hoc analysis of visual regions in the posterior STS revealed that a single subregion bordering the middle STS was insensitive to differences in low-level motion kinematics yet distinguished between visual speech and nonspeech based on multi-voxel activation patterns. These results suggest that auditory and visual speech representations are elaborated gradually within anterior and posterior processing streams, respectively, and may be integrated within the middle STS, which is sensitive to more abstract speech information within and across presentation modalities. The spatial organization of STS is consistent with processing streams that are hypothesized to synthesize perceptual speech representations from sensory signals that provide convergent information from visual and auditory modalities.

Purpose Five experiments probed auditory-visual (AV) understanding of sentences by users of cochlear implants (CIs). Method Sentence material was presented in auditory (A), visual (V), and AV test conditions to listeners with normal hearing and CI users. Results (a) Most CI users report that most of the time, they have access to both A and V information when listening to speech. (b) CI users did not achieve better scores on a task of speechreading than did listeners with normal hearing. (c) Sentences that are easy to speechread provided 12 percentage points more gain to speech understanding than did sentences that were difficult. (d) Ease of speechreading for sentences is related to phrase familiarity. (e) Users of bimodal CIs benefit from low-frequency acoustic hearing even when V cues are available, and a second CI adds to the benefit of a single CI when V cues are available. (f) V information facilitates lexical segmentation by improving the recognition of the number of syllables produced and the relative strength of these syllables. Conclusions Our data are consistent with the view that V information improves CI users' ability to identify syllables in the acoustic stream and to recognize their relative juxtaposed strengths. Enhanced syllable resolution allows better identification of word onsets, which, when combined with place-of-articulation information from visible consonants, improves lexical access.

The research aims to find out how the influence of SAVI model (somatic auditory visual intellectualy)) in terms of student learning outcomes in the fourth grade of social studies subjects social problems in the State Elementary School Cimulya District Cimahi Kuningan. This study is an experimental research with quantitative approach, While the sample used in this study amounted to 32 students, Methods of data collection of IPS learning results using multiple choice test and student skill observation. The data analysis technique used in this research is normality test, homogeneity test of two variance, and hypothesis test (t test) used to know the influence of SAVI learning model (somatic auditory visual intellectualy) in terms of student learning result of grade IV in subject matter social studies Social in State Elementary School Cimulya District Cimahi Kuningan. Based on statistical perceptions obtained obtained value t_hitung = 31.66, while from table t distribution at significance level α = 0.05, obtained value t_ (0.95 (62)) = 1.999. Since 31.66> 1.999 or t_count>〖t〗 _ (1- α) then H_0 is rejected or H_1 is accepted. This means that there is an increase in student learning outcomes after applying SAVI learning model (somatic auditory visual intellectualy) on social science subjects (IPS) social problem material in class IV Cimulya State Elementary District Cimahi District Kuningan

Our brain constantly integrates signals across different senses. Auditory–visual synaesthesia is an unusual form of cross-modal integration in which sounds evoke involuntary visual experiences. Previous research primarily focuses on synaesthetic colour, but little is known about non-colour synaesthetic visual features. Here we studied a group of synaesthetes for whom sounds elicit consistent visual experiences of coloured ‘geometric objects’ located at specific spatial location. Changes in auditory pitch alter the brightness, size, and spatial height of synaesthetic experiences in a systematic manner resembling the cross-modal correspondences of non-synaesthetes, implying synaesthesia may recruit cognitive/neural mechanisms for ‘normal’ cross-modal processes. To objectively assess the impact of synaesthetic objects on behaviour, we devised a multi-feature cross-modal synaesthetic congruency paradigm and asked participants to perform speeded colour or shape discrimination. We found irrelevant sounds influenced performance, as quantified by congruency effects, demonstrating that synaesthetes were not able to suppress their synaesthetic experiences even when these were irrelevant for the task. Furthermore, we found some evidence for task-specific effects consistent with feature-based attention acting on the constituent features of synaesthetic objects: synaesthetic colours appeared to have a stronger impact on performance than synaesthetic shapes when synaesthetes attended to colour, and vice versa when they attended to shape. We provide the first objective evidence that visual synaesthetic experience can involve multiple features forming object-like percepts and suggest that each feature can be selected by attention despite it being internally generated. These findings suggest theories of the brain mechanisms of synaesthesia need to incorporate a broader neural network underpinning multiple visual features, perceptual knowledge, and feature integration, rather than solely focussing on colour-sensitive areas.

International audience; Traditionally, virtual reality exposure-based treatment concentrates primarily on the presentation of a high fidelity visual experience. However, adequately combining the visual and the auditory experience provides a powerful tool to enhance sensory processing and modulate attention. We present the design and usability testing of an auditory-visual interactive environment for investigating virtual reality exposurebased treatment for cynophobia. The specificity of our application is that it involves 3D sound, allowing the presentations and spatial manipulation of a fearful stimulus in the auditory modality, in the visual modality, and both. We conducted an evaluation test with 10 dog fearful participants in order to assess the capacity of our auditory-visual virtual environment to generate fear reactions. The specific perceptual characteristics of the dog model that were implemented in the virtual environment were highly arousing, suggesting that virtual reality is a promising tool to treat cynophobia.

International audience; Down Syndrome (DS) is a genetic disease involving a number of anatomical, physiological and cognitive impairments. More particularly it affects speech production abilities. This results in reduced intelligibility which has however only been evaluated auditorily. Yet, many studies have demonstrated that adding vision to audition helps perception of speech produced by people without impairments especially when it is degraded as is the case in noise. The present study aims at examining whether the visual information improves intelligibility of people with DS. 24 participants without DS were presented with VCV sequences (vowel-consonant-vowel) produced by four adults (2 with DS and 2 without DS). These stimuli were presented in noise in three modalities: auditory, auditory-visual and visual. The results confirm a reduced auditory intelligibility of speakers with DS. They also show that, for the speakers involved in this study, visual intelligibility is equivalent to that of speakers without DS and compensates for the auditory intelligibility loss. An analysis of the perceptual errors shows that most of them involve confusions between consonants. These results put forward the crucial role of multimodality in the improvement of the intelligibility of people with DS.

This study examined the effect of auditory-visual (AV) speech stimuli on working memory in older adults with poorer-hearing (PH) in comparison to age- and education-matched older adults with better hearing (BH). Participants completed a working memory n-back task (0- to 2-back) in which sequences of digits were presented in visual-only (i.e., speech-reading), auditory-only (A-only), and AV conditions. Auditory event-related potentials (ERP) were collected to assess the relationship between perceptual and working memory processing. The behavioral results showed that both groups were faster in the AV condition in comparison to the unisensory conditions. The ERP data showed perceptual facilitation in the AV condition, in the form of reduced amplitudes and latencies of the auditory N1 and/or P1 components, in the PH group. Furthermore, a working memory ERP component, the P3, peaked earlier for both groups in the AV condition compared to the A-only condition. In general, the PH group showed a more robust AV benefit; however, the BH group showed a dose-response relationship between perceptual facilitation and working memory improvement, especially for facilitation of processing speed. Two measures, reaction time and P3 amplitude, suggested that the presence of visual speech cues may have helped the PH group to counteract the demanding auditory processing, to the level that no group differences were evident during the AV modality despite lower performance during the A-only condition. Overall, this study provides support for the theory of an integrated perceptual-cognitive system. The practical significance of these findings is also discussed.

Through a series of investigations involving different levels of contextual fidelity we devel- oped scales of perceived urgency for several dimensions of the auditory, visual, and tactile modalities. Psychophysical ratings of perceived urgency, annoyance, and acceptability as well as behavioral re- sponses to signals in each modality were obtained and analyzed using Steven's Power Law to allow comparison across modalities. Obtained results and their implications for use as in-vehicle alerts and warnings are discussed.

From birth, infants detect associations between the locations of static visual objects and sounds they emit, but there is limited evidence regarding their sensitivity to the dynamic equivalent when a sound-emitting object moves. In 4 experiments involving thirty-six 2-month-olds, forty-eight 5-month-olds, and forty-eight 8-month-olds, we investigated infants' ability to process this form of spatial colocation. Whereas there was no evidence of spontaneous sensitivity, all age groups detected a dynamic colocation during habituation and looked longer at test trials in which sound and sight were dislocated. Only 2-month-olds showed clear sensitivity to the dislocation relation, although 8-month-olds did so following additional habituation. These results are discussed relative to the intersensory redundancy hypothesis and work suggesting increasing specificity in processing with age.

Auditory prominence is defined as when an acoustic segment is made salient in its context. Prominence is one of the prosodic functions that has been shown to be strongly correlated with facial movements. In this work, we investigate the effects of facial prominence cues, in terms of gestures, when synthesized on animated talking heads. In the first study, a speech intelligibility experiment is conducted, speech quality is acoustically degraded and the fundamental frequency is removed from the signal, then the speech is presented to 12 subjects through a lip synchronized talking head carrying head-nods and eyebrows raise gestures, which are synchronized with the auditory prominence. The experiment shows that presenting prominence as facial gestures significantly increases speech intelligibility compared to when these gestures are randomly added to speech. We also present a follow-up study examining the perception of the behavior of the talking heads when gestures are added over pitch accents. Using eye-gaze tracking technology and questionnaires on 10 moderately hearing impaired subjects, the results of the gaze data show that users look at the face in a similar fashion to when they look at a natural face when gestures are coupled with pitch accents opposed to when the face carries no gestures. From the questionnaires, the results also show that these gestures significantly increase the naturalness and the understanding of the talking head.

In auditory–visual synesthesia, sounds automatically elicit conscious and reliable visual experiences. It is presently unknown whether this reflects early or late processes in the brain. It is also unknown whether adult audiovisual synesthesia resembles auditory-induced visual illusions that can sometimes occur in the general population or whether it resembles the electrophysiological deflection over occipital sites that has been noted in infancy and has been likened to synesthesia. Electrical brain activity was recorded from adult synesthetes and control participants who were played brief tones and required to monitor for an infrequent auditory target. The synesthetes were instructed to attend either to the auditory or to the visual (i.e., synesthetic) dimension of the tone, whereas the controls attended to the auditory dimension alone. There were clear differences between synesthetes and controls that emerged early (100 msec after tone onset). These differences tended to lie in deflections of the auditory-evoked potential (e.g., the auditory N1, P2, and N2) rather than the presence of an additional posterior deflection. The differences occurred irrespective of what the synesthetes attended to (although attention had a late effect). The results suggest that differences between synesthetes and others occur early in time, and that synesthesia is qualitatively different from similar effects found in infants and certain auditory-induced visual illusions in adults. In addition, we report two novel cases of synesthesia in which colors elicit sounds, and vice versa.

It is known that the brain adaptively recalibrates itself to small (approximately 100 ms) auditory-visual (AV) temporal asynchronies so as to maintain intersensory temporal coherence. Here we explored whether spatial disparity between a sound and light affects AV temporal recalibration. Participants were exposed to a train of asynchronous AV stimulus pairs (sound-first or light-first) with sounds and lights emanating from either the same or a different location. Following a short exposure phase, participants were tested on an AV temporal order judgement (TOJ) task. Temporal recalibration manifested itself as a shift of subjective simultaneity in the direction of the adapted audiovisual lag. The shift was equally big when exposure and test stimuli were presented from the same or different locations. These results provide strong evidence for the idea that spatial co-localisation is not a necessary constraint for intersensory pairing to occur.

It has been proposed that there is a perceptual compensation for the difference between the speeds of light and sound. We examined this possibility using a range of auditory visual tasks, in which performance depends on the relative timing of auditory and visual information, and manipulated viewing distance to test for perceptual compensation. We explored auditory-visual integration, cross modal causal attributions, and auditory-visual temporal order judgments. We observed timing shifts with viewing distance following loudspeaker, but not headphone, presentations. We were unable to find reliable evidence of perceptual compensation. Our findings suggest that auditory and visual signals of an event that reach an observer at the same point in time tend to become perceptually bound, even when the sources of those signals could not have occurred together. (c) 2004 Elsevier Ltd. All rights reserved.

We investigated the extent to which auditory and visual motion signals are combined when observers are asked to predict the location of a virtually moving target. In Condition 1, the unimodal and bimodal signals were noisy, but the target object was continuously visible and audible; in Condition 2, the virtually moving object was hidden (invisible and inaudible) for a short period prior to its arrival at the target location. Our main finding was that the facilitation due to simultaneous visual and auditory input is very different for the two conditions. When the target is continuously visible and audible (Condition 1), the bimodal performance is twice as good as the unimodal performances, thus suggesting a very effective integration mechanism. On the other hand, if the object is hidden for a short period (Condition 2) and the task therefore requires the extrapolation of motion speed over a temporal and spatial period, the facilitation due to both sensory inputs is almost absent, and the bimodal performance is limited by the visual performance.

The McGurk effect, in which auditory [ba] dubbed onto [go] lip movements is perceived as da or tha, was employed in a real-time task to investigate auditory-visual speech perception in prelingual infants. Experiments 1A and 1B established the validity of real-time dubbing for producing the effect. In Experiment 2, 4(1)/(2)-month-olds were tested in a habituation-test paradigm, in which 2 an auditory-visual stimulus was presented contingent upon visual fixation of a live face. The experimental group was habituated to a McGurk stimulus (auditory [ba] visual [ga]), and the control group to matching auditory-visual [ba]. Each group was then presented with three auditory-only test trials, [ba], [da], and [deltaa] (as in then). Visual-fixation durations in test trials showed that the experimental group treated the emergent percept in the McGurk effect, [da] or [deltaa], as familiar (even though they had not heard these sounds previously) and [ba] as novel. For control group infants [da] and [deltaa] were no more familiar than [ba]. These results are consistent with infants'perception of the McGurk effect, and support the conclusion that prelinguistic infants integrate auditory and visual speech information. (C) 2004 Wiley Periodicals, Inc.

Recent studies of auditory-visual integration have reached diametrically opposed conclusions as to whether individuals differ in their ability to integrate auditory and visual speech cues. A study by Massaro and Cohen [J. Acoust. Soc. Am. 108(2), 784–789 (2000)] reported that individuals are essentially equivalent in their ability to integrate auditory and visual speech information, whereas a study by Grant and Seitz [J. Acoust. Soc. Am. 104(4), 2438–2450 (1998)] reported substantial variability across subjects in auditory-visual integration for both sentences and nonsense syllables. This letter discusses issues related to the measurement of auditory-visual integration and modeling efforts employed to separate information extraction from information processing.

The speech intelligibility of syllables spoken under conditions of conversational and clear speech was compared. The stimuli were 18 monosyllables (/C-v/) and 18 bisyllables (/v-C-v/) consisting of six voiced-consonants (/b, d, g, v, z, ʒ /) presented in each of three vowel contexts (/a, i, y/). Six female adults were recorded while they produced four iterations of the stimulus set in each of the two speaking styles. The 1728 videotaped test items were edited, randomized and presented to 12 subjects with normal hearing and normal visual acuity under three conditions: visual-only, auditory-only and audiovisually. A broadband noise was mixed with the signal for the latter two conditions. The results revealed a significant three-way interaction of talker, speaking style and perceptual modality. Post-hoc analyses revealed intra and interspeaker differences in speech intelligibility for both speaking styles, in all three perceptual modalities. Overall, positive clear speech effects were observed in all three modalities. Intermodality comparisons revealed differences in the pattern of clear speech effects displayed by individual talkers. This finding indicates that there is not a direct association between the beneficial effects of clear speech in one perceptual modality and its effects on speech intelligibility in another perceptual modality.

Linda Septiyana. S891408022. 2015. Somatic, Auditory, Visual, Intellectual (SAVI): Its Effectiveness to Teach Writing from the Perspective of Students’ Critical Thinking (An Experimental Study at the Eleventh Grade Students of SMAN 3 Metro in the academic year 2015/2016). A Thesis. First Consultant: Dra. Dewi Rochsantiningsih, M.Ed, Ph.D.; Second Consultant: Dr. Ngadiso, M.Pd. English Education Department, Graduate Progam, Sebelas Maret University. This research aims to verify whether: (1) SAVI is more effective than TSTS to teach writing; (2) The students who have high critical thinking have better writing skill than those who have low critical thinking; and (3) There is an interaction between teaching methods and students’ critical thinking in teaching writing. This experimental study was conducted at the eleventh grade students of SMAN 3 Metro in the academic year 2015/2016. The sample of this research was taken using cluster random sampling. The sample of this research was two classes; XI-IPA-2 as an experimental class and XI-IPA-1 as a control class. The experimental class was taught using SAVI, while the control class was taught using TSTS. Each class consisted of two groups based on the level of critical thinking (high and low). The instruments for collecting the data were writing test (to measure the writing skill) and critical thinking test (to measure the critical thinking level). The data were analyzed in term of their frequency distribution, normality of the sample distribution, and the data homogeneity. Then, the data were analyzed using Multifactor Analysis of Variance (ANOVA) test 2×2 and TUCKEY test. Research findings shows that: (1) SAVI differs significantly from TSTS to teach writing because Fo > Ft and qo (between A1 and A2) > qt; (2) students having high critical thinking differ significantly from those having low critical thinking because Fo > Ft and qo (between B1 and B2) > qt; and (3) there is an interaction between teaching methods and students’ critical thinking in teaching writing because Fo > Ft, and the result of Tukey test shows that: (a) for the students having high critical thinking, SAVI method differs significantly from TSTS method to teach writing because qo (between A1B1 and A2B1) > qt; and (b) for the students having low critical thinking, TSTS method differs significantly from SAVI method to teach writing because qo (between A1B2 and A2B2) > qt. Finally, the results of this research imply that SAVI is an effective method to teach writing. The research also shows that the effect of SAVI depends on the students’ critical thinking. Therefore, should the teachers implement SAVI, they have to be able to improve the students’ critical thinking. Keywords: writing, Somatic, Auditory, Visual, Intellectual (SAVI), Two Stay Two Stray, critical thinking, experimental study

This paper introduces and analyses a collaborative task for eliciting auditory-visual dialogs based on the viewing of two versions of the same cartoon film. The original film was edited and cut in such a way that the story must be reconstructed by joining information from two incomplete versions which however share between them all the scenes in a consecutive fashion. Our intention is to elicit a relatively balanced dialog between the two participants throughout the conversation as they are piecing together the story from the beginning to the end. The current paper describes the production of the auditory-visual corpus using audio, video and motion capturing of 22 pairs of Australian English speaking participants, and presents first results regarding turndistribution and raw prosodic features. Our analysis shows that the task is indeed relatively balanced between talkers though this does not apply equally to all pairs. Analysis of raw prosodic features does not suggest convergence throughout the conversation, but replicates, for instance earlier findings of similarity between partners as compared to others.

The ability of 3-month-old infants to learn arbitrary auditory–visual associations between voices and faces was investigated by familiarizing each infant to two alternating stimuli presented on a VCR monitor. Each stimulus was a voice–face combination, where the voices and faces were male and/or female. On the post-familiarization test trials each infant was presented alternately with a familiar and a novel voice–face combination, where the novel combination consisted of a voice and a face they had heard and seen previously (but not together), and on these test trials attention was significantly higher to the novel combination. These findings are a clear demonstration that 3-month-olds can learn arbitrary voice–face associations, and they are discussed in terms of early intermodal perception and face perception. Copyright © 2001 John Wiley & Sons, Ltd.

The fuzzy logical model of perception [FLMP, Massaro, Perceiving Talking Faces: From Speech Perception to a Behavioral Principle (MIT Press, Cambridge, MA, 1998)] has been extremely successful at describing performance across a wide range of ecological domains as well as for a broad spectrum of individuals. Because the model predicts optimal or maximally efficient integration, an important issue is whether this is the case for most individuals. Three databases are evaluated to determine to what extent a significant quantitative improvement in predictive ability can be obtained if integration is assumed to be somewhat inefficient. For the most part, there were no significant signs of inefficient integration. The previous differences found by Grant and Seitz [J. Acoust. Soc. Am. 104, 2438–2450 (1998)] must be due to their measures of efficiency, which appear to be invalid and/or conflate information with integration efficiency. Finally, the descriptive ability of the FLMP is shown to be theoretically informative and not simply the model’s ability to describe any possible outcome.

The experiments reported here used auditory–visual mismatches to compare three approaches to speaker normalization in speech perception: radical invariance, vocal tract normalization, and talker normalization. In contrast to the first two, the talker normalization theory assumes that listeners' subjective, abstract impressions of talkers play a role in speech perception. Experiment 1 found that the gender of a visually presented face affects the location of the phoneme boundary between [Ω] and [Λ] in the perceptual identification of a continuum of auditory–visual stimuli ranging from hood to hud. This effect was found for both “stereotypical” and “non-stereotypical” male and female voices. The experiment also found that voice stereotypicality had an effect on the phoneme boundary. The difference between male and female talkers was greater when the talkers were rated by listeners as “stereotypical”. Interestingly, for the two female talkers in this experiment, rated stereotypicality was correlated with voice breathiness rather than vowel fundamental frequency. Experiment 2 replicated and extended experiment 1 and tested whether the visual stimuli in experiment 1 were being perceptually integrated with the acoustic stimuli. In addition to the effects found in experiment 1, there was a boundary effect for the visually presented word: listeners responded hood more frequently when the acoustic stimulus was paired with a movie clip of a talker saying hood. Experiment 3 tested the abstractness of the talker information used in speech perception. Rather than seeing movie clips of male and female talkers, listeners were instructed to imagine a male or female talker while performing an audio-only identification task with a gender-ambiguous hood-hud continuum. The phoneme boundary differed as a function of the imagined gender of the talker. The results from these experiments suggest that listeners integrate abstract gender information with phonetic information in speech perception. This conclusion supports the talker normalization theory of perceptual speaker normalization.

This investigation examined the accuracy of judgements of traffic speed by comparing audio, visual and audio-visual perception in two experiments. In the first study 60 adults estimated the approach speed of a car on video. Five speeds (10,20, 30,40 and 50m.p.h.) were presented. In the second study 37 adults were presented with videos of an approaching car travelling at either 30, 40 or 50m.p.h. and chose the actual speed from five given alternatives. Results indicated a low level of accuracy overall. For the first study extent of error increased in the order audio-visual, visual and auditory. In the second study, no significant effect of perceptual mode was observed for extent of error. In both studies an interaction between speed and perceptual mode was observed for extent of error. Speed was over-estimated in the auditory mode and under-estimated in the visual and audio-visual modes. Findings were discussed in relation to witness capabilities.

In this study the ability of newborn infants to learn arbitrary auditory–visual associations in the absence versus presence of amodal (redundant) and contingent information was investigated. In the auditory-noncontingent condition 2-day-old infants were familiarized to two alternating visual stimuli (differing in colour and orientation), each accompanied by its ‘own’ sound: when the visual stimulus was presented the sound was continuously presented, independently of whether the infant looked at the visual stimulus. In the auditory-contingent condition the auditory stimulus was presented only when the infant looked at the visual stimulus: thus, presentation of the sound was contingent upon infant looking. On the post-familiarization test trials attention recovered strongly to a novel auditory–visual combination in the auditory-contingent condition, but remained low, and indistinguishable from attention to the familiar combination, in the auditory-noncontingent condition. These findings are a clear demonstration that newborn infants’ learning of arbitrary auditory–visual associations is constrained and guided by the presence of redundant (amodal) contingent information. The findings give strong support to Bahrick’s theory of early intermodal perception.

After preliminary computerized training on visual-visual identity matching, a 5-year-old boy with autism (Sam) was given visual-visual and auditory-visual matching-to-sample tests with new stimuli. He did well in matching dictated name samples to 20 pictures, 26 printed upper case letters, and 9 single-digit numbers. In matching the visual stimuli (pictures, letters, or numbers) to themselves, however, he did not perform well. We then increased the number of picture comparisons per trial from two to three. In tests after this three-comparison training, Sam correctly matched on 95% of the original 20-stimulus, four-comparison, identity-matching test trials. He went on to demonstrate accurate identity matching of the numbers, letters, and new pictures. In identity-matching tests on the table top, he performed poorly until the stimulus array was made to resemble the stimulus arrangement on the computer. These findings showed that seemingly small procedural changes can influence performance and demonstrated that successful auditory-visual matching does not guarantee proficiency in visual-visual identity matching.

The role of spatial co-location between sight and sound in infants' cross-modal learning was examined in three experiments. Four-and 6-month-old infants were familiarized with a toy and an accompanying soundtrack. Across conditions, spatial congruity between sight and sound was varied. Following familiarization, infants were tested to determine under which conditions they learned to associate the toy with the sound. Results indicated age-related differences in how discrepant in location a sight and sound could be for infants to form a cross-modal association based on the amodal invariant of co-location. Specifically, 4-month-olds formed cross-modal associations under conditions of less precise co-location than did 6-month-olds. Parallel improvements in infants' sound localization abilities across this age span are likely a contributing factor to the observed developmental trend in cross-modal learning.

The majority of research on infant crossmodal perception has addressed the question of whether infants 4 months and older demonstrate the ability. By contrast, the focus in the present study was on newborns and the primary goal was twofold: to determine if newborns can learn arbitrary sight-sound pairings and to evaluate possible inferences that neonates make when repeatedly presented arbitrary sight-sound pairings. Alert neonates were familiarized with toy-sound pairs using an infant-controlled habituation procedure. They were then given a violation-of-expectancy test to determine if they had learned these sight-sound pairings and drawn any inferences about certain properties of these bimodal events. Results supported the conclusion that infants just a few hours old can learn sight-sound pairings. Furthermore, the findings suggested that newborns expected sight-sound pairs to remain colocated even when they move to a new location, they recognized that the sound was an attribute of a specific object, and they recognized that a specific spatial location was not an attribute of the sight-sound pair. Thus, infants made important and correct inferences based on limited experiences with specific auditory-visual events, and they did so just after birth. Implications of these findings for theories of crossmodal perceptual development are discussed.

Most of the stimuli that we experience are intermodal in that they provide information to more than one sensory modality. Some of these intermodal relationships are amodal in that they provide equivalent information to the senses, while others are quite arbitrary. For instance, there is no information specifying that a particular voice has to be associated with a particular face, or that a particular animal makes a particular sound. The ability of newborn infants to learn arbitrary visual–auditory associations was investigated by familiarizing 2-day-old infants to two alternating visual stimuli (differing in colour and orientation), each accompanied by its ‘own’ sound. On post-familiarization test trials attention recovered to a novel visual–auditory combination. These findings are a clear demonstration that newborn infants can learn arbitrary visual–auditory associations, and they are discussed in terms of Bahrick's theory of early intermodal perception. ©1997 John Wiley & Sons, Ltd.