Search

Human perception depends upon internal representations of the environment that help to organize the raw information available from the senses by acting as reference patterns. Internal representations are widely characterized using reverse correlation, a method capable of producing unconstrained estimates of the representation itself, all on the basis of simple responses to random stimuli. Despite its advantages, reverse correlation is often infeasible to apply because the number of stimulus-response trials needed to provide an accurate estimate is typically very large. Prior approaches have aimed to overcome this sampling inefficiency by incorporating prior knowledge of the representation, which biases the estimate and ultimately limits the essential power of reverse correlation. The present approach, however, improves efficiency via stimulus whitening, a statistical procedure that decorrelates stimuli, making them less redundant, and commensurately more favorable for efficient estimation of an arbitrary target. We provide a mathematical justification for whitening, and demonstrate in simulation that whitening provides greater than 85% improvement in efficiency for a given estimation accuracy, and also a two- to five-fold increase in accuracy for a given sample size. Improving the efficiency of reverse correlation may enable a broader scope of investigations into individual variability and potential universality of perceptual mechanisms.

Military job and training activities place significant demands on service members' (SMs') cognitive resources, increasing risk of injury and degrading performance. Early detection of cognitive fatigue is essential to reduce risk and support optimal function. This paper describes a multimodal approach, based on changes in measures of speech motor coordination and electrodermal activity (EDA), for predicting changes in performance following sustained cognitive effort.Twenty-nine active duty SMs completed computer-based cognitive tasks for 2 h (load period). Measures of speech derived from audio were acquired, along with concurrent measures of EDA, before and after the load period. Cognitive performance was assessed before and during the load period using the Automated Neuropsychological Assessment Metrics Military Battery (ANAM MIL). Subjective assessments of cognitive effort and alertness were obtained intermittently.Across the load period, participants' ratings of cognitive workload increased, while alertness ratings declined. Cognitive performance declined significantly during the first half of the load period. Three speech and arousal features predicted cognitive performance changes during this period with statistically significant accuracy: EDA (Results suggest that speech and arousal measures may be used to predict changes in performance associated with cognitive fatigue. This work supports ongoing efforts to develop reliable, unobtrusive measures for cognitive state assessment aimed at reducing injury risk, informing return to work decisions, and supporting diverse mobile healthcare applications in civilian and military settings.

This paper reviews the current state of several formal models of speech motor control, with particular focus on the low-level control of the speech articulators. Further development of speech motor control models may be aided by a comparison of model attributes. The review builds an understanding of existing models from first principles, before moving into a discussion of several models, showing how each is constructed out of the same basic domain-general ideas and components-e.g., generalized feedforward, feedback, and model predictive components. This approach allows for direct comparisons to be made in terms of where the models differ, and their points of agreement. Substantial differences among models can be observed in their use of feedforward control, process of estimating system state, and method of incorporating feedback signals into control. However, many commonalities exist among the models in terms of their reliance on higher-level motor planning, use of feedback signals, lack of time-variant adaptation, and focus on kinematic aspects of control and biomechanics. Ongoing research bridging hybrid feedforward/feedback pathways with forward dynamic control, as well as feedback/internal model-based state estimation, is discussed.

USC-TIMIT is an extensive database of multimodal speech production data, developed to complement existing resources available to the speech research community and with the intention of being continuously refined and augmented. The database currently includes real-time magnetic resonance imaging data from five male and five female speakers of American English. Electromagnetic articulography data have also been presently collected from four of these speakers. The two modalities were recorded in two independent sessions while the subjects produced the same 460 sentence corpus used previously in the MOCHA-TIMIT database. In both cases the audio signal was recorded and synchronized with the articulatory data. The database and companion software are freely available to the research community. (C) 2014 Acoustical Society of America.

Speech control models founded on principles of neuroscience have potential to create powerful clinical tools for personalized medicine and for understanding behavior. We explore two prominent speech control models: Directions into Velocities of Articulators (DIVA) (Guenther, 2006) and Task Dynamics (TD) (Saltzman, 1991) toward an understanding of speech disfluency of autism spectrum disorder (ASD). To introduce perturbations into both systems in the common framework of (Parrrell, 2018), we translate the TD model into MATLAB Simulink. For word stimuli, we compare both models’ acoustic outputs, under the hypothesis that speech impediments in ASD arise from overreliance on delayed sensory feedback (Lin, 2015). For DIVA, the output from a 50ms auditory feedback delay contain increased formant production errors and formant oscillations at about 10Hz, sometimes observed in disfluent ASD cases, but also seen in stuttering (Civier, 2010). The same delay introduced in the proprioceptive and tactile feedback in TD takes 5x the time to settle at the target, creating about 20Hz oscillations in TD articulators. Though neither fully capturing disfluency in ASD, together these models lay the groundwork for a more complete neurocomputational speech control model that can be used to longitudinally track and potentially guide ASD speech therapy.

This paper describes a spatio-temporal registration approach for speech articulation data obtained from electromagnetic articulography (EMA) and real-time Magnetic Resonance Imaging (rtMRI). This is motivated by the potential for combining the complementary advantages of both types of data. The registration method is validated on EMA and rtMRI datasets obtained at different times, but using the same stimuli. The aligned corpus offers the advantages of high temporal resolution (from EMA) and a complete mid-sagittal view (from rtMRI). The co-registration also yields optimum placement of EMA sensors as articulatory landmarks on the magnetic resonance images, thus providing richer spatio-temporal information about articulatory dynamics. (C) 2014 Acoustical Society of America

Individuals who have undergone laryngectomy often rely on handheld transducers (i.e., the electrolarynx) to excite the vocal tract and produce speech. Widely used electrolarynx designs are limited, in that they require manual control of voice activity and pitch modulation. It would be advantageous to have an interface that requires less training, perhaps using the remaining, intact speech production system as a scaffold. Strong evidence exists that aspects of head motion and facial gestures are highly correlated with gestures of voicing and pitch. Therefore, the goal of project MANATEE is to develop an electrolarynx control interface which takes advantage of those correlations. The focus of the current study is to determine the feasibility of using head and facial features to accurately and efficiently modulate the pitch of speaker's electrolarynx in real time on a mobile platform using the built-in video camera. A prototype interface, capable of running on desktop machines and compatible Android devices, is implemented using OpenCV for video feature extraction and statistical prediction of the electrolarynx control signal. Initial performance evaluation is promising, showing pitch prediction accuracies at double the chance-level baseline, and prediction delays well below the perceptually-relevant, ~50 ms threshold.

Professional singers are trained to maximize vowel duration and minimize consonant interference, while still maintaining intelligibility. The mechanisms by which they do this, however, are unclear. A deeper understanding of the gestural mechanisms utilized during professional-quality singing could be useful to train singers more effectively. It has been well-established that nasal sounds in syllable coda position have longer and larger velum motion in formal speech as compared to casual speech. The hypothesis of this study was that coda velum gestures would be temporally extended proportional to the duration of the entire syllable. To test this, real-time magnetic resonance (rtMR) images of trained soprano singers were used to track the motion of the velum. The boundaries of the vocal tract were mapped to a semi-polar grid, and the velocity of the velum at every frame of its movement was calculated. An important result shows that operatic singing has minimal change in velum gesture duration compared to speech, while a musical theater style can have velum lowering for the duration of a word. Observed gestural kinematics and their implications for speech are discussed from within the framework and perspective of Articulatory Phonology.

The tongue varies across speakers in terms of the proportion of the overall speech production apparatus that it occupies. Differences in tongue size have the potential to result in speaker-specific articulatory strategies for shaping the vocal tract area function and, in turn, individual patterns of vowel acoustics. The present study examines the interplay between relative tongue size and vowel production using real-time magnetic resonance imaging with synchronous audio. Two populations of native American English subjects are considered, one containing healthy adult speakers with no relevant pathologies, and another containing speakers who had undergone glossectomy as treatment for tongue cancer. All subjects were imaged in the midsagittal plane while reading phonetically balanced English sentences. The size of the tongue and the speech production apparatus were quantified from an overall average posture, and their ratio was correlated with the shape of the vowel space in terms of acoustics (e.g., formant frequencies), constrictions (i.e., location and degree of minimum constriction), and parameterized vocal tract cross-distance functions. Results indicate that relative tongue size can be used to explain and predict observable interspeaker differences in vowel production.

We investigate articulatory behavior in post-glossectomy speech using real-time magnetic resonance imaging. Our data reveal that listeners judge speech produced by partial-glossectomy patients as atypical when the surgical procedure affected the oral tongue. Speech produced by patients whose procedure affected the base of tongue, however, was judged as typical. We observe that preservation and compensation mechanisms are exhibited by the patients with atypical speech. They preserve appropriate modulation of F1 using tongue and/or jaw height despite inability to appropriately modulate F2 due to the reduced size and/or mobility of the tongue. Further, durational differences between tense and lax vowels are maintained. The preservation of these features serves as evidence in support of a framework within which individual gestural parameters are independently controlled; when achievement of a particular parameter specification (e.g., constriction location) is compromised, the remaining (e.g., constriction degree, activation duration) are unchanged. Compensatory behavior is exhibited when coronal tongue movement has been impeded and is exemplified by (i) production of labiodental stops in place of target coronal stops and laterals and (ii) forming a velar constriction to produce frication in place of the alveolar frication for /s/.

We present a computational model of lenition-based sound change. Speech production targets for constriction degree are modeled by differential equations with a single stable fixed point at the target constriction degree that interact with higher order equations that reflect prosodically conditioned variation. This output is then input to the articulatory-to-acoustic forward map, the quantal nature of which causes continuous variation in constriction degree to result in two outcomes: closure or spirantization. The quantized acoustic outcomes are then used as the input for the language learner, who must recover the produced constriction degree in the ambient language environment to produce the speech unit appropriately. We model this recovery as estimating the parameters of an articulatory distribution that is most likely to have produced the acoustic observations (i.e., the maximum likelihood estimate). In particular, we optimize the parameters of a Gaussian distribution with respect to their log-likelihoo...

An articulatory study of sibilant fricatives is described, with the goal of describing variability in lingual articulation across languages. Real-time Magnetic Resonance Imaging (rtMRI) data were collected from three speakers each of English and Mandarin and two speakers of Serbian and reconstructed at a rate of 22.4 frames per second. Parallel acoustic data were also collected and subsequently denoised. Subjects spoke the segments /s/ and /S/ in symmetrical vowel contexts (e.g. 'pa sap' for English and 'asa' for Mandarin). Articulation was analyzed using a semi-polar grid overlaid on the image plane and midsagittal distance functions were obtained by measuring cross distances at ~0.5 cm intervals from the glottis to the lips. Analysis shows that place of articulation for /s/ is more anterior compared to /S/ across languages. Apical articulation is observed for /s/ across languages, while /S/ is produced laminally in English and apically in Serbian and Mandarin. Patterns of tongue shaping variability differ, as well, across languages. For instance, higher standard deviation is observed anterior to the place of articulation for /S/ in Mandarin, compared to Serbian and English. [Work supported by NIH]

Articulatory setting postures adopted during speech production are examined with the goal of determining whether setting postures are more mechanically advantageous than rest positions in facilitating motion of vocal tract articulators toward task goals. Articulatory simulations using the Task Dynamics Application (TADA) suggest that setting postures afford large changes with respect to speech tasks for relatively small changes in low-level speech articulators, thus affording greater mechanical advantage as compared to absolute rest postures. This study investigates this hypothesis using real-time Magnetic Resonance Imaging (rtMRI) data of read and spontaneous speech elicited from five healthy speakers of American English. Frames corresponding to inter-speech pauses, speech-ready intervals, and absolute rest intervals were identified and image features were automatically extracted to quantify the vocal tract postures in terms of both task-level constriction variables and articulatory variables. Locally Weighted Regression is then used to estimate the ratio of task velocities to articulator velocities (i.e., the lever or speed ratio) at postures corresponding to the different intervals of interest. Results show substantially higher speed ratios at inter-speech and ready postures as compared to absolute rest postures. [Work supported by NIH.]

The length of the vocal tract and its relationship with formant frequencies is examined at fine temporal scales with the goal of providing accurate estimates of vocal tract length from acoustics on a spectrum-by-spectrum basis despite unknown articulatory information. Accurate vocal tract length estimation is motivated by applications to speaker normalization and biometrics. Analyses presented are both theoretical and empirical. Various theoretical models are used to predict the behavior of vocal tract resonances in the presence of different vocal tract lengths and constrictions. Real-time MRI with synchronized audio is also utilized for detailed measurements of vocal tract length and formant frequencies during running speech, facilitating the examination of short-time changes in vocal tract length and corresponding changes in formant frequencies, both within and across speakers. Previously proposed methods for estimating vocal tract length are placed within a coherent framework, and their effectiveness is evaluated and compared. A data-driven method for VTL estimation emerges as a natural extension of this framework, which is then developed and shown to empirically outperform previous methods on both synthetic and real speech data. A theoretical justification for the effectiveness of this new method is also explained. [Work supported by NIH.]

We propose a method for co-registrating speech articulatory/acoustic data from two modalities that provide complementary advantages. Electromagnetic Articulography (EMA) provides high temporal resolution (100 samples/second in WAVE system) and flesh-point tracking, while real-time Magnetic Resonance Imaging, rtMRI, (23 frames/second) offers a complete midsagittal view of the vocal tract, including articulated structures and the articulatory environment. Co-registration was achieved through iterative alignment in the acoustic and articulatory domains. Acoustic signals were aligned temporally using Dynamic Time Warping, while articulatory signals were aligned variously by minimization of mean total error between articulatometry data and estimated corresponding flesh points and by using mutual information derived from articulatory parameters for each sentence. We demonstrate our method on a subset of the TIMIT corpus elicited from a male and a female speaker of American English, and illustrate the benefits of co-registered multi-modal data in the study of liquid and fricative consonant production in rapid speech. [Supported by NIH and NSF grants.]

Spondees, both true disyllabic words and concatenated monosyllabic word pairs, had their middle section replaced by silence that extended from the midpoint of the first to the midpoint of the second vowel; the silence thus included the final consonant(s) of the first and the initial consonant(s) of the second syllable. These stimuli were presented to normal-hearing young listeners instructed to guess both monosyllabic half words. Input and response spondees were orthographically aligned and analyzed as confusion matrices. Articulatory gestures were estimated for each token via the Haskins Laboratories TADA articulatory synthesis. After time alignment, articulatory distances were calculated for each stimulus-response pair. In the blanked middle of the spondee, phoneme-based confusions of place-of-articulation were high (39% accuracy), while gestural distances underlying this feature (location and degree of tongue tip constriction, lip aperture, and velar constriction) were under 10%. These results suggest ...

It has been shown that subjects may recruit extra-linguistic articulators during the rhythmic production of speech (Tiede, 2010), that sympathetic head movement is associated with dysfluencies in running speech (Hadar, 1984), and that the magnitude of this activity positively correlates with both speaking rate and increased effort in accelerating speech production tasks (Hadar, 1991; Tiede, 2011). It is unclear at what level of linguistic organization this entrainment occurs. We examined patterns of articulation and head movement during an accelerated repetitive speech task involving multi-gestural segments, in order to gain more insights into the coordinative bases of this behavior. Real-time magnetic resonance imaging was used to examine the production of paired English words, contrasting along various articulations, e.g., “cop-top,” “kid-kim,” “flee-free,” “muck-duck.” Subjects' upper airways were imaged in the midsagittal plane while producing trochaic repetitions of each word pair in time to an accel...

Most examinations of glottal abduction and adduction during speech have employed laryngoscopic video or transillumination (Hoole, 2006). While these provide accurate information about timing of glottal movements, they are invasive and cannot provide absolute measurements about glottal width. At the same time, recent medical studies have used ultrasonic imaging to accurately capture glottal movements and laryngeal anatomy [Hu, J. Ultrasound Med. (2010); Jadcherla etal., Dysphagia (2006)]. We demonstrate novel methods of using ultrasound to measure both temporal and spatial aspects of glottal movements during speech. While previous work on glottal ultrasound has been limited by the need to manually analyze each acquired frame, we present methods to automatically quantify glottal aperture in ultrasound images. Finally, since glottal ultrasound does not interfere with the acquisition of supra-laryngeal articulatory data, we present the results of preliminary experiments that record laryngeal and supra-larynge...

The human speech production system can be fundamentally characterized by the kinematic relationships between low-level articulator variables and relatively high-level tasks. Such kinematics can be illuminating about many system aspects from degrees of freedom and redundancy to dynamics and even control. Since these relationships are generally complex and infeasible to express in closed form, recent work has focused on statistical methods for estimating the relevant relationships from data (Saltzman, 2006, in Dynamics of Speech Production and Perception; Lammert, 2010, Proc. INTERSPEECH). Such methods have been applied to synthetic speech data in order to evaluate their effectiveness, but they have not yet been demonstrated on real data. Here, we apply these methods to real speech data acquired from real-time magnetic resonance imaging. We extract articulator variables that are consistent with those chosen by various articulatory models, and we relate them to high-level task variables such as constriction locations and degrees and formant frequencies. This is done to facilitate an analysis of articulatory motor control during speech production. [Work supported by NIH.]

Articulometry studies have demonstrated that speech errors often involve gestural intrusions that are coproduced with other gestures and can go unnoticed because they are not always perceived to be segmental errors. [Mowrey and MacKay (1990)]. Real time magnetic resonance imaging was used to examine tongue movement during the production of English minimal pairs “cop‐top,” “kid‐kim,” “sop‐shop,” and “bad‐bang.” Subjects’ upper airways were imaged in the midsagittal plane while they uttered repetitions of each word pair as rapidly as possible for 15 sec. Degree of constriction in each region of interest (labial, alveolar, post‐alveolar, and velic) was automatically tracked with direct image processing techniques. We calculate the mean intensity of temporally correlated pixels in that region–a metric which has been found to provide a robust estimate of constriction degree in noisy data. [Lammert (in preparation).] Gestural intrusions are observed throughout these data; frequency of occurrence increases with speech rate. Intrusions are invariably synchronous with the alternating contrastive gesture, e.g., additional velic gestures are observed at the same time as the tongue tip constriction in the word pair “cop‐top.” These data are consistent with the hypothesis that speech production involves the coordination of dynamically defined action units. [Goldstein et al. (2007).]

An algorithm was developed to denoise speech presented in matching speech‐spectrum noise by estimating the ideal binary mask (IBM) that retains the time‐frequency (T‐F) regions with favorable local signal‐to‐noise ratio (SNR) and removes the remaining regions [Hu et al., in Proceedings of the 4th SAPA Workshop, Interspeech 2008]. With this method sentence intelligibility was tested in 20 elderly and 20 normal‐hearing young listeners. The elderly group had, at worst, moderate SNHL between 0.5 and 4 kHz. Speech material consisted of IEEE sentences spoken by a single female talker. Intelligibility was tested at two SNRs in the elderly (−2 and −4 dB) and three SNRs in the young group (−4, −6, and −8 dB). Subjects listened to 50 sentences at each SNR in three conditions: denoising by the estimated IBM by the algorithm, denoising by the IBM calculated from premixed signals, and the sentences presented in the original noise without processing. With this last condition taken as the baseline, both keyword‐ and phonemic intelligibility of the sentences processed with the algorithm showed a slight improvement for the young but no improvement for the elderly listeners, while both groups performed substantially better than the baseline in the IBM condition. [Work supported by the VA Medical Research.]

Debate exists about the nature of schwa in terms of articulatory planning and execution. It is commonly believed that schwa lacks a specific constriction target. Another view is that the constriction target for schwa is the mean of the language’s targeted vowels. Some have argued that both lines of thinking are correct [I. Smorodinsky, J. Acoust. Soc. Am. 109, 2446 (2001)]. Specifically, epenthetic schwas (e.g., needed) may indeed be targetless, while lexical schwas (e.g., panda) have a target. To investigate this, we collected midsagittal realtime l MR images [S. Narayanan et al., J. Acoust. Soc. Am. 115, 1771 (2004)] from two native English speakers. The speakers read sentences aloud which contained, variously, lexical or epenthetic schwas in multiple contexts. Images of the various schwas were extracted from the MR data and a profile of the vocal tract was fit to the image. If both targeted and targetless schwas exist, then they should be evident by examining the vocal tract shape in these images. Lexi...

Articulatory synthesis methods, classic and contemporary, have demonstrated that it is possible to generate speech from an ensemble of functions derived from articulatory gestures. Such gesture‐to‐waveform transforms suggest that, inversely, the speech signal should be also decomposable into the same set of gesture, or gesture‐like, functions. These functions vary slowly in time and their association with the speech waveform (words as well as sentences) can be established by machine learning algorithms. In a recent study at our laboratory, listeners were asked to type the word or the sentence they heard, with speech (degraded in diverse ways) as the stimulus. The subjects' responses were synthesized, time‐aligned with the stimulus, and decomposed into a set of eight gestures, as specified by the Haskins Laboratories TADA system (http://www.haskins.yale.edu/tada_download/index.html). When the running distance between input and response gesture functions is calculated, results indicate a significant degree ...