Search

Purpose: This study investigated the criterion (analytical and clinical) and construct (divergent) validity of a novel, acoustic-based framework composed of five key components of motor control: Coordination, Consistency, Speed, Precision, and Rate. Method: Acoustic and kinematic analyses were performed on audio recordings from 22 subjects with amyotrophic lateral sclerosis during a sequential motion rate task. Perceptual analyses were completed by two licensed speech-language pathologists, who rated each subject's speech on the five framework components and their overall severity. Analytical and clinical validity were assessed by comparing performance on the acoustic features to their kinematic correlates and to clinician ratings of the five components, respectively. Divergent validity of the acoustic-based framework was then assessed by comparing performance on each pair of acoustic features to determine whether the features represent distinct articulatory constructs. Bivariate correlations and partial correlations with severity as a covariate were conducted for each comparison. Results: Results revealed moderate-to-strong analytical validity for every acoustic feature, both with and without controlling for severity, and moderate-to-strong clinical validity for all acoustic features except Coordination, without controlling for severity. When severity was included as a covariate, the strong associations for Speed and Precision became weak. Divergent validity was supported by weak-to-moderate pairwise associations between all acoustic features except Speed (second-formant [F2] slope of consonant transition) and Precision (between-consonant variability in F2 slope). Conclusions: This study demonstrated that the acoustic-based framework has potential as an objective, valid, and clinically useful tool for profiling articulatory deficits in individuals with speech motor disorders. The findings also suggest that compared to clinician ratings, instrumental measures are more sensitive to subtle differences in articulatory function. With further research, this framework could provide more accurate and reliable characterizations of articulatory impairment, which may eventually increase clinical confidence in the diagnosis and treatment of patients with different articulatory phenotypes.

GoalThis study validates an approach to characterizing the sounds experienced by tinnitus patients via reverse correlation, with potential for characterizing a wider range of sounds than currently possible.MethodsTen normal-hearing subjects assessed the subjective similarity of random auditory stimuli and target tinnitus-like sounds (“buzzing” and “roaring”). Reconstructions of the targets were obtained by regressing subject responses on the stimuli, and were compared for accuracy to the frequency spectra of the targets using Pearson’sr.ResultsReconstruction accuracy was significantly higher than chance across subjects: buzzing (M= 0.53,SD= 0.27):t(9) = 5.766,p <0.001; roaring (M= 0.57,SD= 0.30):t(9) = 5.76,p <0.001.ConclusionReverse correlation can accurately reconstruct nontonal tinnitus-like sounds in normal-hearing subjects, indicating its potential for characterizing the sounds experienced by patients with non-tonal tinnitus.Impact StatementCharacterization of tinnitus sounds can inform treatment by facilitating individualized sound therapies, leading to better outcomes for patients suffering from the cognitive and psychological effects of tinnitus.

Human perception depends upon internal representations of the environment that help to organize the raw information available from the senses by acting as reference patterns. Internal representations are widely characterized using reverse correlation, a method capable of producing unconstrained estimates of the representation itself, all on the basis of simple responses to random stimuli. Despite its advantages, reverse correlation is often infeasible to apply because the number of stimulus-response trials needed to provide an accurate estimate is typically very large. Prior approaches have aimed to overcome this sampling inefficiency by incorporating prior knowledge of the representation, which biases the estimate and ultimately limits the essential power of reverse correlation. The present approach, however, improves efficiency via stimulus whitening, a statistical procedure that decorrelates stimuli, making them less redundant, and commensurately more favorable for efficient estimation of an arbitrary target. We provide a mathematical justification for whitening, and demonstrate in simulation that whitening provides greater than 85% improvement in efficiency for a given estimation accuracy, and also a two- to five-fold increase in accuracy for a given sample size. Improving the efficiency of reverse correlation may enable a broader scope of investigations into individual variability and potential universality of perceptual mechanisms.

Despite the impacts of neurodegeneration on speech function, little is known about how to comprehensively characterize the resulting speech abnormalities using a set of objective measures. Quantitative phenotyping of speech motor impairments may have important implications for identifying clinical syndromes and their underlying etiologies, monitoring disease progression over time, and improving treatment efficacy. The goal of this research was to investigate the validity and classification accuracy of comprehensive acoustic-based articulatory phenotypes in speakers with distinct neurodegenerative diseases. Articulatory phenotypes were characterized based on acoustic features that were selected to represent five components of motor performance: Coordination, Consistency, Speed, Precision, and Rate. The phenotypes were first used to characterize the articulatory abnormalities across four progressive neurologic diseases known to have divergent speech motor deficits: amyotrophic lateral sclerosis (ALS), progressive ataxia (PA), Parkinson's disease (PD), and the nonfluent variant of primary progressive aphasia and progressive apraxia of speech (nfPPA + PAOS). We then examined the efficacy of articulatory phenotyping for disease classification. Acoustic analyses were conducted on audio recordings of 217 participants (i.e., 46 ALS, 52 PA, 60 PD, 20 nfPPA + PAOS, and 39 controls) during a sequential speech task. Results revealed evidence of distinct articulatory phenotypes for the four clinical groups and that the phenotypes demonstrated strong classification accuracy for all groups except ALS. Our results highlight the phenotypic variability present across neurodegenerative diseases, which, in turn, may inform (1) the differential diagnosis of neurological diseases and (2) the development of sensitive outcome measures for monitoring disease progression or assessing treatment efficacy.

Goal: Smartphones can be used to passively assess and monitor patients’ speech impairments caused by ailments such as Parkinson’s disease, Traumatic Brain Injury (TBI), Post-Traumatic Stress Disorder (PTSD) and neurodegenerative diseases such as Alzheimer’s disease and dementia. However, passive audio recordings in natural settings often capture the speech of non-target speakers (cross-talk). Consequently, speaker separation, which identifies the target speakers’ speech in audio recordings with two or more speakers’ voices, is a crucial pre-processing step in such scenarios. Prior speech separation methods analyzed raw audio. However, in order to preserve speaker privacy, passively recorded smartphone audio and machine learning-based speech assessment are often performed on derived speech features such as Mel-Frequency Cepstral Coefficients (MFCCs). In this paper, we propose a novel Deep MFCC bAsed SpeaKer Separation (Deep-MASKS). Methods: Deep-MASKS uses an autoencoder to reconstruct MFCC components of an individual’s speech from an i-vector, x-vector or d-vector representation of their speech learned during the enrollment period. Deep-MASKS utilizes a Deep Neural Network (DNN) for MFCC signal reconstructions, which yields a more accurate, higher-order function compared to prior work that utilized a mask. Unlike prior work that operates on utterances, Deep-MASKS operates on continuous audio recordings. Results: Deep-MASKS outperforms baselines, reducing the Mean Squared Error (MSE) of MFCC reconstruction by up to 44% and the number of additional bits required to represent clean speech entropy by 36%.

Background and aims: Autoimmune hepatitis (AIH) has been well characterised and codified through the development of diagnostic criteria. These criteria have been adapted and simplified and are widely used in clinical practice. However, there is a need to update and precisely define the criteria for both treatment response and treatment. Methods: a systematic review was performed and a modified Delphi consensus process was used to identify and redefine the response criteria in autoimmune hepatitis. Results: the consensus process initiated by the International Autoimmune Hepatitis Group proposes that the term 'complete biochemical response' defined as 'normalization of serum transaminases and IgG below the upper limit of normal' be adopted to include a time point at 6 months after initiation of treatment. An insufficient response by 6 months was a failure to meet the above definition. Non-response was defined as ', YAEL Foundation

Uncovering high-level cognitive representations of categories such as faces is an elusive goal that has been frequently sought using reverse correlation, a technique employed in fields ranging from neurophysiology to cognitive psychology. In reverse correlation, subjects are asked to make perceptual judgments (e.g., “do you see a face?”) about richly varying stimuli (e.g., white noise), and observed responses are then regressed against stimuli to yield reconstructions of the underlying cognitive representation. However, many thousands of stimulus-response pairs are frequently required, which severely limits the breadth of studies that are feasible using this powerful method. Techniques that are currently employed to improve efficiency, such as filtering the reconstruction, nevertheless bias the outcome. Here, we show that an advanced signal processing technique for improving sampling efficiency – compressive sensing – is directly compatible with reverse correlation. A trio of simulations are performed to demonstrate that compressive sensing can reduce the required stimulus-response pairs by up to 90% without biasing the reconstruction or can retrospectively improve the accuracy of the reconstructions on existing data. This work concludes by outlining the potential of compressive sensing to improve representation reconstruction throughout the field of neuroscience and beyond.Significance StatementUncovering cognitive representations is an elusive goal that is increasingly pursued using the reverse correlation method, wherein human subjects make judgments about vague stimuli. Employing reverse correlation often entails collecting thousands of stimulus-response pairs, which severely limits the breadth of studies that are feasible using the method. Here we show that this methodological barrier can be overcome using compressive sensing, an advanced signal processing technique designed to improve sampling efficiency. Three sets of simulations are performed to demonstrate that compressive sensing can improve the accuracy of reconstructed cognitive representations and dramatically reduce the required number of stimulus-response pairs. This work concludes by outlining the potential of compressive sensing to improve representation reconstruction throughout the field of neuroscience and beyond.

Tone quality termed “dark” is an aesthetically important property of Western classical voice performance and has been associated with lowered formant frequencies, lowered larynx, and widened pharynx. The present study uses real-time magnetic resonance imaging with synchronous audio recordings to investigate dark tone quality in four professionally trained sopranos with enhanced ecological validity and a relatively complete view of the vocal tract. Findings differ from traditional accounts, indicating that labial narrowing may be the primary driver of dark tone quality across performers, while many other aspects of vocal tract shaping are shown to differ significantly in a performer-specific way.

Modern operational environments can place significant demands on a service member's cognitive resources, increasing the risk of errors or mishaps due to overburden. The ability to monitor cognitive burden and associated performance within operational environments is critical to improving mission readiness. As a key step toward a field-ready system, we developed a simulated marksmanship scenario with an embedded working memory task in an immersive virtual reality environment. As participants performed the marksmanship task, they were instructed to remember numbered targets and recall the sequence of those targets at the end of the trial. Low and high cognitive load conditions were defined as the recall of three- and six-digit strings, respectively. Physiological and behavioral signals recorded included speech, heart rate, breathing rate, and body movement. These features were input into a random forest classifier that significantly discriminated between the low- and high-cognitive load conditions (AUC = 0.94). Behavioral features of gait were the most informative, followed by features of speech. We also showed the capability to predict performance on the digit recall (AUC = 0.71) and marksmanship (AUC = 0.58) tasks. The experimental framework can be leveraged in future studies to quantify the interaction of other types of stressors and their impact on operational cognitive and physical performance.

Lapses in vigilance and slowed reactions due to mental fatigue can increase risk of accidents and injuries and degrade performance. This paper describes a method for rapid, unobtrusive detection of mental fatigue based on changes in electrodermal arousal (EDA), and changes in neuromotor coordination derived from speaking. Twenty-nine Soldiers completed a 2-hour battery of cognitive tasks intended to induce fatigue. Behavioral markers derived from audio and video during speech were acquired before and after the 2hour cognitive load tasks, as was EDA. Exposure to cognitive load produced detectable increases in neuromotor variability in speech and facial measures after load and even after a recovery period. A Gaussian mixture model classifier with crossvalidation and fusion across speech, video, and EDA produced an accuracy of AUC=0.99 in detecting a change in cognitive fatigue relative to a personalized baseline.

none 55 si First published: 22 December 2016 Background: High-quality data on the management of autoimmune hepatitis (AIH) are scarce. Despite published guidelines, management of AIH is still expert based rather than evidence based. Aim: To survey expert hepatologists, asking each to describe their practices in the management of patients with AIH. Methods: A survey questionnaire was distributed to members of the International AIH Group. The questionnaire consisted of four clinical scenarios on different presentations of AIH. Results: Sixty surveys were sent, out of which 37 were returned. None reported budesonide as a first line induction agent for the acute presentation of AIH. Five (14%) participants reported using thiopurine S-methyltransferase measurements before commencement of thiopurine maintenance therapy. Thirteen (35%) routinely perform liver biopsy at 2 years of biochemical remission. If histological inflammatory activity is absent, four (11%) participants reduced azathioprine, whereas 10 (27%) attempted withdrawal altogether. Regarding the management of difficult-to-treat patients, mycophenolate mofetil is the most widely used second-line agent (n = ~450 in 28 centres), whereas tacrolimus (n = ~115 in 21 centres) and ciclosporin (n = ~112 in 18 centres) are less often reported. One centre reported considerable experience with infliximab, while rescue therapy with rituximab has been tried in seven centres. Conclusions: There is a wide variation in the management of patients with autoimmune hepatitis even among the most expert in the field. Although good quality evidence is lacking, there is considerable experience with second-line therapies. Future prospective studies should address these issues, so that we move from an expert- to an evidence- and personalised-based care in autoimmune hepatitis. mixed R. Liberal; Y. S. de Boer; R. J. Andrade; G. Bouma; G. N. Dalekos; A. Floreani; D. Gleeson; G. M. Hirschfield; P. Invernizzi; M. Lenzi; A. W. Lohse; G. Macedo; P. Milkiewicz; B. Terziroli; B. van Hoek; J. M. Vierling; M. A. Heneghan; on behalf of the International Autoimmune Hepatitis Group (IAIHG); A. Pares; A. J. Montano-Loza; A. Granito; A. E. Kremer; A. Floreani; A. W. Lohse; B. van Hoek; B. Terziroli; C. Schramm; C. Lammert; E. L. R. Cancado; E. Bjornsson; F. Lammert; G. N. Dalekos; G. Bouma; G. M. Hirschflield; G. Macedo; H. Hofer; I. Mackay; J. Neuberger; J. M. Vierling; J.P.H. Drenth; C.M.J. van Nieuwkerk; L. Muratori; M. Lenzi; M. Swain; M. Heneghan; M. Zeniya; O. Chazouilleres; P. Gines; P. Invernizzi; P. Milkiewicz; R. J. Andrade; R. Chapman; T. Berg; H. van Buuren; X. Ma R. Liberal; Y. S. de Boer; R. J. Andrade; G. Bouma; G. N. Dalekos; A. Floreani; D. Gleeson; G. M. Hirschfield; P. Invernizzi; M. Lenzi; A. W. Lohse; G. Macedo; P. Milkiewicz; B. Terziroli; B. van Hoek; J. M. Vierling; M. A. Heneghan; on behalf of the International Autoimmune Hepatitis Group (IAIHG); A. Pares; A. J. Montano-Loza; A. Granito; A. E. Kremer; A. Floreani; A. W. Lohse; B. van Hoek; B. Terziroli; C. Schramm; C. Lammert; E. L. R. Cancado; E. Bjornsson; F. Lammert; G. N. Dalekos; G. Bouma; G. M. Hirschflield; G. Macedo; H. Hofer; I. Mackay; J. Neuberger; J. M. Vierling; J.P.H. Drenth; C.M.J. van Nieuwkerk; L. Muratori; M. Lenzi; M. Swain; M. Heneghan; M. Zeniya; O. Chazouilleres; P. Gines; P. Invernizzi; P. Milkiewicz; R. J. Andrade; R. Chapman; T. Berg; H. van Buuren; X. Ma

Current clinical tests lack the sensitivity needed for detecting subtle balance impairments associated with mild traumatic brain injury (mTBI). Patient-reported symptoms can be significant and have a huge impact on daily life, but impairments may remain undetected or poorly quantified using clinical measures. Our central hypothesis was that provocative sensorimotor perturbations, delivered in a highly instrumented, immersive virtual environment, would challenge sensory subsystems recruited for balance through conflicting multi-sensory evidence, and therefore reveal that not all subsystems are performing optimally. The results show that, as compared to standard clinical tests, the provocative perturbations illuminate balance impairments in subjects who have had mild traumatic brain injuries. Perturbations delivered while subjects were walking provided greater discriminability (average accuracy ≈ 0.90) than those delivered during standing (average accuracy ≈ 0.65) between mTBI subjects and healthy controls. Of the categories of features extracted to characterize balance, the lower limb accelerometry-based metrics proved to be most informative. Further, in response to perturbations, subjects with an mTBI utilized hip strategies more than ankle strategies to prevent loss of balance and also showed less variability in gait patterns. We have shown that sensorimotor conflicts illuminate otherwise-hidden balance impairments, which can be used to increase the sensitivity of current clinical procedures. This augmentation is vital in order to robustly detect the presence of balance impairments after mTBI and potentially define a phenotype of balance dysfunction that enhances risk of injury.

This paper reviews the current state of several formal models of speech motor control, with particular focus on the low-level control of the speech articulators. Further development of speech motor control models may be aided by a comparison of model attributes. The review builds an understanding of existing models from first principles, before moving into a discussion of several models, showing how each is constructed out of the same basic domain-general ideas and components-e.g., generalized feedforward, feedback, and model predictive components. This approach allows for direct comparisons to be made in terms of where the models differ, and their points of agreement. Substantial differences among models can be observed in their use of feedforward control, process of estimating system state, and method of incorporating feedback signals into control. However, many commonalities exist among the models in terms of their reliance on higher-level motor planning, use of feedback signals, lack of time-variant adaptation, and focus on kinematic aspects of control and biomechanics. Ongoing research bridging hybrid feedforward/feedback pathways with forward dynamic control, as well as feedback/internal model-based state estimation, is discussed.

This paper reviews the current state of several formal models of speech motor control with particular focus on the low level control of the speech articulators. Further development of speech motor control models may be aided by a comparison of model attributes. The review builds an understanding of existing models from first principles, before moving into a discussion of several models, showing how each is constructed out of the same basic domain-general ideas and components – e.g., generalized feedforward, feedback, and model predictive components. This approach allows for direct comparisons to be made in terms of where the models differ, and their points of agreement. Substantial differences among models can be observed in their use of feedforward control, process of estimating system state, and method of incorporating feedback signals into control. However, many commonalities exist among the models in terms of their reliance on higher-level motor planning, use of feedback signals, lack of time-variant adaptation, and focus on kinematic aspects of control and biomechanics. Ongoing research bridging hybrid feedforward/feedback pathways with forward dynamic control, as well as feedback/internal model-based state estimation is discussed.

USC-TIMIT is an extensive database of multimodal speech production data, developed to complement existing resources available to the speech research community and with the intention of being continuously refined and augmented. The database currently includes real-time magnetic resonance imaging data from five male and five female speakers of American English. Electromagnetic articulography data have also been presently collected from four of these speakers. The two modalities were recorded in two independent sessions while the subjects produced the same 460 sentence corpus used previously in the MOCHA-TIMIT database. In both cases the audio signal was recorded and synchronized with the articulatory data. The database and companion software are freely available to the research community. (C) 2014 Acoustical Society of America.

Speech control models founded on principles of neuroscience have potential to create powerful clinical tools for personalized medicine and for understanding behavior. We explore two prominent speech control models: Directions into Velocities of Articulators (DIVA) (Guenther, 2006) and Task Dynamics (TD) (Saltzman, 1991) toward an understanding of speech disfluency of autism spectrum disorder (ASD). To introduce perturbations into both systems in the common framework of (Parrrell, 2018), we translate the TD model into MATLAB Simulink. For word stimuli, we compare both models’ acoustic outputs, under the hypothesis that speech impediments in ASD arise from overreliance on delayed sensory feedback (Lin, 2015). For DIVA, the output from a 50ms auditory feedback delay contain increased formant production errors and formant oscillations at about 10Hz, sometimes observed in disfluent ASD cases, but also seen in stuttering (Civier, 2010). The same delay introduced in the proprioceptive and tactile feedback in TD takes 5x the time to settle at the target, creating about 20Hz oscillations in TD articulators. Though neither fully capturing disfluency in ASD, together these models lay the groundwork for a more complete neurocomputational speech control model that can be used to longitudinally track and potentially guide ASD speech therapy.

HighlightsA practical feature-level and score-level fusion approach for speaker verification with articulatory information.Concatenating real articulatory measurements with MFCCs improves the performance.Concatenating acoustic-to-articulatory inversion features with MFCCs also improves the result. We propose a practical, feature-level and score-level fusion approach by combining acoustic and estimated articulatory information for both text independent and text dependent speaker verification. From a practical point of view, we study how to improve speaker verification performance by combining dynamic articulatory information with the conventional acoustic features. On text independent speaker verification, we find that concatenating articulatory features obtained from measured speech production data with conventional Mel-frequency cepstral coefficients (MFCCs) improves the performance dramatically. However, since directly measuring articulatory data is not feasible in many real world applications, we also experiment with estimated articulatory features obtained through acoustic-to-articulatory inversion. We explore both feature level and score level fusion methods and find that the overall system performance is significantly enhanced even with estimated articulatory features. Such a performance boost could be due to the inter-speaker variation information embedded in the estimated articulatory features. Since the dynamics of articulation contain important information, we included inverted articulatory trajectories in text dependent speaker verification. We demonstrate that the articulatory constraints introduced by inverted articulatory features help to reject wrong password trials and improve the performance after score level fusion. We evaluate the proposed methods on the X-ray Microbeam database and the RSR 2015 database, respectively, for the aforementioned two tasks. Experimental results show that we achieve more than 15% relative equal error rate reduction for both speaker verification tasks.

This paper describes a spatio-temporal registration approach for speech articulation data obtained from electromagnetic articulography (EMA) and real-time Magnetic Resonance Imaging (rtMRI). This is motivated by the potential for combining the complementary advantages of both types of data. The registration method is validated on EMA and rtMRI datasets obtained at different times, but using the same stimuli. The aligned corpus offers the advantages of high temporal resolution (from EMA) and a complete mid-sagittal view (from rtMRI). The co-registration also yields optimum placement of EMA sensors as articulatory landmarks on the magnetic resonance images, thus providing richer spatio-temporal information about articulatory dynamics. (C) 2014 Acoustical Society of America

The English past tense allomorph following a coronal stop (e.g., /bɑndəd/) includes a vocoid that has traditionally been transcribed as a schwa or as a barred i. Previous evidence has suggested that this entity does not involve a specific articulatory gesture of any kind. Rather, its presence may simply result from temporal coordination of the two temporally adjacent coronal gestures, while the interval between those two gestures remains voiced and is acoustically reminiscent of a schwa. The acoustic and articulatory characteristics of this vocoid are reexamined in this work using real-time MRI with synchronized audio which affords complete midsagittal views of the vocal tract. A novel statistical analysis is developed to address the issue of articulatory targetlessness based on previous models that predict articulatory action from segmental context. Results reinforce the idea that this vocoid is different, both acoustically and articulatorily, than lexical schwa, but its targetless nature is not supported. Data suggest that an articulatory target does exist, especially in the pharynx where it is revealed by the new data acquisition methodology. Moreover, substantial articulatory differences are observed between subjects, which highlights both the difficulty in characterizing this entity previously, and the need for further study with additional subjects.

We address the hypothesis that postures adopted during grammatical pauses in speech production are more "mechanically advantageous" than absolute rest positions for facilitating efficient postural motor control of vocal tract articulators. We quantify vocal tract posture corresponding to inter-speech pauses, absolute rest intervals as well as vowel and consonant intervals using automated analysis of video captured with real-time magnetic resonance imaging during production of read and spontaneous speech by 5 healthy speakers of American English. We then use locally-weighted linear regression to estimate the articulatory forward map from low-level articulator variables to high-level task/goal variables for these postures. We quantify the overall magnitude of the first derivative of the forward map as a measure of mechanical advantage. We find that postures assumed during grammatical pauses in speech as well as speech-ready postures are significantly more mechanically advantageous than postures assumed during absolute rest. Further, these postures represent empirical extremes of mechanical advantage, between which lie the postures assumed during various vowels and consonants. Relative mechanical advantage of different postures might be an important physical constraint influencing planning and control of speech production.

PurposeAdult human vocal tracts display considerable morphological variation across individuals, but the nature and extent of this variation has not been extensively studied for many vocal tract structures. There exists a need to analyze morphological variation and, even more basically, to develop a methodology for morphological analysis of the vocal tract. Such analysis will facilitate fundamental characterization of the speech production system, with broad implications from modeling to explaining interspeaker variability.MethodA data-driven methodology to automatically analyze the extent and variety of morphological variation is proposed and applied to a diverse subject pool of 36 adults. Analysis is focused on two key aspects of vocal tract structure: the midsagittal shape of the hard palate and the posterior pharyngeal wall.ResultPalatal morphology varies widely in its degree of concavity but also in anteriority and sharpness. Pharyngeal wall morphology, by contrast, varies mostly in terms of concavity alone. The distribution of morphological characteristics is complex, and analysis suggests that certain variations may be categorical in nature.ConclusionMajor modes of morphological variation are identified, including their relative magnitude, distribution, and categorical nature. Implications of these findings for speech articulation strategies and speech acoustics are discussed.

In speech production research, the integration of articulatory data derived from multiple measurement modalities can provide rich description of vocal tract dynamics by overcoming the limited spatio-temporal representations offered by individual modalities. This paper presents a spatial and temporal alignment method between two promising modalities using a corpus of TIMIT sentences obtained from the same speaker: flesh point tracking from Electromagnetic Articulography (EMA) that offers high temporal resolution but sparse spatial information and real time Magnetic Resonance Imaging (MRI) that offers good spatial details but at lower temporal rates. Spatial alignment is done by using palate tracking of EMA, but distortion in MRI audio and articulatory data variability make temporal alignment challenging. This paper proposes a novel alignment technique using joint acoustic-articulatory features which combines dynamic time warping and automatic feature extraction from MRI images. Experimental results show that the temporal alignment obtained using this technique is better (12% relative) than that using acoustic feature only.

This paper uses a crowd-sourced definition of a speech phenomenon we have called “focus”. Given sentences, text and speech, in isolation and in context, we asked annotators to identify what we term the “focus” word. We present their consistency in identifying the focused word, when presented with text or speech stimuli. We then build models to show how well we predict that focus word from lexical (and higher) level features. Also, using spectral and prosodic information, we show the differences in these focus words when spoken with and without context. Finally, we show how we can improve speech synthesis of these utterances given focus information. Index Terms: Focus, Context, Speech Synthesis, Prominence, Emphasis, Prosody

We present and evaluate two statistical methods for estimating kinematic relationships of the speech production system: artificial neural networks and locally-weighted regression. The work is motivated by the need to characterize this motor system, with particular focus on estimating differential aspects of kinematics. Kinematic analysis will facilitate progress in a variety of areas, including the nature of speech production goals, articulatory redundancy and, relatedly, acoustic-to-articulatory inversion. Statistical methods must be used to estimate these relationships from data since they are infeasible to express in closed form. Statistical models are optimized and evaluated - using a heldout data validation procedure - on two sets of synthetic speech data. The theoretical and practical advantages of both methods are also discussed. It is shown that both direct and differential kinematics can be estimated with high accuracy, even for complex, nonlinear relationships. Locally-weighted regression displays the best overall performance, which may be due to practical advantages in its training procedure. Moreover, accurate estimation can be achieved using only a modest amount of training data, as judged by convergence of performance. The algorithms are also applied to real-time MRI data, and the results are generally consistent with those obtained from synthetic data.

We present MRI-TIMIT: a large-scale database of synchronized audio and real-time magnetic resonance imaging (rtMRI) data for speech research. The database currently consists of speech data acquired from two male and two female speakers of American English. Subjects’ upper airways were imaged in the midsagittal plane while reading the same 460 sentence corpus used in the MOCHA-TIMIT corpus [1]. Accompanying acoustic recordings were phonemically transcribed using forced alignment. Vocal tract tissue boundaries were automatically identified in each video frame, allowing for dynamic quantification of each speaker’s midsagittal articulation. The database and companion toolset provide a unique resource with which to examine articulatory-acoustic relationships in speech production. Index Terms: speech production, speech corpora, real-time MRI, multi-modal database, large-scale phonetic tools

Individuals who have undergone laryngectomy often rely on handheld transducers (i.e., the electrolarynx) to excite the vocal tract and produce speech. Widely used electrolarynx designs are limited, in that they require manual control of voice activity and pitch modulation. It would be advantageous to have an interface that requires less training, perhaps using the remaining, intact speech production system as a scaffold. Strong evidence exists that aspects of head motion and facial gestures are highly correlated with gestures of voicing and pitch. Therefore, the goal of project MANATEE is to develop an electrolarynx control interface which takes advantage of those correlations. The focus of the current study is to determine the feasibility of using head and facial features to accurately and efficiently modulate the pitch of speaker's electrolarynx in real time on a mobile platform using the built-in video camera. A prototype interface, capable of running on desktop machines and compatible Android devices, is implemented using OpenCV for video feature extraction and statistical prediction of the electrolarynx control signal. Initial performance evaluation is promising, showing pitch prediction accuracies at double the chance-level baseline, and prediction delays well below the perceptually-relevant, ~50 ms threshold.

SUMMARY For early vertebrates, a long-standing hypothesis is that vertebrae evolved as a locomotor adaptation, stiffening the body axis and enhancing swimming performance. While supported by biomechanical data, this hypothesis has not been tested using an evolutionary approach. We did so by extending biomimetic evolutionary analysis (BEA), which builds physical simulations of extinct systems, to include use of autonomous robots as proxies of early vertebrates competing in a forage navigation task. Modeled after free-swimming larvae of sea squirts (Chordata, Urochordata), three robotic tadpoles (`Tadros'), each with a propulsive tail bearing a biomimetic notochord of variable spring stiffness, k (N m-1), searched for, oriented to, and orbited in two dimensions around a light source. Within each of ten generations, we selected for increased swimming speed, U (m s-1) and decreased time to the light source, t (s),average distance from the source, R (m) and wobble maneuvering, W (rad s-2). In software simulation, we coded two quantitative trait loci (QTL) that determine k: bending modulus, E (Nm-2) and length, L (m). Both QTL were mutated during replication, independently assorted during meiosis and, as haploid gametes, entered into the gene pool in proportion to parental fitness. After random mating created three new diploid genotypes, we fabricated three new offspring tails. In the presence of both selection and chance events(mutation, genetic drift), the phenotypic means of this small population evolved. The classic hypothesis was supported in that k was positively correlated (r2=0.40) with navigational prowess, NP, the dimensionless ratio of U to the product of R, t and W. However, the plausible adaptive scenario, even in this simplified system, is more complex, since the remaining variance in NP was correlated with the residuals of R and U taken with respect to k, suggesting that changes in k alone are insufficient to explain the evolution of NP.

Professional singers are trained to maximize vowel duration and minimize consonant interference, while still maintaining intelligibility. The mechanisms by which they do this, however, are unclear. A deeper understanding of the gestural mechanisms utilized during professional-quality singing could be useful to train singers more effectively. It has been well-established that nasal sounds in syllable coda position have longer and larger velum motion in formal speech as compared to casual speech. The hypothesis of this study was that coda velum gestures would be temporally extended proportional to the duration of the entire syllable. To test this, real-time magnetic resonance (rtMR) images of trained soprano singers were used to track the motion of the velum. The boundaries of the vocal tract were mapped to a semi-polar grid, and the velocity of the velum at every frame of its movement was calculated. An important result shows that operatic singing has minimal change in velum gesture duration compared to speech, while a musical theater style can have velum lowering for the duration of a word. Observed gestural kinematics and their implications for speech are discussed from within the framework and perspective of Articulatory Phonology.

The tongue varies across speakers in terms of the proportion of the overall speech production apparatus that it occupies. Differences in tongue size have the potential to result in speaker-specific articulatory strategies for shaping the vocal tract area function and, in turn, individual patterns of vowel acoustics. The present study examines the interplay between relative tongue size and vowel production using real-time magnetic resonance imaging with synchronous audio. Two populations of native American English subjects are considered, one containing healthy adult speakers with no relevant pathologies, and another containing speakers who had undergone glossectomy as treatment for tongue cancer. All subjects were imaged in the midsagittal plane while reading phonetically balanced English sentences. The size of the tongue and the speech production apparatus were quantified from an overall average posture, and their ratio was correlated with the shape of the vowel space in terms of acoustics (e.g., formant frequencies), constrictions (i.e., location and degree of minimum constriction), and parameterized vocal tract cross-distance functions. Results indicate that relative tongue size can be used to explain and predict observable interspeaker differences in vowel production.

We investigate articulatory behavior in post-glossectomy speech using real-time magnetic resonance imaging. Our data reveal that listeners judge speech produced by partial-glossectomy patients as atypical when the surgical procedure affected the oral tongue. Speech produced by patients whose procedure affected the base of tongue, however, was judged as typical. We observe that preservation and compensation mechanisms are exhibited by the patients with atypical speech. They preserve appropriate modulation of F1 using tongue and/or jaw height despite inability to appropriately modulate F2 due to the reduced size and/or mobility of the tongue. Further, durational differences between tense and lax vowels are maintained. The preservation of these features serves as evidence in support of a framework within which individual gestural parameters are independently controlled; when achievement of a particular parameter specification (e.g., constriction location) is compromised, the remaining (e.g., constriction degree, activation duration) are unchanged. Compensatory behavior is exhibited when coronal tongue movement has been impeded and is exemplified by (i) production of labiodental stops in place of target coronal stops and laterals and (ii) forming a velar constriction to produce frication in place of the alveolar frication for /s/.

We present a computational model of lenition-based sound change. Speech production targets for constriction degree are modeled by differential equations with a single stable fixed point at the target constriction degree that interact with higher order equations that reflect prosodically conditioned variation. This output is then input to the articulatory-to-acoustic forward map, the quantal nature of which causes continuous variation in constriction degree to result in two outcomes: closure or spirantization. The quantized acoustic outcomes are then used as the input for the language learner, who must recover the produced constriction degree in the ambient language environment to produce the speech unit appropriately. We model this recovery as estimating the parameters of an articulatory distribution that is most likely to have produced the acoustic observations (i.e., the maximum likelihood estimate). In particular, we optimize the parameters of a Gaussian distribution with respect to their log-likelihoo...

An articulatory study of sibilant fricatives is described, with the goal of describing variability in lingual articulation across languages. Real-time Magnetic Resonance Imaging (rtMRI) data were collected from three speakers each of English and Mandarin and two speakers of Serbian and reconstructed at a rate of 22.4 frames per second. Parallel acoustic data were also collected and subsequently denoised. Subjects spoke the segments /s/ and /S/ in symmetrical vowel contexts (e.g. 'pa sap' for English and 'asa' for Mandarin). Articulation was analyzed using a semi-polar grid overlaid on the image plane and midsagittal distance functions were obtained by measuring cross distances at ~0.5 cm intervals from the glottis to the lips. Analysis shows that place of articulation for /s/ is more anterior compared to /S/ across languages. Apical articulation is observed for /s/ across languages, while /S/ is produced laminally in English and apically in Serbian and Mandarin. Patterns of tongue shaping variability differ, as well, across languages. For instance, higher standard deviation is observed anterior to the place of articulation for /S/ in Mandarin, compared to Serbian and English. [Work supported by NIH]

Articulatory setting postures adopted during speech production are examined with the goal of determining whether setting postures are more mechanically advantageous than rest positions in facilitating motion of vocal tract articulators toward task goals. Articulatory simulations using the Task Dynamics Application (TADA) suggest that setting postures afford large changes with respect to speech tasks for relatively small changes in low-level speech articulators, thus affording greater mechanical advantage as compared to absolute rest postures. This study investigates this hypothesis using real-time Magnetic Resonance Imaging (rtMRI) data of read and spontaneous speech elicited from five healthy speakers of American English. Frames corresponding to inter-speech pauses, speech-ready intervals, and absolute rest intervals were identified and image features were automatically extracted to quantify the vocal tract postures in terms of both task-level constriction variables and articulatory variables. Locally Weighted Regression is then used to estimate the ratio of task velocities to articulator velocities (i.e., the lever or speed ratio) at postures corresponding to the different intervals of interest. Results show substantially higher speed ratios at inter-speech and ready postures as compared to absolute rest postures. [Work supported by NIH.]

The length of the vocal tract and its relationship with formant frequencies is examined at fine temporal scales with the goal of providing accurate estimates of vocal tract length from acoustics on a spectrum-by-spectrum basis despite unknown articulatory information. Accurate vocal tract length estimation is motivated by applications to speaker normalization and biometrics. Analyses presented are both theoretical and empirical. Various theoretical models are used to predict the behavior of vocal tract resonances in the presence of different vocal tract lengths and constrictions. Real-time MRI with synchronized audio is also utilized for detailed measurements of vocal tract length and formant frequencies during running speech, facilitating the examination of short-time changes in vocal tract length and corresponding changes in formant frequencies, both within and across speakers. Previously proposed methods for estimating vocal tract length are placed within a coherent framework, and their effectiveness is evaluated and compared. A data-driven method for VTL estimation emerges as a natural extension of this framework, which is then developed and shown to empirically outperform previous methods on both synthetic and real speech data. A theoretical justification for the effectiveness of this new method is also explained. [Work supported by NIH.]

We propose a method for co-registrating speech articulatory/acoustic data from two modalities that provide complementary advantages. Electromagnetic Articulography (EMA) provides high temporal resolution (100 samples/second in WAVE system) and flesh-point tracking, while real-time Magnetic Resonance Imaging, rtMRI, (23 frames/second) offers a complete midsagittal view of the vocal tract, including articulated structures and the articulatory environment. Co-registration was achieved through iterative alignment in the acoustic and articulatory domains. Acoustic signals were aligned temporally using Dynamic Time Warping, while articulatory signals were aligned variously by minimization of mean total error between articulatometry data and estimated corresponding flesh points and by using mutual information derived from articulatory parameters for each sentence. We demonstrate our method on a subset of the TIMIT corpus elicited from a male and a female speaker of American English, and illustrate the benefits of co-registered multi-modal data in the study of liquid and fricative consonant production in rapid speech. [Supported by NIH and NSF grants.]

Spondees, both true disyllabic words and concatenated monosyllabic word pairs, had their middle section replaced by silence that extended from the midpoint of the first to the midpoint of the second vowel; the silence thus included the final consonant(s) of the first and the initial consonant(s) of the second syllable. These stimuli were presented to normal-hearing young listeners instructed to guess both monosyllabic half words. Input and response spondees were orthographically aligned and analyzed as confusion matrices. Articulatory gestures were estimated for each token via the Haskins Laboratories TADA articulatory synthesis. After time alignment, articulatory distances were calculated for each stimulus-response pair. In the blanked middle of the spondee, phoneme-based confusions of place-of-articulation were high (39% accuracy), while gestural distances underlying this feature (location and degree of tongue tip constriction, lip aperture, and velar constriction) were under 10%. These results suggest ...

It has been shown that subjects may recruit extra-linguistic articulators during the rhythmic production of speech (Tiede, 2010), that sympathetic head movement is associated with dysfluencies in running speech (Hadar, 1984), and that the magnitude of this activity positively correlates with both speaking rate and increased effort in accelerating speech production tasks (Hadar, 1991; Tiede, 2011). It is unclear at what level of linguistic organization this entrainment occurs. We examined patterns of articulation and head movement during an accelerated repetitive speech task involving multi-gestural segments, in order to gain more insights into the coordinative bases of this behavior. Real-time magnetic resonance imaging was used to examine the production of paired English words, contrasting along various articulations, e.g., “cop-top,” “kid-kim,” “flee-free,” “muck-duck.” Subjects' upper airways were imaged in the midsagittal plane while producing trochaic repetitions of each word pair in time to an accel...

Most examinations of glottal abduction and adduction during speech have employed laryngoscopic video or transillumination (Hoole, 2006). While these provide accurate information about timing of glottal movements, they are invasive and cannot provide absolute measurements about glottal width. At the same time, recent medical studies have used ultrasonic imaging to accurately capture glottal movements and laryngeal anatomy [Hu, J. Ultrasound Med. (2010); Jadcherla etal., Dysphagia (2006)]. We demonstrate novel methods of using ultrasound to measure both temporal and spatial aspects of glottal movements during speech. While previous work on glottal ultrasound has been limited by the need to manually analyze each acquired frame, we present methods to automatically quantify glottal aperture in ultrasound images. Finally, since glottal ultrasound does not interfere with the acquisition of supra-laryngeal articulatory data, we present the results of preliminary experiments that record laryngeal and supra-larynge...

The human speech production system can be fundamentally characterized by the kinematic relationships between low-level articulator variables and relatively high-level tasks. Such kinematics can be illuminating about many system aspects from degrees of freedom and redundancy to dynamics and even control. Since these relationships are generally complex and infeasible to express in closed form, recent work has focused on statistical methods for estimating the relevant relationships from data (Saltzman, 2006, in Dynamics of Speech Production and Perception; Lammert, 2010, Proc. INTERSPEECH). Such methods have been applied to synthetic speech data in order to evaluate their effectiveness, but they have not yet been demonstrated on real data. Here, we apply these methods to real speech data acquired from real-time magnetic resonance imaging. We extract articulator variables that are consistent with those chosen by various articulatory models, and we relate them to high-level task variables such as constriction locations and degrees and formant frequencies. This is done to facilitate an analysis of articulatory motor control during speech production. [Work supported by NIH.]

Articulometry studies have demonstrated that speech errors often involve gestural intrusions that are coproduced with other gestures and can go unnoticed because they are not always perceived to be segmental errors. [Mowrey and MacKay (1990)]. Real time magnetic resonance imaging was used to examine tongue movement during the production of English minimal pairs “cop‐top,” “kid‐kim,” “sop‐shop,” and “bad‐bang.” Subjects’ upper airways were imaged in the midsagittal plane while they uttered repetitions of each word pair as rapidly as possible for 15 sec. Degree of constriction in each region of interest (labial, alveolar, post‐alveolar, and velic) was automatically tracked with direct image processing techniques. We calculate the mean intensity of temporally correlated pixels in that region–a metric which has been found to provide a robust estimate of constriction degree in noisy data. [Lammert (in preparation).] Gestural intrusions are observed throughout these data; frequency of occurrence increases with speech rate. Intrusions are invariably synchronous with the alternating contrastive gesture, e.g., additional velic gestures are observed at the same time as the tongue tip constriction in the word pair “cop‐top.” These data are consistent with the hypothesis that speech production involves the coordination of dynamically defined action units. [Goldstein et al. (2007).]