Search

Automatic speech recognition (ASR) can be deployed in a previously unknown language, in less than 24 h, given just three resources: an acoustic model trained on other languages, a set of language-model training data, and a grapheme-to-phoneme (G2P) transducer to connect them. The LanguageNet G2Ps were created with the goal of being small, fast, and easy to port to a previously unseen language. Data come from pronunciation lexicons if available, but if there are no pronunciation lexicons in the target language, then data are generated from minimal resources: from a Wikipedia description of the target language, or from a one-hour interview with a native speaker of the language. Using such methods, the LanguageNet G2Ps now include simple models in nearly 150 languages, with trained finite state transducers in 122 languages, 59 of which are sufficiently well-resourced to permit measurement of their phone error rates. This paper proposes a measure of the distance between the G2Ps in different languages, and demonstrates that agglomerative clustering of the LanguageNet languages bears some resemblance to a phylogeographic language family tree. The LanguageNet G2Ps proposed in this paper have already been applied in three cross-language ASRs, using both hybrid and end-to-end neural architectures, and further experiments are ongoing.

Browsing large audio archives is challenging because of the limitations of human audition and attention. However, this task becomes easier with a suitable visualization of the audio signal, such as a spectrogram transformed to make unusual audio events salient. This transformation maximizes the mutual information between an isolated event's spectrogram and an estimate of how salient the event appears in its surrounding context. When such spectrograms are computed and displayed with fluid zooming over many temporal orders of magnitude, sparse events in long audio recordings can be detected more quickly and more easily. In particular, in a 1/10-real-time acoustic event detection task, subjects who were shown saliency-maximized rather than conventional spectrograms performed significantly better. Saliency maximization also improves the mutual information between the ground truth of nonbackground sounds and visual saliency, more than other common enhancements to visualization.

We have built a virtual-reality simulator to design quadrotor robots that increase the independence of elderly persons living alone. The simulator lets us measure under what conditions a robot makes someone feel intrigued, or alarmed, or unconcerned, extending to nonhumanoids the studies of emotion that have been done for robots with facial expressions. This better model of how the virtual robot is perceived lets us agilely improve its appearance and behavior, containing the combinatorial explosion of parameters to study in costlier real-world experiments. The residence and the robot are rendered with Unity, and viewed with an Oculus Rift. Audio corresponding to flight maneuvers is resynthesized from recordings of an actual quadrotor. Flight dynamics using L1 adaptive control run in MATLAB and Simulink. Through menus operated by a Leap Motion hand tracker, the human subject commands the quadrotor to fly to various rooms (Unity waypoints), along precomputed paths that avoid collisions. The flight path is varied through a mathematical mapping to Laban Motion Factors (e.g., directness, sustainedness, lightness, and boundness) to provoke a range of responses. The subject's emotion is estimated from heart rate, skin conductance, and head tilt from the Rift's IMU (which is a quick metric for discomfort level).

Distributed scene graphs are important in virtual reality, both in collaborative virtual environments and in cluster rendering. Modern scalable visualization systems have high local throughput, but collaborative virtual environments (VEs) over a wide-area network (WAN) share data at much lower rates. This complicates the use of one scene graph across the whole application. Myriad is an extension of the Syzygy VR toolkit in which individual scene graphs form a peer-to-peer network. Myriad connections filter scene graph updates and create flexible relationships between nodes of the scene graph. Myriad's sharing is fine-grained: the properties of individual scene graph nodes to share are dynamically specified (in C++ or Python). Myriad permits transient inconsistency, relaxing resource requirements in collaborative VEs. A test application, WorldWideCrowd, demonstrates collaborative prototyping of a 300-avatar crowd animation viewed on two PC-cluster displays and edited on low-powered laptops, desktops, and over a WAN. We have further used our framework to facilitate collaborative educational experiences and as a vehicle for undergraduates to experiment with shared virtual worlds. Copyright © 2006 John Wiley & Sons, Ltd.

Software-based musical instruments have controls for input, a sound synthesizer for output, and mappings connecting the two. An effective layout of controls considers how many degrees of freedom each has, as well as the overhead of selecting each one while performing. An isolated mapping from one control to one synthesis parameter needs an appropriate choice of proportional, integral or derivative control (the control's value, or that value's rate of change, drives the synthesis parameter's value, or that value's rate of change). Beyond this, a compound mapping cross-coupling several controls and synthesis parameters can surprisingly increase the performer's intuitive understanding of the instrument.

Timeliner is a browser for long audio recordings and features that it derives from such recordings. Features can be either signal-based, like spectrograms, or model-based, like categorical classifiers. Unlike conventional audio editors, Timeliner pans and zooms smoothly across many orders of magnitude, from days-long overviews to millisecond-scale details, with zero latency, zero flicker, and low CPU load. Also, to suggest which details are worth zooming in to examine, Timeliner's agglomerative hierarchical caches propagate feature-specific details up to wider zoom levels. Because these details are not averaged away, "big data" can be browsed rapidly and effectively. Several studies demonstrate this.

We propose a saliency-maximized audio spectrogram as a representation that lets human analysts quickly search for and detect events in audio recordings. By rendering target events as visually salient patterns, this representation minimizes the time and effort needed to examine a recording. In particular, we propose a transformation of a conventional spectrogram that maximizes the mutual information between the spectrograms of isolated target events and the estimated saliency of the overall visual representation. When subjects are shown spectrograms that are saliency-maximized, they perform significantly better in a 1/10-real-time acoustic event detection task.

The Syzygy software library consists of tools for programming VR applications on PC clusters. Since the PC cluster environment presents application development constraints, it is impossible to simultaneously optimize for efficiency, flexibility, and portability between the single-computer and cluster cases. Consequently Syzygy includes two application frameworks: a distributed scene graph framework for rendering a single application's graphics database on multiple rendering clients, and a master/slave framework for applications with multiple synchronized instances. Syzygy includes a simple distributed OS and supports networked input devices, sound renderers, and graphics renderers, all built on a robust networking layer.

We are developing the audio display for a CAVE‐type virtual reality theater, a 3‐m cube with displays covering all six rigid faces. The user’s headgear continuously reports ear positions so headphones would be possible, but we nevertheless prefer loudspeakers because this enhances the sense of total immersion. Because sounds produced at the loudspeakers are distorted by the room impulse responses, we therefore face the problem of controlling the sound at the listener’s two ears. Our proposed solution consists of open‐loop acoustic point control, i.e., dereverberation. The room impulse responses from each loudspeaker to each ear of the listener are inverted using multichannel inversion methods, to create exactly the desired sound field at the listener’s ears. Because the actual room impulse responses cannot be measured in real time (as the listener walks around), instead the impulse responses simulated by the image‐source method is used. A new evaluation criterion is proposed to quantitatively evaluate both the simulation and the open‐loop dereverberation. The actual impulse responses used for this evaluation are measured with a starter pistol, since this best approximates the point source assumed by the image‐source method.

Non‐speech audio event detection (AED) could be used for low‐cost, spatially diffuse surveillance applications, e.g., monitoring of vehicle activity in a national park, or of footsteps in a hallway. Experiments have shown that non‐speech AED benefits from the dynamic inference strategies such as the hidden Markov model (HMM), but that the acoustic features useful for non‐speech events may not be the same as those useful for speech. One possible solution is a tandem HMM: an HMM whose observation vector is constructed from the output of an instantaneous discriminative classifier, e.g., a neural network. The use of tandem HMMs for non‐speech AED is hindered, however, by the relatively small size of most non‐speech‐audio training corpora. This talk will demonstrate that tandem HMMs can be trained to detect non‐speech audio events using a novel form of regularized training: Baum–Welch back‐propagation (as proposed by Bengio et al.), using the conjugate‐gradient adaptive form of the Baum–Welch auxiliary functio...