Search

In this paper, we propose an online adaptation method for Fourier series-based acoustic transfer function (FS-ATF) models for robot audition systems using microphone array signal processing. The ATF represents the characteristics of signal propagation from a sound source to a microphone, which is essential for sound source localization (SSL) and sound source separation (SSS). For practical applications in dynamically changing real-world environments, ATF models must satisfy two criteria: (1) they must be adaptable to changes in the acoustic environment; and (2) they must be lightweight to be suitable for resource-constrained systems, such as robots with limited memory and computational capacity. The proposed method addresses these challenges using Fourier series expansions for interpolation, which reduces the memory footprint of the ATF model and facilitates online adaptation to acoustic environmental changes. The experimental results demonstrate that the proposed online adaptation method both improves the SSL and SSS performance while reducing the size of the ATF model, which represents a significant improvement over existing online ATF adaptation methods. [ABSTRACT FROM AUTHOR]

With the rapid industrialization utilizing multi-rotor drones in recent years, an increase in urban flights is expected in the near future. This may potentially result in noise pollution due to the operation of drones. This study investigates the near- and far-field acoustic characteristics of low-noise propellers inspired by Gurney flaps. In addition, we examine the impact of these low-noise propellers on the sound source localization performance of drones equipped with a microphone array, which are expected to be used for rescuing people in disasters. Results from in-flight noise measurements indicate significant noise reduction mainly in frequency bands above 1 kHz in both the near- and far-field. An improvement in the success rate of sound source localization with low-noise propellers was also observed. However, the influence of the position of the microphone array with respect to the propellers is more pronounced than that of propeller shape manipulation, suggesting the importance of considering the positional relationships. Computational fluid dynamics analysis of the flow field around the propellers suggests potential mechanisms for noise reduction in the developed low-noise propellers. The results obtained in this study hold potential for contributing to the development of integrated drones aimed at reducing noise and improving sound source localization performance.

Locating the sound source is one of the most important capabilities of robot audition. In recent years, single-source localization techniques have increasingly matured. However, localizing and tracking specific sound sources in multi-source scenarios, which is known as the cocktail party problem, is still unresolved. In order to address this challenge, in this paper, we propose a system for dynamically localizing and tracking sound sources based on audio–visual information that can be deployed on a mobile robot. Our system first locates specific targets using pre-registered voiceprint and face features. Subsequently, the robot moves to track the target while keeping away from other sound sources in the surroundings instructed by the motion module, which helps the robot gather clearer audio data of the target to perform downstream tasks better. Its effectiveness has been verified via extensive real-world experiments with a 20% improvement in the success rate of specific speaker localization and a 14% reduction in word error rate in speech recognition compared to its counterparts. [ABSTRACT FROM AUTHOR]

This study is the first to quantitatively measure the courtship display flights of Latham's snipe (Gallinago hardwickii), which is a "near threatened" species as of 2022 (IUCN red list of threatened species). By using a 16‐channel microphone array and 8‐channel microphone arrays, we localized the fine‐scale movements of courtship flights of one male performing at high altitude and high speed, and we estimated the direction from which each sound arrived using robot audition. Preliminary analyses of the azimuthal and elevation angles of the courtship flights partially revealed a fine‐scale flight trajectory. First, a male Latham's snipe gradually gained altitude while vocalizing sharp and harsh repeating calls, until it reached the flight peak altitude, then dove down while producing winnowing sound to the ground along the wetland zones without tall vegetation. This observation method is methodologically useful to establish a better understanding of Latham's snipe courtship flight site selection. Furthermore, this method can be extended to investigate other rare nocturnal or crepuscular birds that are too timid to risk ringing or tagging. [ABSTRACT FROM AUTHOR]

Songbirds are one of the study targets for both bioacoustic and ecoacoustic research. In this paper, we discuss the applicability of robot audition techniques to understand the dynamics of forest bird vocalizations in a soundscape measured in azimuth and elevation angles with a single 16-channel microphone array, using HARK and HARKBird. First, we evaluated the accuracy in estimating the azimuth and elevation angles of bird vocalizations replayed from a loudspeaker on a tree, 6.55 m above the height of the array, from different horizontal distances in a forest. The results showed that the localization error of azimuth and elevation angle was equal to or less than 5 degrees and 15 degrees, respectively, in most of cases when the horizontal distance from the array was equal to or less than 35 m. We then conducted a field observation of vocalizations to monitor birds in a forest. The results showed that the system can successfully detect how birds use the soundscape horizontally and vertically. This can contribute to bioacoustic and ecoacoustic research, including behavioral observations and study of biodiversity. [ABSTRACT FROM AUTHOR]

• A new model based on DRR is proposed to detect room contour with audible sound. • Be insensitive to reverberation time. • Be still effective when signal-to-diffuse noise ratio is as low as −10 dB. • Be validated with real-world experiments under different conditions. An acoustic detection model based on the direct-to-reverberant energy ratio (DRR) is proposed to estimate room shape with audible sound. Besides the direct propagation, the first-order reflection is considered in the calculation of the DRR in this work. Thus, the direct-to-reflected acoustic path ratio is estimated in the proposed model to compute the reflection point, which is used to calculate the pose estimate of the reflector. The mapping of the proposed model is completed by fusing the pose estimates of walls with the spatio-temporal diversity. An initialization process is developed to obtain the initial value of the detection model. An acquisition system working under audible frequencies is built. The experimental results show that even with a low signal-to-diffuse noise ratio (−10 dB), the error of the proposed method is still less than 0.5 degrees in angle and 0.03 m in distance, demonstrating the robustness of the proposed model under the environment with multipath reflections. [ABSTRACT FROM AUTHOR]

To understand the social interactions among songbirds, extracting the timing, position, and acoustic properties of their vocalizations is essential. We propose a framework for automatic and fine-scale extraction of spatial-spectral-temporal patterns of bird vocalizations in a densely populated environment. For this purpose, we used robot audition techniques to integrate information (i.e., the timing, direction of arrival, and separated sound of localized sources) from multiple microphone arrays (array of arrays) deployed in an environment, which is non-invasive. As a proof of concept of this framework, we examined the ability of the method to extract active vocalizations of multiple Zebra Finches in an outdoor mesh tent as a realistic situation in which they could fly and vocalize freely. We found that localization results of vocalizations reflected the arrangements of landmark spots in the environment such as nests or perches and some vocalizations were localized at non-landmark positions. We also classified their vocalizations as either songs or calls by using a simple method based on the tempo and length of the separated sounds, as an example of the use of the information obtained from the framework. Our proposed approach has great potential to understand their social interactions and the semantics or functions of their vocalizations considering the spatial relationships, although detailed understanding of the interaction would require analysis of more long-term recordings.

Locating the sound source is one of the most important capabilities of robot audition. In recent years, single-source localization techniques have increasingly matured. However, localizing and tracking specific sound sources in multi-source scenarios, which is known as the cocktail party problem, is still unresolved. In order to address this challenge, in this paper, we propose a system for dynamically localizing and tracking sound sources based on audio–visual information that can be deployed on a mobile robot. Our system first locates specific targets using pre-registered voiceprint and face features. Subsequently, the robot moves to track the target while keeping away from other sound sources in the surroundings instructed by the motion module, which helps the robot gather clearer audio data of the target to perform downstream tasks better. Its effectiveness has been verified via extensive real-world experiments with a 20% improvement in the success rate of specific speaker localization and a 14% reduction in word error rate in speech recognition compared to its counterparts.

In this article, we present a novel localization method for multiple sources in indoor environments. Our approach can estimate different propagation paths, including the reflection and diffraction paths of sound waves based on a backward ray tracing technique. To estimate diffraction propagation paths, we combine a ray tracing algorithm with a uniform theory of diffraction model by exploiting the diffraction properties as propagation paths bend around the wedges of obstacles. We reconstruct the 3-D environments and wedges of obstacles in the precomputation phase and utilize these outcomes to generate primary, reflection, and diffraction acoustic rays in the runtime phase. We localize multiple sources when identifying the convergence regions of these acoustic rays based on Monte Carlo localization (MCL). Our approach supports not only stationary but also moving sources of human speech and clapping sounds. Our approach can also handle nonline-of-sight (NLOS) sources and distinguish between active and inactive source states. We evaluated and analyzed our algorithm in multiple scenarios containing obstacles and NLOS sources. Our approach can localize moving sources with the average of distance errors of 0.65 and 0.74 m in single and multiple source cases, respectively, in rooms, 7 m by 7 m in size with a height of 3 m; errors are measured according to the L2 distance between the estimated and actual source positions. We observed a 130% improvement of the localization accuracy over the prior work (J.-M. Valin et al.). [ABSTRACT FROM AUTHOR]

Abstract Ecoacoustics needs sophisticated acoustic monitoring tools to extract a wide level of features from an observed mixture of sounds. We have developed a portable acoustic monitoring system called ‘HARKBird’ which consists of a laptop PC and an inexpensive commercial microphone array with the robot audition software HARK. HARKBird can extract acoustic events in a recording, and we can obtain the begin and end timings, the spatial information (e.g., position or direction from the microphone array), and the spectrogram of the sound separated from the original recording. In this study, we report how robot audition techniques contribute to monitoring spatio‐spectro‐temporal dynamics of bird behaviors, using an extended and minimal system based on multiple microphone arrays. The dimension reduction of separated sounds is important to integrate the information from multiple microphone arrays. As a dimension reduction algorithm, we use t‐SNE to help manual annotation of each sound and to generate the vocalization distribution automatically. We conduct playback experiments to Spotted Towhee (Pipilo maculatus) to simulate different cases of territorial intrusions (song/call/no playback). Our hypothesis in playback experiments is that playback of conspecific vocalizations would invoke aggressive responses of males against song playbacks and the effects would be more prominent than those of call playbacks. Our primary aim is to test whether our system can extract the necessary information on the aggressiveness of target individuals to examine our hypothesis. We show the system with manual annotation of vocalizations can extract their different spatio‐spectro‐temporal dynamics in different conditions, which supported our hypothesis. We also consider the spectral affinity‐based automatic matching of localized sounds from different microphone arrays. The relative number of localized songs depending on the playback conditions reflected a similar trend to those in the manual approach, implying that we can grasp the long‐term dynamics of vocalizations without costly annotations.

Sound source localization and separation are essential functions for robot audition to comprehend acoustic environments. The widely-used multiple signal classification (MUSIC) can precisely estimate the directions of arrival (DoAs) of multiple sound sources if its hyperparameters are selected appropriately depending on the surrounding environment. A popular separation method based on a complex Gaussian mixture model (CGMM), on the other hand, can extract multiple sources even in noisy environments if its latent variables are properly initialized to avoid bad local optima. To overcome the drawbacks of both the MUSIC and CGMM, we propose a robot audition framework that complementarily combines the MUSIC and CGMM in a probabilistic manner. Our method is based on a variant of the CGMM conditioned by the localization results of MUSIC. The hyperparameters of MUSIC are estimated by the type II maximum likelihood estimation of the CGMM, and the CGMM itself is efficiently initialized and regularized by using the localization results of MUSIC. Experimental results show that our method outperformed conventional localization and separation methods even when the number of sound sources is unknown. we also demonstrate that our method can work even with moving sound sources in real time.

Artificial audition aims at providing hearing capabilities to machines, computers and robots. Existing frameworks in robot audition offer interesting sound source localization, tracking and separation performance, although involve a significant amount of computations that limit their use on robots with embedded computing capabilities. This paper presents ODAS, the Open embeddeD Audition System framework, which includes strategies to reduce the computational load and perform robot audition tasks on low-cost embedded computing systems. It presents key features of ODAS, along with cases illustrating its uses in different robots and artificial audition applications.

This paper proposes the use of the standing waves created by the interference between transmitted and reflected acoustic signals to recognize the size and the shape of a target object. This study shows that the profile of the distance spectrum generated by the interference encodes not only the distance to the target, but also the distance to the edges of the target surface. To recognize the extent of the surface, a high-resolution distance spectrum is proposed, and a method to estimate the points on the edges by incorporating observations from multiple measurement is introduced. Numerical simulations validated the approach and showed that the method worked even in the presence of noise. Experimental results are also shown to verify that the method works in a real environment.

Bioacoustics monitoring has become increasingly popular for studying the behavior and ecology of vocalizing birds. This study aims to verify the practical effectiveness of localization technology for auditory monitoring of endangered Eurasian bittern (Botaurus stellaris) which inhabits wetlands in remote areas with thick vegetation. Their crepuscular and highly secretive nature, except during the breeding season when they vocalize advertisement calls, make them difficult to monitor. Because of the increasing rates of habitat loss, surveying accurate numbers and their habitat needs are both important conservation tasks. We investigated the feasibility of localizing their booming calls, at a low frequency range between 100–200 Hz, using microphone arrays and robot audition HARK (Honda Research Institute, Audition for Robots with Kyoto University). We first simulated sound source localization of actual bittern calls for microphone arrays of radii 10 cm, 50 cm, 1 m, and 10 m, under different noise levels. Second, we monitored bitterns in an actual field environment using small microphone arrays (height = 12 cm; width = 8 cm), in the Sarobetsu Mire, Hokkaido Island, Japan. The simulation results showed that the spectral detectability was higher for larger microphone arrays, whereas the temporal detectability was higher for smaller microphone arrays. We identified that false detection in smaller microphone arrays, which was coincidentally generated in the calculation proximate to the transfer function for the opposite side. Despite technical limitations, we successfully localized booming calls of at least two males in a reverberant wetland, surrounded by thick vegetation and riparian trees. This study is the first case of localizing such rare birds using small-sized microphone arrays in the field, thereby presenting how this technology could contribute to auditory surveys of population numbers, behaviors, and microhabitat selection, all of which are difficult to investigate using other observation methods. This methodology is not only useful for the better understanding of bitterns, but it can also be extended to investigate other rare nocturnal birds with low-frequency vocalizations, without direct ringing or tagging. Our results also suggest a future necessity for a robust localization system to avoid reverberation and echoing in the field, resulting in the false detection of the target birds.

Abstract Humanoid robots require to use microphone arrays to acquire speech signals from the human communication partner while suppressing noise, reverberation, and interferences. Unlike many other applications, microphone arrays in humanoid robots have to face the restrictions in size and geometry. To address these challenges, this paper presents an approach to differential beamforming with arbitrary planar array geometries. The major contributions of this work are as follows: (1) a method is presented to design differential beamformers, which works for regular geometries such as linear, circular, and concentric circular ones, as well as irregular geometries, as long as the sensors’ positions are given or can be measured; (2) fundamental requirements for the design of different orders of linear differential microphone arrays (DMAs), partially steerable DMAs, fully steerable DMAs, and robust DMAs are discussed; (3) the validity and limitations of the Jacobi-Anger expansion approximation is analyzed, where we discuss how to achieve an optimal approximation by properly choosing the reference point; and (4) we show how to design an Nth-order DMA with 2N microphones using the Jacobi-Anger expansion.

Abstract Ego-noise, i.e., the noise a robot causes by its own motions, significantly corrupts the microphone signal and severely impairs the robot’s capability to interact seamlessly with its environment. Therefore, suitable ego-noise suppression techniques are required. For this, it is intuitive to use also motor data collected by proprioceptors mounted to the joints of the robot since it describes the physical state of the robot and provides additional information about the ego-noise sources. In this paper, we use a dictionary-based approach for ego-noise suppression in a semi-supervised manner: first, an ego-noise dictionary is learned and subsequently used to estimate the ego-noise components of a mixture by computing a weighted sum of dictionary entries. The estimation of the weights is very sensitive against other signals beside ego-noise contained in the mixture. For increased robustness, we therefore propose to incorporate knowledge about the physical state of the robot to the estimation of the weights. This is achieved by introducing a motor data-based regularization term to the estimation problem which promotes similar weights for similar physical states. The regularization is derived by representing the motor data as a graph and imprints the intrinsic structure of the motor data space onto the dictionary model. We analyze the proposed method and evaluate its ego-noise suppression performance for a large variety of different movements and demonstrate the superiority of the proposed method compared to an approach without using motor data.

As the problem of an aging population becomes more and more serious, social robots have an increasingly significant influence on human life. By employing regular question-and-answer conversations or wearable devices, some social robotics products can establish personal health archives. But those robots are unable to collect diet information automatically through robot vision or audition. A healthy diet can reduce a person's risk of developing cancer, diabetes, heart disease, and other age-related diseases. In order to automatically perceive the dietary composition of the elderly by listening to people's chatting, this paper proposed a chat-based automatic dietary composition perception algorithm (DCPA). DCPA uses social robot audition to understand the semantic information and percept dietary composition for Mandarin Chinese. Firstly, based on the Mel-frequency cepstrum coefficient and convolutional neural network, a speaker recognition method is designed to identify speech data. Based on speech segmentation and speaker recognition algorithm, an audio segment classification method is proposed to distinguish different speakers, store their identity information and the sequence of expression in a speech conversation. Secondly, a dietetic lexicon is established, and two kinds of dietary composition semantic understanding algorithms are proposed to understand the eating semantics and sensor dietary composition information. To evaluate the performance of the proposed DCPA algorithm, we implemented the proposed DCPA in our social robot platform. Then we established two categories of test datasets relating to a one-person and a multi-person chat. The test results show that DCPA is capable of understanding users' dietary compositions, with an F1 score of 0.9505, 0.8940 and 0.8768 for one-person talking, a two-person chat and a three-person chat, respectively. DCPA has good robustness for obtaining dietary information.

Songbirds are one of the study targets for both bioacoustic and ecoacoustic research. In this paper, we discuss the applicability of robot audition techniques to understand the dynamics of forest bird vocalizations in a soundscape measured in azimuth and elevation angles with a single 16-channel microphone array, using HARK and HARKBird. First, we evaluated the accuracy in estimating the azimuth and elevation angles of bird vocalizations replayed from a loudspeaker on a tree, 6.55 m above the height of the array, from different horizontal distances in a forest. The results showed that the localization error of azimuth and elevation angle was equal to or less than 5 degrees and 15 degrees, respectively, in most of cases when the horizontal distance from the array was equal to or less than 35 m. We then conducted a field observation of vocalizations to monitor birds in a forest. The results showed that the system can successfully detect how birds use the soundscape horizontally and vertically. This can contribute to bioacoustic and ecoacoustic research, including behavioral observations and study of biodiversity.

This paper proposes a new method that fuses acoustic measurements in the reverberation field and low-accuracy inertial measurement unit (IMU) motion reports for simultaneous localization and mapping (SLAM). Different from existing studies that only use acoustic data for direction-of-arrival (DoA) estimates, the source’s distance from sensors is calculated with the direct-to-reverberant energy ratio (DRR) and applied to eliminate the nonlinear noise from motion reports. A particle filter is applied to estimate the critical distance, which is key for associating the source’s distance with the DRR. A keyframe method is used to eliminate the deviation of the source position estimation toward the robot. The proposed DoA-DRR acoustic SLAM (D-D SLAM) is designed for three-dimensional motion and is suitable for drones. The method is the first acoustic SLAM algorithm that has been validated on a real-world drone dataset that contains only acoustic data and IMU measurements. Compared with previous methods, D-D SLAM has acceptable performance in locating the drone and building a source map from a real-world drone dataset. The average location accuracy is 0.48 m, while the source position error converges to less than 0.25 m within 2.8 s. These results prove the effectiveness of D-D SLAM in real-world scenes.

Sensor array-based systems, which adopt time difference of arrival (TDOA) measurements among the sensors, have found many robotic applications. However, for existing frameworks and systems to be useful, the sensor array needs to be calibrated accurately. Of particular interest in this article are microphone array-based robot audition systems. In our recent work, by using a moving sound source, and the graph-based formulation of simultaneous localization and mapping (SLAM), we have proposed a framework for joint sound source localization and calibration of microphone array geometrical information, together with the estimation of microphone time offset and clock difference/drift rates. However, a thorough study on the identifiability question, termed observability analysis here, in the SLAM framework for microphone array calibration and sound source localization, is still lacking in the literature. In this article, we will fill the abovementioned gap via a Fisher information matrix approach. Motivated by the equivalence between the full column rankness of the Fisher information matrix and the Jacobian matrix, we leverage the structure of the latter associated with the SLAM formulation, and present necessary and sufficient conditions guaranteeing its full column rankness, which lead to parameter identifiability. We have thoroughly discussed the 3-D case with asynchronous (with both time offset and clock drifts, or with only one of them) and synchronous microphone array, respectively. These conditions are closely related to the motion varieties of the sound source and the microphone array configuration, and have intuitive and physical interpretations. Based on the established conditions, we have also discovered some particular cases where observability is impossible. Connections with calibration of other sensors will also be discussed, amongst others. To our best knowledge, this is the first systematic work on observability analysis of SLAM-based microphone array calibration and sound source localization. The tools and concepts used in this article are also applicable to other TDOA sensing modalities such as ultrawide band (UWB) sensors. [ABSTRACT FROM AUTHOR]

Ecoacoustics needs sophisticated acoustic monitoring tools to extract a wide level of features from an observed mixture of sounds. We have developed a portable acoustic monitoring system called 'HARKBird' which consists of a laptop PC and an inexpensive commercial microphone array with the robot audition software HARK. HARKBird can extract acoustic events in a recording, and we can obtain the begin and end timings, the spatial information (e.g., position or direction from the microphone array), and the spectrogram of the sound separated from the original recording. In this study, we report how robot audition techniques contribute to monitoring spatio‐spectro‐temporal dynamics of bird behaviors, using an extended and minimal system based on multiple microphone arrays. The dimension reduction of separated sounds is important to integrate the information from multiple microphone arrays. As a dimension reduction algorithm, we use t‐SNE to help manual annotation of each sound and to generate the vocalization distribution automatically. We conduct playback experiments to Spotted Towhee (Pipilo maculatus) to simulate different cases of territorial intrusions (song/call/no playback). Our hypothesis in playback experiments is that playback of conspecific vocalizations would invoke aggressive responses of males against song playbacks and the effects would be more prominent than those of call playbacks. Our primary aim is to test whether our system can extract the necessary information on the aggressiveness of target individuals to examine our hypothesis. We show the system with manual annotation of vocalizations can extract their different spatio‐spectro‐temporal dynamics in different conditions, which supported our hypothesis. We also consider the spectral affinity‐based automatic matching of localized sounds from different microphone arrays. The relative number of localized songs depending on the playback conditions reflected a similar trend to those in the manual approach, implying that we can grasp the long‐term dynamics of vocalizations without costly annotations. [ABSTRACT FROM AUTHOR]

Robot audition aims at developing robot's ears that work in the real world, that is, machine listening of multiple sound sources. Its critical problem is noise. Speech interfaces have become more familiar and more indispensable as smartphones and artificial intelligence (AI) speakers spread. Their critical problems are noise and multiple simultaneous speakers. Recently two technological advances have contributed to significantly improve the performance of speech interfaces and robot audition. Emerging deep learning technology has improved noise robustness of automatic speech recognition, whereas microphone array processing has improved the performance of preprocessing such as noise reduction. Herein, an overview and history of robot audition are provided together with introduction of an open‐source software for robot audition and its wide applications in the real world. Also, it is discussed how robot audition contributes to the development of computational auditory scene analysis, that is, understanding of real‐world auditory environments.

Abstract Acoustic interactions are important for understanding intra‐ and interspecific communication in songbird communities from the viewpoint of soundscape ecology. It has been suggested that birds may divide up sound space to increase communication efficiency in such a manner that they tend to avoid overlap with other birds when they sing. We are interested in clarifying the dynamics underlying the process as an example of complex systems based on short‐term behavioral plasticity. However, it is very problematic to manually collect spatiotemporal patterns of acoustic events in natural habitats using data derived from a standard single‐channel recording of several species singing simultaneously. Our purpose here was to investigate fine‐scale spatiotemporal acoustic interactions of the great reed warbler. We surveyed spatial and temporal patterns of several vocalizing color‐banded great reed warblers (Acrocephalus arundinaceus) using an open‐source software for robot audition HARK (Honda Research Institute Japan Audition for Robots with Kyoto University) and three new 16‐channel, stand‐alone, and water‐resistant microphone arrays, named DACHO spread out in the bird's habitat. We first show that our system estimated the location of two color‐banded individuals’ song posts with mean error distance of 5.5 ± 4.5 m from the location of observed song posts. We then evaluated the temporal localization accuracy of the songs by comparing the duration of localized songs around the song posts with those annotated by human observers, with an accuracy score of average 0.89 for one bird that stayed at one song post. We further found significant temporal overlap avoidance and an asymmetric relationship between songs of the two singing individuals, using transfer entropy. We believe that our system and analytical approach contribute to a better understanding of fine‐scale acoustic interactions in time and space in bird communities.

This paper proposes an environmental sound segmentation method using Mask U-Net. In recent years, human–robot interactions, especially speech dialogue, have been assessed by auditory scene analysis. Methods, such as noise reduction, section detection, and sound source separation have been proposed for robot audition, acoustic signal processing, and machine learning. However, such conventional approaches have three drawbacks: (1) Many studies have analyzed individual functions, which are regarded as being a cascade. Cascade systems can, however, result in the accumulation of errors generated at each functional block. (2) Unlike conventional cascade systems, deep learning-based methods that simultaneously detect sections, separate sound sources, and identify classes have also been proposed for speech separation. These techniques can be extended for multiple classes of environmental sounds, but their performance becomes degraded with large variations in sound event lengths among classes. (3) In addition, these methods have recurrent neural network layers making it difficult to process calculations in parallel. This paper proposes an environmental sound segmentation method called Mask U-Net, which robustly differentiates sound event lengths among classes. Simulation experiments using a developed 75-class environmental sound data set showed that the proposed method was faster than conventional methods and showed high segmentation performance. [ABSTRACT FROM AUTHOR]

Robot audition aims at developing robot's ears that work in the real world, that is, machine listening of multiple sound sources. Its critical problem is noise. Speech interfaces have become more familiar and more indispensable as smartphones and artificial intelligence (AI) speakers spread. Their critical problems are noise and multiple simultaneous speakers. Recently two technological advances have contributed to significantly improve the performance of speech interfaces and robot audition. Emerging deep learning technology has improved noise robustness of automatic speech recognition, whereas microphone array processing has improved the performance of preprocessing such as noise reduction. Herein, an overview and history of robot audition are provided together with introduction of an open‐source software for robot audition and its wide applications in the real world. Also, it is discussed how robot audition contributes to the development of computational auditory scene analysis, that is, understanding of real‐world auditory environments. [ABSTRACT FROM AUTHOR]

Ego-noise, i.e., the noise a robot causes by its own motions, significantly corrupts the microphone signal and severely impairs the robot's capability to interact seamlessly with its environment. Therefore, suitable ego-noise suppression techniques are required. For this, it is intuitive to use also motor data collected by proprioceptors mounted to the joints of the robot since it describes the physical state of the robot and provides additional information about the ego-noise sources. In this paper, we use a dictionary-based approach for ego-noise suppression in a semi-supervised manner: first, an ego-noise dictionary is learned and subsequently used to estimate the ego-noise components of a mixture by computing a weighted sum of dictionary entries. The estimation of the weights is very sensitive against other signals beside ego-noise contained in the mixture. For increased robustness, we therefore propose to incorporate knowledge about the physical state of the robot to the estimation of the weights. This is achieved by introducing a motor data-based regularization term to the estimation problem which promotes similar weights for similar physical states. The regularization is derived by representing the motor data as a graph and imprints the intrinsic structure of the motor data space onto the dictionary model. We analyze the proposed method and evaluate its ego-noise suppression performance for a large variety of different movements and demonstrate the superiority of the proposed method compared to an approach without using motor data. [ABSTRACT FROM AUTHOR]

Drone audition, drone's auditory capabilities for a multi-rotor helicopter (hereinafter, drone), has been developed to improve real-world tasks, e.g. search-and-rescue tasks, by compensating for the weakness of visual capabilities due to darkness and occlusion. Most of current implementations of robot audition focus on a single sound source. This paper focuses on the estimation of multiple sound source positions from acoustic signals captured by a drone equipped with a microphone array. Due to ego-noise such as rotor and airflow noise around the drone, the estimation of the sound source position is obscured and prone to error. In particular, in case of multiple sound sources, data association between localization information and sound sources is critical to the performance of such estimation. To cope with uncertainty in data association, we extend Global Nearest Neighbor (GNN) to exploit sound source features (GNN-c) because drone audition needs real-time or least latency. The resulting system demonstrates that it can estimate multiple sound source positions with an accuracy of about 3 m. [ABSTRACT FROM AUTHOR]

Speech interfaces for household robots utilizing third‐party automatic speech recognition (ASR) services face the challenge of overcoming stationary ego‐noise that decreases ASR accuracy. Previous studies on signal processing have proposed numerous noise reduction methods that increase the signal‐to‐noise ratio of speech audio and subjective speech clarity. However, severe limitations on the cost of hardware of household robots and the use of closed 'black box' ASR services require us to re‐examine the efficacy of noise reduction methods in this context. Here we compare the effect of several basic noise filters on the performance of ASR services when speech sounds include the stationary ego‐noise of a humanoid Pepper robot. The result revealed that a spectrum subtraction filter improves the accuracy of ASR services best. We also demonstrate that the filter improves ASR performance on an actual Pepper robot system. This study not only provides practical knowledge on the selection of noise filters for a robot system but also discusses further improvements to the listening capabilities of the robot utilizing ASR. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc. [ABSTRACT FROM AUTHOR]

Robot audition refers to a range of hearing capabilities which help robots explore and understand their environment. Among them, sound source localization is the problem of estimating the location of a sound source given measurements of its angle of arrival with respect to a microphone array mounted on the robot. In addition, robot motion can help quickly solve the front-back ambiguity existing in a linear microphone array. In this article, we focus on the problem of exploiting robot motion to improve the estimation of the location of an intermittent and possibly moving source in a noisy and reverberant environment. We first propose a robust extended mixture Kalman filtering framework for jointly estimating the source location and its activity over time. Building on this framework, we then propose a long-term robot motion planning algorithm based on Monte Carlo tree search to find an optimal robot trajectory according to two alternative criteria: the Shannon entropy or the standard deviation of the estimated belief on the source location. These criteria are integrated over time using a discount factor. Experimental results show the robustness of the proposed estimation framework to false angle of arrival measurements within ± 20 ∘ and 10% false source activity detection rate. The proposed robot motion planning technique achieves an average localization error 48.7% smaller than a one-step-ahead method. In addition, we compare the correlation between the estimation error and the two criteria, and investigate the effect of the discount factor on the performance of the proposed motion planning algorithm. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of 2D sound source localization using multiple microphone arrays in an outdoor environment. Two main issues exist in such localization. Since the localization performance depends on a variety of parameters, the lack of knowledge about how to design the system is one of those issues. A thorough analysis in respect to the accuracy of the localization results with different simulation conditions has been performed. Obtained characteristics lead to a discussion on limitations and applicability of the system. The distinction between multiple simultaneous sound sources is another problem. This is directly related to the appearance of outliers in the localization process. To solve this issue, an outlier removal method is proposed, which takes the properties of the observed sounds into consideration. In this paper a VR-based visualization method of the obtained results is also introduced. As the application scenario, we selected bird song analysis, which provides a challenging environment in terms of constantly changing signal-to-noise ratio and relative sensor-to-target position. A prototype system has been established using the proposed method. Several simulation results have been presented followed by a discussion on the issues. This leads to establishing system design guidelines that ensure a predictable performance. [ABSTRACT FROM AUTHOR]

Abstract Human–robot interaction in natural settings requires filtering out the different sources of sounds from the environment. Such ability usually involves the use of microphone arrays to localize, track and separate sound sources online. Multi-microphone signal processing techniques can improve robustness to noise but the processing cost increases with the number of microphones used, limiting response time and widespread use on different types of mobile robots. Since sound source localization methods are the most expensive in terms of computing resources as they involve scanning a large 3D space, minimizing the amount of computations required would facilitate their implementation and use on robots. The robot's shape also brings constraints on the microphone array geometry and configurations. In addition, sound source localization methods usually return noisy features that need to be smoothed and filtered by tracking the sound sources. This paper presents a novel sound source localization method, called SRP-PHAT-HSDA, that scans space with coarse and fine resolution grids to reduce the number of memory lookups. A microphone directivity model is used to reduce the number of directions to scan and ignore non significant pairs of microphones. A configuration method is also introduced to automatically set parameters that are normally empirically tuned according to the shape of the microphone array. For sound source tracking, this paper presents a modified 3D Kalman (M3K) method capable of simultaneously tracking in 3D the directions of sound sources. Using a 16-microphone array and low cost hardware, results show that SRP-PHAT-HSDA and M3K perform at least as well as other sound source localization and tracking methods while using up to 4 and 30 times less computing resources respectively. Highlights • Coarse-fine scanning reduces computational load and preserves localization accuracy. • Microphone directivity model reduces the amount of computations and false detections. • The TDOA uncertainty model optimizes the maximum sliding window size. • The modified Kalman filter provides efficient tracking while reducing computations. [ABSTRACT FROM AUTHOR]

This paper proposes a control framework based on auditory perception. Generally, in robot audition, the motion control of a robot from the sense of hearing relies on sound source localization. We propose in this paper an alternative approach, aural servo, which is derived from the sensor-based control framework. In this approach, robot motions are directly connected to the aural perception: The variation of low-level auditory features dictates the motions applied to the robot through a feedback loop. It has the advantage of being robust to spurious measurements and modeling approximations for a low computational cost. This paper presents the theoretical concept of the aural servo framework. Besides a theoretical analysis, the aural servo framework is validated through several experiments on different robotic platforms and under real-world conditions. [ABSTRACT FROM AUTHOR]

In this work a method for an unsupervised lateral localization of simultaneous sound sources is presented. Following a binaural approach, the kurtosis-driven split-EM algorithm (KDS-EM) implemented is able to estimate the direction of arrival of relevant sound sources without knowing a priori their number. Information about the localization is integrated within a period of observation time to serve as an auditory memory in the context of social robotics. Experiments have been conducted using two types of observation times, one shorter with the purpose of analyzing its performance in a reactive level, and other longer that allows the analysis of its contribution as an input of the building process of the sorroundings auditory models that servesto drive a more deliberative behavior. The system has been tested in real and reverberant environments, achieving a good performance based on an over-modeling process that is able to isolate the location of the relevant sources from adverse acoustic effects, such as reverberations. [ABSTRACT FROM AUTHOR]

In static scenarios, binaural sound localization is fundamentally limited by front-back ambiguity and distance non-observability. Over the past few years, “active” schemes have been shown to overcome these shortcomings, by combining spatial binaural cues with the motor commands of the sensor. In this context, given a Gaussian prior on the relative position to a source, this paper determines an admissible motion of a binaural head which leads, on average, to the one-step-ahead most informative audio-motor localization. To this aim, a constrained optimization problem is set up, which consists in maximizing the entropy of the next predicted measurement probability density function over a cylindric admissible set. The method is appraised through geometrical arguments, and validated in simulations and on real-life robotic experiments. [ABSTRACT FROM AUTHOR]

Acoustic interactions are important for understanding intra- and interspecific communication in songbird communities from the viewpoint of soundscape ecology. It has been suggested that birds may divide up sound space to increase communication efficiency in such a manner that they tend to avoid overlap with other birds when they sing. We are interested in clarifying the dynamics underlying the process as an example of complex systems based on short-term behavioral plasticity. However, it is very problematic to manually collect spatiotemporal patterns of acoustic events in natural habitats using data derived from a standard single-channel recording of several species singing simultaneously. Our purpose here was to investigate fine-scale spatiotemporal acoustic interactions of the great reed warbler. We surveyed spatial and temporal patterns of several vocalizing color-banded great reed warblers ( Acrocephalus arundinaceus) using an open-source software for robot audition HARK (Honda Research Institute Japan Audition for Robots with Kyoto University) and three new 16-channel, stand-alone, and water-resistant microphone arrays, named DACHO spread out in the bird's habitat. We first show that our system estimated the location of two color-banded individuals' song posts with mean error distance of 5.5 ± 4.5 m from the location of observed song posts. We then evaluated the temporal localization accuracy of the songs by comparing the duration of localized songs around the song posts with those annotated by human observers, with an accuracy score of average 0.89 for one bird that stayed at one song post. We further found significant temporal overlap avoidance and an asymmetric relationship between songs of the two singing individuals, using transfer entropy. We believe that our system and analytical approach contribute to a better understanding of fine-scale acoustic interactions in time and space in bird communities. [ABSTRACT FROM AUTHOR]