Search

This paper proposes an approach to improve the performance of no‐reference video quality assessment for sports videos with dynamic motion scenes using an efficient spatiotemporal model. In the proposed method, we divide the video sequences into video blocks and apply a 3D shearlet transform that can efficiently extract primary spatiotemporal features to capture dynamic natural motion scene statistics from the incoming video blocks. The concatenation of a deep residual bidirectional gated recurrent neural network and logistic regression is used to learn the spatiotemporal correlation more robustly and predict the perceptual quality score. In addition, conditional video block‐wise constraints are incorporated into the objective function to improve quality estimation performance for the entire video. The experimental results show that the proposed method extracts spatiotemporal motion information more effectively and predicts the video quality with higher accuracy than the conventional no‐reference video quality assessment methods.

Context-aware computing, which gathers information about the environment at a given time and adapts its behavior accordingly, can be effectively applied to a monitoring system that automatically detects and copes with emergency situations. In this paper, we propose an indoor emergency awareness alarm system using a deep neural network on a mobile device for at-risk people, such as the elderly and children. The proposed system detects emergency situations and quickly delivers the state of the person being monitored to a guardian using a mobile transmission system. To do this, three types of contextual information, including sound, human activity, and indoor location, are instantly recognized by the protected person’s mobile device. Both sound and human activity are used to improve the recognition accuracy of emergency situations provided to guardians’ mobile devices. Sound events are detected by a residual neural network, and human activities are recognized by applying the accelerometer and gyroscope signals of the mobile device to a deep spiking neural network. In addition, the indoor location where the emergency occurred is detected by the deep spiking neural network. The experimental results show the high accuracy of the indoor emergency awareness framework.

In this paper, a new method for the calculation of the observation-confidence value that is applied in the modified adaptive Gaussian mixture model framework is proposed for speaker verification. First, an adaptive version of the multiple low-rank representation method, for which a weighted decomposition that incorporates the prior information regarding the speech/non-speech content is considered, is proposed to find the enhanced speech and for the estimation of the frame signal-to-noise ratio (SNR) values. Then, a simple sigmoid function is applied to convert the frame SNR values into the observation-confidence values. To verify the accuracy of the system, we use utterances from the Korean movie You Came From The Stars. The experiment results show that our proposed approach achieves a greater accuracy compared with the other well-known baseline methods, such as the GMM-based universal background model, the GMM supervector-based support vector machine (SVM), the i-vector-based SVM, and the sparse representation, under the noisy environment.

In this paper, we propose a high-performance audio fingerprinting system used in real-world query-by-example applications for acoustic audio-based content identification, especially for use in heterogeneous portable consumer devices or on-line audio distributed system. In the proposed method, audio fingerprints are generated using a modulated complex lapped transform-based non-repeating foreground audio extraction and an adaptive thresholding method for prominent peak detection. Effective matching is performed using a robust peak-pair-based hash function of non-repeating foreground audio to protect against noise, echo, artifacts from pitch-shifting, time-stretching, resampling, equalization, or compression. Experimental results confirm that the proposed method is quite robust in various distorted conditions and achieves preliminarily promising accuracy results.

Recently, deep recurrent neural networks have achieved great success in various machine learning tasks, and have also been applied for sound event detection. The detection of temporally overlapping sound events in realistic environments is much more challenging than in monophonic detection problems. In this paper, we present an approach to improve the accuracy of polyphonic sound event detection in multichannel audio based on gated recurrent neural networks in combination with auditory spectral features. In the proposed method, human hearing perception‐based spatial and spectral‐domain noise‐reduced harmonic features are extracted from multichannel audio and used as high‐resolution spectral inputs to train gated recurrent neural networks. This provides a fast and stable convergence rate compared to long short‐term memory recurrent neural networks. Our evaluation reveals that the proposed method outperforms the conventional approaches.

The robustness of an audio fingerprinting system in an actual noisy environment is a major challenge for audio-based content identification. This paper proposes a high-performance audio fingerprint extraction method for use in portable consumer devices. In the proposed method, a salient audio peak-pair fingerprint, based on a modulated complex lapped transform, improves the accuracy of the audio fingerprinting system in actual noisy environments with low computational complexity. Experimental results confirm that the proposed method is quite robust in different noise conditions and achieves promising preliminary accuracy results.

Nowadays, user localization in indoor environments is more necessary to build many location-based services. This paper presents a robust audio identification method for enhancing a real-time indoor localization system on a mobile device using the audio signals emitted by nearby loudspeakers. The proposed audio identification method deals with various noise distortions due to different noisy indoor locations by using foreground/background audio separation, prominent spectral pitch-based binary audio fingerprinting, and spectral peak-triplet-based audio fingerprinting. Experimental results confirm that the proposed audio identification method is quite robust in different noise conditions and achieves preliminary promising results for discriminating the location and orientation of a user in large indoor locations.

In this paper, we propose a new method to calculate observation confidence values that are applied in a modified adaptive GMM training for speaker verification. First, we use low rank matrix recovery (LRR) to find enhanced speeches and estimate frame SNR values. Then, a simple sigmoid function is applied to convert the frame SNR values into the observation confidence values. We also combine the frame SNR values estimated by the MMSE log-STSA and LRR methods in order to enhance performance of speaker verification system. To verify the accuracy of the system, we use utterances from a Korean movie “You came from the stars”. The experimental results show that our proposed approach achieves better accuracy than the baseline GMM-UBM under both clean and noisy environments.

In this paper, a novel visual signal reliability (VSR) measure is proposed to consider video degradation at the signal level in audio-visual speaker identification (AVSI). The VSR estimation is formulated using a Gaussian fuzzy membership function (GFMF) to measure lighting variations. The variance parameters of GFMF are optimized in order to maximize the performance of the overall AVSI. The experimental results show that the proposed method outperforms the score-based reliability measuring technique.

In this paper, we propose an integrated music video browsing system for personalized digital television. The system has the functions of automatic music emotion classification, automatic theme-based music classification, salient region detection, and shot classification. From audio (music) tracks, highlight detection and emotion classification are performed on the basis of information on temporal energy, timbre and tempo. For video tracks, shot detection is fulfilled to classify shots into face shots and color-based shots. Lastly automatic grouping of themes is executed on music titles and their lyrics. With a database of international music videos, we evaluate the performance of each function implemented in this paper. The experimental results show that the music browsing system achieves remarkable performances. Thus, our system can be adopted in any digital television for providing personalized services.

Time-shift is a crucial function of interactive televisions as like DVR and Internet TV broadcasting services. Automatic important event detection allows users utilize time- shift function conveniently. In this paper, we propose a method to extract important events in baseball videos. In the proposed method, we first detect play scenes and audio events separately from video and audio tracks. For robust play scene extraction, we proposed off-line learning model having local adaptation based on ongoing analyzed video. And we implemented the audio event detection with a SVM-based classifier. Final important events are determined by a combination of each audio-visual detection results in real time. We evaluated our method with a baseball database of Korean and Major League games. Experimental results show that the implemented system runs in real time and achieves a remarkable performance of 0.85 recall and 0.97 precision rates.

Recent developments in the field of separation of mixed signals into music/voice components have attracted the attention of many researchers....

In addition to the traditional video surveillance, various audio processing techniques can also be added to the existing CCTV cameras. They can be used as additional features to help in analyzing the scene better and autonomously detecting violence or any unwanted activity in the scene. For this purpose, a deep learning based scream sound detection approach is proposed in this paper. MFCC features after interpolation are used as input of the system. The proposed system is experimented using a self-recorded scream database and with controlled and calculated parameters 100 % accuracy is achieved.

In this paper we propose an speaker verification approach by applying low-rank recovery approach under total variability space, which is trained by a modified Gaussian Mixture Modeling (MGMM) with the observation confidence. In this model, we construct UBM mean supervector by MGMM in order to train total variability matrix and obtain i-vectors. Besides, the low-rank recovery method is exploited to model i-vectors under the total variability space. Experiment results on utterances from Korean movie ("You came from the stars") show that our proposed approach can significantly enhance the performance of speaker verification and outperform the baseline GMM_UBM, GMM-supervector in noisy environments

In this paper, we present an MPEG-7-based audio classification and retrieval technique targeted for analysis of film material. The technique consists of low-level descriptors and high-level description schemes. For low-level descriptors, low-dimensional features such as audio spectrum projection based on audio spectrum basis descriptors is produced in order to find a balanced tradeoff between reducing dimensionality and retaining maximum information content. High-level description schemes are used to describe the modeling of reduced-dimension features, the procedure of audio classification, and retrieval. A classifier based on continuous hidden Markov models is applied. The sound model state path, which is selected according to the maximum-likelihood model, is stored in an MPEG-7 sound database and used as an index for query applications. Various experiments are presented where the speaker- and sound-recognition rates are compared for different feature extraction methods. Using independent component analysis, we achieved better results than normalized audio spectrum envelope and principal component analysis in a speaker recognition system. In audio classification experiments, audio sounds are classified into selected sound classes in real time with an accuracy of 96%.

In recent years, there has been a great interest in detecting and tracking multiple individuals due to its complexity as well as wide application. Among various approaches, poselet has been proven to be applicable in various scenarios for locating people. Motivated by its advantages together with the tracking efficiency of Particle Filter, we would like to propose a parallel detection and tracking method by integrating the two techniques of handling this challenging problem. The experiments are carried out in a school area including both indoor and outdoor environment. The results have shown that our method can detect and keep track of the targeted individuals even under some complicated circumstances such as occlusion or varying lightning condition.

While stroking a rigid tool over an object surface, vibrations induced on the tool, which represent the interaction between the tool and the surface texture, can be measured by means of an accelerometer. Such acceleration signals can be used to recognize or to classify object surface textures. The temporal and spectral properties of the acquired signals, however, heavily depend on different parameters like the applied force on the surface or the lateral velocity during the exploration. Robust features that are invariant against such scan-time parameters are currently lacking, but would enable texture classification and recognition using uncontrolled human exploratory movements. In this paper, we introduce a haptic texture database which allows for a systematic analysis of feature candidates. The publicly available database includes recorded accelerations measured during controlled and well-defined texture scans, as well as uncontrolled human free hand texture explorations for 43 different textures. As a preliminary feature analysis, we test and compare six well-established features from audio and speech recognition together with a Gaussian Mixture Model-based classifier on our recorded free hand signals. Among the tested features, best results are achieved using Mel-Frequency Cepstral Coefficients (MFCCs), leading to a texture recognition accuracy of 80.2%.

This paper presents a music emotion classification based on music highlight detection. To find a highlight segment of songs, we use only energy information based on normalized MDCT coefficients of audio streams. With AdaBoost algorithm, the proposed tempo feature is combined with timbre features and improves the performance of music emotion classification based on the detected music highlight segment. Experimental results confirm that the proposed method achieves preliminary promising results in terms of accuracy.

In this paper, we propose a golf video browsing system using three stage audio segmentation. To increase the accuracy of audio segment boundaries for golf video browsing on digital video recorder, the proposed three-stage audio segmentation algorithm consists of Gaussian Mixture Model (GMM)-based classification, GMM-Linear Discriminant Analysis (LDA)-based classification, and occupation ratio-based segmentation. The detected audio segment is synchronized with the video frame for the extraction of thumbnails for use in the intelligent browsing of golf videos.

In this paper, we present an automatic summarization of highlights in golf videos based on audio information alone without video information. The proposed highlight summarization system is carried out based on semantic audio segmentation and detection on action units from audio signals. Studio speech, field speech, music, and applause are segmented by means of sound classification. Swing is detected by the methods of impulse onset detection. Sounds like swing and applause form a complete action unit, while studio speech and music parts are used to anchor the program structure. With the advantage of highly precise detection of applause, highlights are extracted effectively. Our experimental results obtain high classification precision on 18 golf games. It proves that the proposed system is very effective and computationally efficient to apply the technology to embedded consumer electronic devices.

In this paper, we propose music and voice separation using kernel sptectrogram models backfitting based on log-spectral amplitude estimator. The existing method separates sources based on the estimate of a desired objects by training MSE (Mean Square Error) designed Winer filter. We introduce rather clear music and voice signals with application of log-spectral amplitude estimator, instead of adaptation of MSE which has been treated as an existing method. Experimental results reveal that the proposed method shows higher performance than the existing methods.

We present a hybrid speaker-based segmentation, which combines metric-based and model-based techniques. Without a priori information about the number of speakers and speaker identities, the speech stream is segmented in three stages: (1) the most likely speaker changes are detected; (2) to group segments of identical speakers, a two-level clustering algorithm is performed using a Bayesian information criterion (BIC) and HMM model scores - every cluster is assumed to contain only one speaker; (3) the speaker models are reestimated from each cluster by HMM. Finally a resegmentation step performs a more refined segmentation using these speaker models. To measure the performance, we compare the segmentation results of the proposed hybrid method versus metric-based segmentation. Results show that the hybrid approach using two-level clustering significantly outperforms direct metric-based segmentation.

This paper proposes a tempo feature extraction method based on the long-term modulation spectrum analysis. To transform the modulation spectrum to a condensed feature vector, the log-scale modulation frequency coefficients are introduced. This idea aims at averaging the modulation frequency energy via the constant-Q filter-banks. Further it is pointed out that the feature can be extracted directly from the perceptually compressed data of digital music archives. To verify the effectiveness of the feature and its utility to music applications, the feature vector is used in a music emotion classification system. The system consisting two layers of Adaboost classifiers. In the first layer the conventional timbre features are employed. Then by adding the tempo feature in the second layer, the classification precision is improved dramatically. By this way the discriminability of the classifier based on the given features can be exploited extremely. The system obtains high classification precision on a small corpus. It proves that the proposed feature is very effective and computationally efficient to characterize the tempo information of music.

In this paper, we present an automatic extraction of goal events in soccer videos by using audio track features alone without relying on expensive-to-compute video track features. The extracted goal events can be used for high-level indexing and selective browsing of soccer videos. The detection of soccer video highlights using audio contents comprises three steps: 1) extraction of audio features from a video sequence, 2) event candidate detection of highlight events based on the information provided by the feature extraction Methods and the Hidden Markov Model (HMM), 3) goal event selection to finally determine the video intervals to be included in the summary. For this purpose we compared the performance of the well known Mel-scale Frequency Cepstral Coefficients (MFCC) feature extraction method vs. MPEG-7 Audio Spectrum Projection feature (ASP) extraction method based on three different decomposition methods namely Principal Component Analysis( PCA), Independent Component Analysis (ICA) and Non-Negative Matrix Factorization (NMF). To evaluate our system we collected five soccer game videos from various sources. In total we have seven hours of soccer games consisting of eight gigabytes of data. One of five soccer games is used as the training data (e.g., announcers' excited speech, audience ambient speech noise, audience clapping, environmental sounds). Our goal event detection results are encouraging.

This paper presents a phone-based approach of spoken document retrieval (SDR), developed in the framework of the emerging MPEG-7 standard. We describe an indexing and retrieval system that uses phonetic information only. The retrieval method is based on the vector space IR model, using phone N-grams as indexing terms. We propose a technique to expand the representation of documents by means of phone confusion probabilities in order to improve the retrieval performance. This method is tested on a collection of short German spoken documents, using 10 city names as queries.

We evaluate the MPEG-7 audio spectrum projection (ASP) features for general sound recognition performance against the well established MFCC. The recognition tasks of interest are speaker recognition, sound classification, and segmentation of audio using sound/speaker identification. For sound classification we use three approaches: direct approach; hierarchical approach without hints; hierarchical approach with hints. For audio segmentation, the MPEG-7 ASP features and MFCCs are used to train hidden Markov models (HMM) for individual speakers and sounds. The trained sound/speaker models are then used to segment conversational speech involving a given subset of people in panel discussion television programs. Results show that the MFCC approach yields a sound/speaker recognition rate superior to MPEG-7 implementations.

In this paper, we present a classification and retrieval technique targeted for retrieval of home video abstract using dimension-reduced, decorrelated spectral features of audio content. The feature extraction based on MPEG-7 descriptors consists of three main stages: Normalized Audio Spectrum Envelope (NASE), basis decomposition algorithm and basis projection, obtained by multiplying the NASE with a set of extracted basis functions. A classifier based on continuous hidden Markov models is applied. For retrieval with accurate performance the system consists of a two-level hierarchy method using speech recognition and sound classification. For the measure of the performance we compare the classification results of MPEG-7 standardized features vs. Mel-scale Frequency Cepstrum Coefficients (MFCC). Results show that the MFCC features yield better performance compared to MPEG-7 features.