Search

Accurate inspection of subtle defects on apple surfaces is necessary in agricultural engineering. However, existing methods often rely on expensive equipment and encounter difficulty in detecting small defect areas effectively. To address this challenge, we introduce the Subtle Surface Defects in Apples (SSDA) dataset, a custom dataset specifically collected and annotated for two defect types: scratches and pest damage. For benchmarking, we propose a heatmap-based anchor-free (HAFREE) detector, a novel end-to-end object detection architecture designed to localize subtle defects on apple surfaces. Unlike prior methods that rely on anchor boxes, HAFREE employs a heatmap-based approach to represent defects as keypoints using two-dimensional Gaussian heatmaps. We introduce a multiscale feature fusion block, targeting small objects by incorporating local and global contextual information. To mitigate overfitting, we also implement a patch training strategy incorporating full images and cropped patches during training as a regularizer. The proposed method achieves a mAP50 of 50.05% on the SSDA dataset, outperforming one- and two-stage anchor-based detectors and previous anchor-free approaches. Code is available at: https://github.com/nbngochan/HAFREE.

Multimodal emotion recognition has emerged as a promising approach to capture the complex nature of human emotions by integrating information from various sources such as physiological signals, visual behavioral cues, and audio-visual content. However, current methods often struggle with effectively processing redundant or conflicting information across modalities and may overlook implicit inter-modal correlations. To address these challenges, this paper presents a novel multimodal emotion recognition framework which integrates audio-visual features with viewers’ EEG data to enhance emotion classification accuracy. The proposed approach employs modality-specific encoders to extract spatiotemporal features, which are then aligned through contrastive learning to capture inter-modal relationships. Additionally, cross-modal attention mechanisms are incorporated for effective feature fusion across modalities. The framework, comprising pre-training, fine-tuning, and testing phases, is evaluated on multiple datasets of emotional responses. The experimental results demonstrate that the proposed multimodal approach, which combines audio-visual features with EEG data, is highly effective in recognizing emotions, highlighting its potential for advancing emotion recognition systems.

In the face of increasing irregular temperature patterns and climate shifts, the need for accurate power consumption prediction is becoming increasingly important to ensure a steady supply of electricity. Existing deep learning models have sought to improve prediction accuracy but commonly require greater computational demands. In this research, on the other hand, we introduce DelayNet, a lightweight deep learning model that maintains model efficiency while accommodating extended time sequences. Our DelayNet is designed based on the observation that electronic series data exhibit recurring irregular patterns over time. Furthermore, we present two substantial datasets of electricity consumption records from South Korean buildings spanning nearly two years. Empirical findings demonstrate the model’s performance, achieving 21.23%, 43.60%, 17.05% and 21.71% improvement compared to recurrent neural networks, gated-recurrent units, temporal convolutional neural networks and ARIMA models, as well as greatly reducing model complexity and computational requirements. These findings indicate the potential for micro-level power consumption planning, as lightweight models can be implemented on edge devices.

AbstractElectroencephalography (EEG) recordings taken during the perception of music tempo contain information that estimates the tempo of a music piece. If information about this tempo stimulus in EEG recordings can be extracted and classified, it can be effectively used to construct a music‐based brain–computer interface. This study proposes a novel convolutional recurrent attention model (CRAM) to extract and classify features corresponding to tempo stimuli from EEG recordings of listeners who listened with concentration to the tempo of musics. The proposed CRAM is composed of six modules, namely, network inputs, two‐dimensional convolutional bidirectional gated recurrent unit‐based sample encoder, sample‐level intuitive attention, segment encoder, segment‐level intuitive attention, and softmax layer, to effectively model spatiotemporal features and improve the classification accuracy of tempo stimuli. To evaluate the proposed method's performance, we conducted experiments on two benchmark datasets. The proposed method achieves promising results, outperforming recent methods.

To understand human emotional states, local activities in various regions of the cerebral cortex and the interactions among different brain regions must be considered. This paper proposes a hierarchical emotional context feature learning model that improves multichannel electroencephalography (EEG)-based emotion recognition by learning spatiotemporal EEG features from a local brain region to a global brain region. The proposed method comprises a regional brain-level encoding module, a global brain-level encoding module, and a classifier. First, multichannel EEG signals grouped into nine regions based on the functional role of the brain are input into a regional brain-level encoding module to learn local spatiotemporal information. Subsequently, the global brain-level encoding module improved emotional classification performance by integrating local spatiotemporal information from various brain regions to learn the global context features of brain regions related to emotions. Next, we applied a two-layer bidirectional gated recurrent unit (BGRU) with self-attention to the regional brain-level module and a one-layer BGRU with self-attention to the global brain-level module. Experiments were conducted using three datasets to evaluate the EEG-based emotion recognition performance of the proposed method. The results proved that the proposed method achieves superior performance by reflecting the characteristics of multichannel EEG signals better than state-of-the-art methods.

In this paper, we addressed the challenges in sorting high-yield apple cultivars that traditionally relied on manual labor or system-based defect detection. Existing single-camera methods failed to uniformly capture the entire surface of apples, potentially leading to misclassification due to defects in unscanned areas. Various methods were proposed where apples were rotated using rollers on a conveyor. However, since the rotation was highly random, it was difficult to scan the apples uniformly for accurate classification. To overcome these limitations, we proposed a multi-camera-based apple sorting system with a rotation mechanism that ensured uniform and accurate surface imaging. The proposed system applied a rotation mechanism to individual apples while simultaneously utilizing three cameras to capture the entire surface of the apples. This method offered the advantage of quickly and uniformly acquiring the entire surface compared to single-camera and random rotation conveyor setups. The images captured by the system were analyzed using a CNN classifier deployed on embedded hardware. To maintain excellent CNN classifier performance while reducing its size and inference time, we employed knowledge distillation techniques. The CNN classifier demonstrated an inference speed of 0.069 s and an accuracy of 93.83% based on 300 apple samples. The integrated system, which included the proposed rotation mechanism and multi-camera setup, took a total of 2.84 s to sort one apple. Our proposed system provided an efficient and precise solution for detecting defects on the entire surface of apples, improving the sorting process with high reliability.

This paper proposes an approach to improve the performance of no‐reference video quality assessment for sports videos with dynamic motion scenes using an efficient spatiotemporal model. In the proposed method, we divide the video sequences into video blocks and apply a 3D shearlet transform that can efficiently extract primary spatiotemporal features to capture dynamic natural motion scene statistics from the incoming video blocks. The concatenation of a deep residual bidirectional gated recurrent neural network and logistic regression is used to learn the spatiotemporal correlation more robustly and predict the perceptual quality score. In addition, conditional video block‐wise constraints are incorporated into the objective function to improve quality estimation performance for the entire video. The experimental results show that the proposed method extracts spatiotemporal motion information more effectively and predicts the video quality with higher accuracy than the conventional no‐reference video quality assessment methods.

The brain is more sensitive to stress than other organs and can develop many diseases under excessive stress. In this study, we developed a method to improve the accuracy of emotional stress recognition using multi-channel electroencephalogram (EEG) signals. The method combines a three-dimensional (3D) convolutional neural network with an attention mechanism to build a 3D convolutional gated self-attention neural network. Initially, the EEG signal is decomposed into four frequency bands, and a 3D convolutional block is applied to each frequency band to obtain EEG spatiotemporal information. Subsequently, long-range dependencies and global information are learned by capturing prominent information from each frequency band via a gated self-attention mechanism block. Using frequency band mapping, complementary features are learned by connecting vectors from different frequency bands, which is reflected in the final attentional representation for stress recognition. Experiments conducted on three benchmark datasets for assessing the performance of emotional stress recognition indicate that the proposed method outperforms other conventional methods. The performance analysis of proposed methods confirms that EEG pattern analysis can be used for studying human brain activity and can accurately distinguish the state of stress.

Graph convolutional neural networks (GCNNs) have been successfully applied to a wide range of problems, including low-dimensional Euclidean structural domains representing images, videos, and speech and high-dimensional non-Euclidean domains, such as social networks and chemical molecular structures. However, in computer vision, the existing GCNNs are not provided with positional information to distinguish between graphs of new structures; therefore, the performance of the image classification domain represented by arbitrary graphs is significantly poor. In this work, we introduce how to initialize the positional information through a random walk algorithm and continuously learn the additional position-embedded information of various graph structures represented over the superpixel images we choose for efficiency. We call this method the graph convolutional network with learnable positional embedding applied on images (IMGCN-LPE). We apply IMGCN-LPE to three graph convolutional models (the Chebyshev graph convolutional network, graph convolutional network, and graph attention network) to validate performance on various benchmark image datasets. As a result, although not as impressive as convolutional neural networks, the proposed method outperforms various other conventional convolutional methods and demonstrates its effectiveness among the same tasks in the field of GCNNs.

Recently, deep recurrent neural networks have achieved great success in various machine learning tasks, and have also been applied for sound event detection. The detection of temporally overlapping sound events in realistic environments is much more challenging than in monophonic detection problems. In this paper, we present an approach to improve the accuracy of polyphonic sound event detection in multichannel audio based on gated recurrent neural networks in combination with auditory spectral features. In the proposed method, human hearing perception‐based spatial and spectral‐domain noise‐reduced harmonic features are extracted from multichannel audio and used as high‐resolution spectral inputs to train gated recurrent neural networks. This provides a fast and stable convergence rate compared to long short‐term memory recurrent neural networks. Our evaluation reveals that the proposed method outperforms the conventional approaches.

Recent developments in the field of separation of mixed signals into music/voice components have attracted the attention of many researchers. Recently, iterative kernel back-fitting, also known as kernel additive modeling, was proposed to achieve good results for music/voice separation. To obtain minimum mean square error (MMSE) estimates of short-time Fourier transforms of sources, generalized spatial Wiener filtering (GW) is typically used. In this paper, we propose an advanced music/voice separation method that utilizes a generalized weighted β-order MMSE estimation (WbE) based on iterative kernel back-fitting (KBF). In the proposed method, WbE is used for the step of mixed music signal separation, while KBF permits kernel spectrogram model fitting at each iteration. Experimental results show that the proposed method achieves better separation performance than GW and existing Bayesian estimators. [ABSTRACT FROM AUTHOR]

The GMM is a conventional approach which has been recently applied in many face recognition studies. However, the question about how to deal with illumination changes while ensuring high performance is still a challenge, especially with real-world databases. In this paper, we propose a Visual Observation Confidence (VOC) measure for robust face recognition for illumination changes. Our VOC value is a combined confidence value of three measurements: Flatness Measure (FM), Centrality Measure (CM), and Illumination Normality Measure (IM). While FM measures the discrimination ability of one face, IM represents the degree of illumination impact on that face. In addition, we introduce CM as a centrality measure to help FM to reduce some of the errors from unnecessary areas such as the hair, neck or background. The VOC then accompanies the feature vectors in the EM process to estimate the optimal models by modified-GMM training. In the experiments, we introduce a real-world database, called KoFace, besides applying some public databases such as the Yale and the ORL database. The KoFace database is composed of 106 face subjects under diverse illumination effects including shadows and highlights. The results show that our proposed approach gives a higher Face Recognition Rate (FRR) than the GMM baseline for indoor and outdoor datasets in the real-world KoFace database (94% and 85%, respectively) and in ORL, Yale databases (97% and 100% respectively). [ABSTRACT FROM AUTHOR]

The robustness of an audio fingerprinting system in an actual noisy environment is a major challenge for audio-based content identification. This paper proposes a high-performance audio fingerprint extraction method for use in portable consumer devices. In the proposed method, a salient audio peak-pair fingerprint, based on a modulated complex lapped transform, improves the accuracy of the audio fingerprinting system in actual noisy environments with low computational complexity. Experimental results confirm that the proposed method is quite robust in different noise conditions and achieves promising preliminary accuracy results. [ABSTRACT FROM AUTHOR]

Improving the quality of service in IP networks is a major challenge for real-time voice communications. In particular, packet arrival-delay variation, so-called "jitter," is one of the main factors that degrade the quality of voice in mobile devices with the voice-over Internet protocol (VoIP). To resolve this issue, a receiver-based enhanced timing recovery algorithm combined with active jitter estimation is proposed. The proposed algorithm copes with the effect of transmission jitter by expanding or compressing each packet according to the predicted network delay and variations. Additionally, the active network jitter estimation incorporates rapid detection of delay spikes and reacts to changes in network conditions. Extensive simulations have shown that the proposed algorithm delivers high voice quality by pursuing an optimal trade-off between average buffering delay and packet loss rate. [ABSTRACT FROM AUTHOR]

Time-shift is a crucial function of interactive Televisions as like DVR and internet TV broadcasting services. Automatic important event detection allows users utilize time- shift function conveniently. In this paper, we propose a method to extract important events in baseball videos. In the proposed method, we first detect play scenes and audio events separately from video and audio tracks. For robust play scene extraction, we proposed off-line learning model having local adaptation based on ongoing analyzed video. And we implemented the audio event detection with a SVM-based classifier. Final important events are determined by a combination of each audio-visual detection results in real time. We evaluated our method with a baseball database of Korean and Major League games. Experimental results show that the implemented system runs in real time and achieves a remarkable performance of 0.85 recall and 0.97 precision rates. [ABSTRACT FROM AUTHOR]

This paper presents a video retrieval system (VRS) for Interactive-Television as like internet protocol television (IPTV). A video bookmark initiated by users is performed based on snippets of the background soundtrack corresponding to the ongoing program. Our VRS has two special aspects compared with previous bookmark systems. First, we adopt the robust audio fingerprint feature of long-term logarithmic modified DCT modulation coefficients (LMDCT-MC) for audio indexing and retrieval. Second, we propose and apply a two-stage search (TSS) algorithm for fast searching. In the first stage of TSS, candidate video segments are roughly determined with audio index bit vectors (IBV) and then the optimal video clip is obtained by fingerprint bit vectors (FBV). We evaluate the proposed system with a database of 100 TV programs including news, panel discussions, music shows, advertisements, and dramas. The experimental results show that our VRS achieve fast search, robustness to noise and high precision of retrieval. A search accuracy of 99.67% was accomplished. [ABSTRACT FROM AUTHOR]

In this paper, an integrated music recommendation system is proposed, which contains the Junctions of automatic music genre classification, automatic music emotion classification, and music similarity query. A novel tempo feature, named as log-scale modulation frequency coefficients, is presented in this paper. With AdaBoost algorithm, the proposed tempo feature is combined with timbre features and improves the performance of music genre and emotion classification. Comparing with the conventional methods based on timbre features, the precision of five-genre classification is enhanced from 86.8% to 92.2% and the accuracy of four-emotion classification is increased from 86.0% to 90.5%. Based on the results of music genre/emotion classification, we design a similarity query scheme, which can speed up the similarity query process without decreasing the precision. Furthermore, all the features employed in this paper are extracted from the data of MP3 partially decoding, which significantly reduces the feature extraction time. [ABSTRACT FROM AUTHOR]

The authors reflect on the application of scalable techniques for storage and organization of easily accessible audio in the form of streaming audio content and recordings on social media site. They also suggest to have smart techniques for examining and retrieving of these recordings based on their content. They discuss the mood of musical recordings with the use of a discretized emotional model.