Search

This review examines how visual information enhances speech perception in individuals with hearing loss, focusing on the impact of age, linguistic stimuli, and specific hearing loss factors on the effectiveness of audiovisual (AV) integration. While existing studies offer varied and sometimes conflicting findings regarding the use of visual cues, our analysis shows that these key factors can distinctly shape AV speech perception outcomes. For instance, younger individuals and those who receive early intervention tend to benefit more from visual cues, particularly when linguistic complexity is lower. Additionally, languages with dense phoneme spaces demonstrate a higher dependency on visual information, underscoring the importance of tailoring rehabilitation strategies to specific linguistic contexts. By considering these influences, we highlight areas where understanding is still developing and suggest how personalized rehabilitation strategies and supportive systems could be tailored to better meet individual needs. Furthermore, this review brings attention to important aspects that warrant further investigation, aiming to refine theoretical models and contribute to more effective, customized approaches to hearing rehabilitation.

IntroductionThis study aimed to identify differences in voice characteristics and changes between patients with dysphagia-aspiration and healthy individuals using a deep learning model, with a focus on under-researched areas of pre- and post-swallowing voice changes in patients with dysphagia. We hypothesized that these variations may be due to weakened muscles and blocked airways in patients with dysphagia.MethodsA prospective cohort study was conducted on 198 participants aged >40 years at the Seoul National University Bundang Hospital from October 2021 to February 2023. Pre- and post-swallowing voice data of the participants were converted to a 64-kbps mp3 format, and all voice data were trimmed to a length of 2 s. The data were divided for 10-fold cross-validation and stored in HDF5 format with anonymized IDs and labels for the normal and aspiration groups. During preprocessing, the data were converted to Mel spectrograms, and the EfficientAT model was modified using the final layer of MobileNetV3 to effectively detect voice changes and analyze pre- and post-swallowing voices. This enabled the model to probabilistically categorize new patient voices as normal or aspirated.ResultsIn a study of the machine-learning model for aspiration detection, area under the receiver operating characteristic curve (AUC) values were analyzed across sexes under different configurations. The average AUC values for males ranged from 0.8117 to 0.8319, with the best performance achieved at a learning rate of 3.00e-5 and a batch size of 16. The average AUC values for females improved from 0.6975 to 0.7331, with the best performance observed at a learning rate of 5.00e-5 and a batch size of 32. As there were fewer female participants, a combined model was developed to maintain the sex balance. In the combined model, the average AUC values ranged from 0.7746 to 0.7997, and optimal performance was achieved at a learning rate of 3.00e-5 and a batch size of 16.ConclusionThis study evaluated a voice analysis-based program to detect pre- and post-swallowing changes in patients with dysphagia, potentially aiding in real-time monitoring. Such a system can provide healthcare professionals with daily insights into the conditions of patients, allowing for personalized interventions.Clinical Trial Registration:ClinicalTrials.gov, identifier NCT05149976

This paper presents an in-depth examination of the nonlinear deformation induced by dielectric actuation in pre-stressed ideal dielectric elastomers. A nonlinear ordinary differential equation that governs this deformation is formulated based on the hyperelastic model under dielectric stress. By means of numerical integration and neural network approximations, the relationship between voltage and stretch is established. Neural networks are utilized to approximate solutions for voltage-to-stretch and stretch-to- voltage transformations obtained via an explicit Runge-Kutta method. The efficacy of these approximations is illustrated by their use in compensating for nonlinearity through the waveshaping of the input signal. The comparative analysis demonstrates that the approximated solutions are more accurate than baseline methods, resulting in reduced harmonic distortions when dielectric elastomers are used as acoustic actuators. This study highlights the effectiveness of the proposed approach in mitigating nonlinearities and enhancing the performance of dielectric elastomers in acoustic actuation applications.

Generating dance movements from music has been considered a highly challenging task, as it requires the model to comprehend concepts from two different modalities: audio and video. However, recently, research on dance generation based on deep learning has been actively conducted. Existing dance generation researches tend to focus on generating dances in limited genres or for single dancer, so when K-pop music that mixes multiple genres was applied to existing methods, they failed to generate dances of various genres or group dances. In this paper, we propose the K-pop dance generation model in an autoregressive manner, a system designed to generate two-person synchronous dances based on K-pop music. To achieve this, we created a dataset by collecting videos of multiple dancers simultaneously dancing to K-pop music and dancing in various genres. Generating synchronous dances has two meanings: one is to generate a dance that goes well with the input music and dance when both are given, and the other is to simultaneously generate multiple dances that match the given music. We call them secondary dance generation and group dance generation, respectively, and designed the proposed model, which can perform both two generation methods. In addition, we would like to propose additional learning methods to make a model that better generates synchronous dances. To assess the performance of the proposed model, both qualitative and quantitative evaluations are conducted, proving the effectiveness and suitability of the proposed model when generating synchronous dances for K-pop music.

In recent years, the dance field has been able to create diverse content by leveraging technical advancements such as deep learning models, generating content beyond the unique artistic creations that only humans can create. However, in terms of dance data, there are still a lack of video and label datasets or datasets that contain multiple tags for videos. To address this gap, this paper explores the feasibility of generating dance captions from tags using a pseudo-captioning approach, inspired by the significant improvements large language models (LLMs) have shown in other domains. Various tags are generated from features extracted from videos and audio, and LLMs are then instructed to produce dance captions based on these tags. Captions were generated using both the open dance dataset and Internet dance videos, followed by user evaluations of randomly sampled captions. Participants found the captions effective in describing dance movements, of expert quality, and consistent with video content. Additionally, positive feedback was received on the evaluation of the gap in image extraction and the inclusion of tag data. This paper introduces and validates a novel pseudo-captioning method for generating dance captions using predefined tags, contributing to the expansion of data available for dance research and offering a practical solution to the current lack of datasets in this field.

Recent deep learning approaches for melody harmonization have achieved remarkable performance by overcoming the uneven chord distributions of music data. However, most of these approaches have not attempted to capture an original melodic structure and generate structured chord sequences with appropriate rhythms. Hence, we use a Transformer-based architecture that directly maps lower-level melody notes into a semantic higher-level chord sequence. In particular, we encode the binary piano roll of a melody into a note-based representation. Furthermore, we address the flexible generation of various chords with Transformer expanded with a VAE framework. We propose three Transformer-based melody harmonization models: 1) the standard Transformer-based model for the neural translation of a melody to chords (STHarm); 2) the variational Transformer-based model for learning the global representation of complete music (VTHarm); and 3) the regularized variational Transformer-based model for the controllable generation of chords (rVTHarm). Experimental results demonstrate that the proposed models generate more structured, diverse chord sequences than LSTM-based models.

Objectives To develop a simple algorithm for prescreening of obstructive sleep apnea (OSA) on the basis of respiratorysounds recorded during polysomnography during all sleep stages between sleep onset and offset. Methods Patients who underwent attended, in-laboratory, full-night polysomnography were included. For all patients, audiorecordings were performed with an air-conduction microphone during polysomnography. Analyses included allsleep stages (i.e., N1, N2, N3, rapid eye movement, and waking). After noise reduction preprocessing, data were segmentedinto 5-s windows and sound features were extracted. Prediction models were established and validated with10-fold cross-validation by using simple logistic regression. Binary classifications were separately conducted for threedifferent threshold criteria at apnea hypopnea index (AHI) of 5, 15, or 30. Prediction model characteristics, includingaccuracy, sensitivity, specificity, positive predictive value (precision), negative predictive value, and area under thecurve (AUC) of the receiver operating characteristic were computed. Results A total of 116 subjects were included; their mean age, body mass index, and AHI were 50.4 years, 25.5 kg/m2, and23.0/hr, respectively. A total of 508 sound features were extracted from respiratory sounds recorded throughoutsleep. Accuracies of binary classifiers at AHIs of 5, 15, and 30 were 82.7%, 84.4%, and 85.3%, respectively. Predictionperformances for the classifiers at AHIs of 5, 15, and 30 were AUC, 0.83, 0.901, and 0.91; sensitivity, 87.5%,81.6%, and 60%; and specificity, 67.8%, 87.5%, and 94.1%. Respective precision values of the classifiers were89.5%, 87.5%, and 78.2% for AHIs of 5, 15, and 30. Conclusion This study showed that our binary classifier predicted patients with AHI of ≥15 with sensitivity and specificityof >80% by using respiratory sounds during sleep. Since our prediction model included all sleep stage data, algorithmsbased on respiratory sounds may have a high value for prescreening OSA with mobile devices.

Quantitative evaluation of piano performance is of interests in many fields, including music education and computational performance rendering. Previous studies utilized features extracted from audio or musical instrument digital interface (MIDI) files but did not address the difference between hands (DBH), which might be an important aspect of high-quality performance. Therefore, we investigated DBH as an important factor determining performance proficiency. To this end, 34 experts and 34 amateurs were recruited to play two excerpts on a Yamaha Disklavier. Each performance was recorded in MIDI, and handcrafted features were extracted separately for the right hand (RH) and left hand (LH). These were conventional MIDI features representing temporal and dynamic attributes of each note and computed as absolute values (e. g., MIDI velocity) or ratios between performance and corresponding scores (e. g., ratio of duration or inter-onset interval (IOI)). These note-based features were rearranged into additional features representing DBH by simple subtraction between features of both hands. Statistical analyses showed that DBH was more significant in experts than in amateurs across features. Regarding temporal features, experts pressed keys longer and faster with the RH than did amateurs. Regarding dynamic features, RH exhibited both greater values and a smoother change along melodic intonations in experts that in amateurs. Further experiments using principal component analysis (PCA) and support vector machine (SVM) verified that hand-difference features can successfully differentiate experts from amateurs according to performance proficiency. Moreover, existing note-based raw feature values (Basic features) and DBH features were tested repeatedly via 10-fold cross-validation, suggesting that adding DBH features to Basic features improved F1 scores to 93.6% (by 3.5%) over Basic features. Our results suggest that differently controlling both hands simultaneously is an important skill for pianists; therefore, DBH features should be considered in the quantitative evaluation of piano performance.

Abstract Purpose Breathing sounds during sleep are altered and characterized by various acoustic specificities in patients with sleep disordered breathing (SDB). This study aimed to identify acoustic biomarkers indicative of the severity of SDB by analyzing the breathing sounds collected from a large number of subjects during entire overnight sleep. Methods The participants were patients who presented at a sleep center with snoring or cessation of breathing during sleep. They were subjected to full-night polysomnography (PSG) during which the breathing sound was recorded using a microphone. Then, audio features were extracted and a group of features differing significantly between different SDB severity groups was selected as a potential acoustic biomarker. To assess the validity of the acoustic biomarker, classification tasks were performed using several machine learning techniques. Based on the apnea–hypopnea index of the subjects, four-group classification and binary classification were performed. Results Using tenfold cross validation, we achieved an accuracy of 88.3% in the four-group classification and an accuracy of 92.5% in the binary classification. Experimental evaluation demonstrated that the models trained on the proposed acoustic biomarkers can be used to estimate the severity of SDB. Conclusions Acoustic biomarkers may be useful to accurately predict the severity of SDB based on the patient’s breathing sounds during sleep, without conducting attended full-night PSG. This study implies that any device with a microphone, such as a smartphone, could be potentially utilized outside specialized facilities as a screening tool for detecting SDB.

Most of the previous approaches to lyrics-to-audio alignment used a pre-developed automatic speech recognition (ASR) system that innately suffered from several difficulties to adapt the speech model to individual singers. A significant aspect missing in previous works is the self-learnability of repetitive vowel patterns in the singing voice, where the vowel part used is more consistent than the consonant part. Based on this, our system first learns a discriminative subspace of vowel sequences, based on weighted symmetric nonnegative matrix factorization, by taking the self-similarity of a standard acoustic feature as an input. Then, we make use of canonical time warping, derived from a recent computer vision technique, to find an optimal spatiotemporal transformation between the text and the acoustic sequences. Experiments with Korean and English data sets showed that deploying this method after a pre-developed, unsupervised, singing source separation achieved more promising results than the other state-of-the-art unsupervised approaches and an existing ASR-based system.

This study aimed to determine the factors influencing patients' choice of physician at the first visit through database analysis of a tertiary hospital in South Korea. We collected data on the first treatments performed by physicians who had treated patients for at least 3 consecutive years over 10 years (from 2003 to 2012) from the database of Seoul National University's affiliated tertiary hospital. Ultimately, we obtained data on 524,012 first treatments of 319,004 patients performed by 115 physicians. Variables including physicians' age and medical school and patients' age were evaluated as influencing factors for the number of first treatments performed by each physician in each year using a Poisson regression through generalized estimating equations with a log link. The number of first treatments decreased over the study period. Notably, the relative risk for first treatments was lower among older physicians than among younger physicians (relative risk 0.96; 95% confidence interval 0.95 to 0.98). Physicians graduating from Seoul National University (SNU) also had a higher risk for performing first treatments than did those not from SNU (relative risk 1.58; 95% confidence interval 1.18 to 2.10). Finally, relative risk was also higher among older patients than among younger patients (relative risk 1.03; 95% confidence interval 1.01 to 1.04). This study systematically demonstrated that physicians' age, whether the physician graduated from the highest-quality university, and patients' age all related to patients' choice of physician at the first visit in a tertiary university hospital. These findings might be due to Korean cultural factors.

Although vascular pulsatile tinnitus (VPT) has been classified as "objective", VPT is not easily recognizable or documentable in most cases. In response to this, we have developed transcanal sound recording (TSR) and spectro-temporal analysis (STA) for the objective diagnosis of VPT. By refining our initial method, we were able to apply TSR/STA to post-treatment outcome evaluation, as well as pre-treatment objective diagnosis.TSR was performed on seven VPT patients and five normal controls before and after surgical or interventional treatment. VPT was recorded using an inserted microphone with the subjects placed in both upright and supine positions with 1) a neutral head position, 2) head rotated to the tinnitus side, 3) head rotated to the non-tinnitus side, and 4) a neutral position with ipsi-lesional manual cervical compression. The recorded signals were analyzed in both time and time-frequency domains by performing a short-time Fourier transformation.The pre-treatment ear canal signals of all VPT patients demonstrated pulse-synchronous periodic structures and acoustic characteristics that were representative of their presumptive vascular pathologies, whereas those the controls exhibited smaller peaks and weak periodicities. Compared with the pre-treatment signals, the post-treatment signals exhibited significantly reduced peak- and root mean square amplitudes upon time domain analysis. Additionally, further sub-band analysis confirmed that the pulse-synchronous signal of all subjects was not identifiable after treatment and, in particular, that the signal decrement was statistically significant at low frequencies. Moreover, the post-treatment signals of the VPT subjects revealed no significant differences when compared to those of the control group.We reconfirmed that the TSR/STA method is an effective modality to objectify VPT. In addition, the potential role of the TSR/STA method in the objective evaluation of treatment outcomes in patients with VPT was proven. Further studies incorporating a larger sample size and more refined recording techniques are warranted.

We consider a problem of retrieving the extreme value among sensed data under deadline constraints in wireless sensor networks with potential applications to alarm systems. The sensed data is mapped to a score which we adopt as a unified measure of the relative urgency of the data. The objective is to retrieve the data with the maximum score. We propose fully distributed schemes for contention based medium access and data combining. The proposed medium access scheme uses a randomized back-off which is controlled based on the score of the data to be transmitted. Data combining techniques are proposed to further suppress unnecessary traffic and reduce contention. The key observation is that one should aggressively prioritize packets with high score, up to an extent that does not incur excessive contention in channel access. Designed to capture such aspect, the proposed scheme is shown to substantially decrease the latency of the retrieval.

Dance and music often co-occur as evidenced when viewing choreographed dances or singers moving while performing. This study investigated how the viewing of dance motions shapes sound perception. Previous research has shown that dance reflects the temporal structure of its accompanying music, communicating musical meter (i.e. a hierarchical organization of beats) via coordinated movement patterns that indicate where strong and weak beats occur. Experiments here investigated the effects of dance cues on meter perception, hypothesizing that dance could embody the musical meter, thereby shaping participant reaction times (RTs) to sound targets occurring at different metrical positions.In experiment 1, participants viewed a video with dance choreography indicating 4/4 meter (dance condition) or a series of color changes repeated in sequences of four to indicate 4/4 meter (picture condition). A sound track accompanied these videos and participants reacted to timbre targets at different metrical positions. Participants had the slowest RT's at the strongest beats in the dance condition only. In experiment 2, participants viewed the choreography of the horse-riding dance from Psy's "Gangnam Style" in order to examine how a familiar dance might affect meter perception. Moreover, participants in this experiment were divided into a group with experience dancing this choreography and a group without experience. Results again showed slower RTs to stronger metrical positions and the group with experience demonstrated a more refined perception of metrical hierarchy. Results likely stem from the temporally selective division of attention between auditory and visual domains. This study has implications for understanding: 1) the impact of splitting attention among different sensory modalities, and 2) the impact of embodiment, on perception of musical meter. Viewing dance may interfere with sound processing, particularly at critical metrical positions, but embodied familiarity with dance choreography may facilitate meter awareness. Results shed light on the processing of multimedia environments.

Network coding is becoming essential part of network systems since it enhances system performance in various ways. To take full advantage of network coding, however, it is vital to guarantee low latency in the decoding process and thus parallelization of random network coding has drawn broad attention from the network coding community. In this paper, we investigate the problem of parallelizing random network coding for embedded sensor systems with multicore processors. Recently, general purpose graphics processing unit (GPGPU) technology has paved the way for parallelizing random network coding; however, it is not an option on embedded sensor nodes without GPUs and thus it is indispensable to leverage multicore processors which are becoming more common in embedded sensor nodes. We propose a novel random network coding parallelization technique that can fully exploit multicore processors. In our experiments, our parallel method exhibits over 150% throughput enhancement compared to existing state-of-the-art implementations on an embedded system.