Search

Learning from multiple modalities, such as audio and video, offers opportunities for leveraging complementary information, enhancing robustness, and improving contextual understanding and performance. However, combining such modalities presents challenges, especially when modalities differ in data structure, predictive contribution, and the complexity of their learning processes. It has been observed that one modality can potentially dominate the learning process, hindering the effective utilization of information from other modalities and leading to sub-optimal model performance. To address this issue the vast majority of previous works suggest to assess the unimodal contributions and dynamically adjust the training to equalize them. We improve upon previous work by introducing a multi-loss objective and further refining the balancing process, allowing it to dynamically adjust the learning pace of each modality in both directions, acceleration and deceleration, with the ability to phase out balancing effects upon convergence. We achieve superior results across three audio-video datasets: on CREMA-D, models with ResNet backbone encoders surpass the previous best by 1.9% to 12.4%, and Conformer backbone models deliver improvements ranging from 2.8% to 14.1% across different fusion methods. On AVE, improvements range from 2.7% to 7.7%, while on UCF101, gains reach up to 6.1%.

Sleep abnormalities can have severe health consequences. Automated sleep staging, i.e. labelling the sequence of sleep stages from the patient's physiological recordings, could simplify the diagnostic process. Previous work on automated sleep staging has achieved great results, mainly relying on the EEG signal. However, often multiple sources of information are available beyond EEG. This can be particularly beneficial when the EEG recordings are noisy or even missing completely. In this paper, we propose CoRe-Sleep, a Coordinated Representation multimodal fusion network that is particularly focused on improving the robustness of signal analysis on imperfect data. We demonstrate how appropriately handling multimodal information can be the key to achieving such robustness. CoRe-Sleep tolerates noisy or missing modalities segments, allowing training on incomplete data. Additionally, it shows state-of-the-art performance when testing on both multimodal and unimodal data using a single model on SHHS-1, the largest publicly available study that includes sleep stage labels. The results indicate that training the model on multimodal data does positively influence performance when tested on unimodal data. This work aims at bridging the gap between automated analysis tools and their clinical utility., Comment: 10 pages, 4 figures, 2 tables, journal

Objective: Wearable seizure detection devices could provide more reliable seizure documentation outside the hospital compared to seizure self-reporting by patients, which is the current standard. Previously, during the SeizeIT1 project, we studied seizure detection based on behind-the-ear electroencephalography (EEG). However, the obtained sensitivities were too low for practical use, because not all seizures are associated with typical ictal EEG patterns. Therefore, in this paper, we aim to develop a multimodal automated seizure detection algorithm integrating behind-the-ear EEG and electrocardiography (ECG) for detecting focal seizures. In this framework, we quantified the added value of ECG to behind-the-ear EEG. Methods: This study analyzed three multicenter databases consisting of 135 patients having focal epilepsy and a total of 896 seizures. A patient-specific multimodal automated seizure detection algorithm was developed using behind-the-ear/temporal EEG and single-lead ECG. The EEG and ECG data were processed separately using machine learning methods. A late integration approach was applied for fusing those predictions. Results: The multimodal algorithm outperformed the EEG-based algorithm in two of three databases, with an increase of 11% and 8% in sensitivity for the same false alarm rate. Significance: ECG can be of added value to an EEG-based seizure detection algorithm using only behind-the-ear/temporal lobe electrodes for patients with focal epilepsy., Circuits and Systems

Objective: Patients with absence epilepsy sensitivity <10% of their absences. The clinical gold standard to assess absence epilepsy is a 24-h electroencephalographic (EEG) recording, which is expensive, obtrusive, and time-consuming to review. We aimed to (1) investigate the performance of an unobtrusive, two-channel behind-the-ear EEG-based wearable, the Sensor Dot (SD), to detect typical absences in adults and children; and (2) develop a sensitive patient-specific absence seizure detection algorithm to reduce the review time of the recordings. Methods: We recruited 12 patients (median age = 21 years, range = 8–50; seven female) who were admitted to the epilepsy monitoring units of University Hospitals Leuven for a 24-h 25-channel video-EEG recording to assess their refractory typical absences. Four additional behind-the-ear electrodes were attached for concomitant recording with the SD. Typical absences were defined as 3-Hz spike-and-wave discharges on EEG, lasting 3 s or longer. Seizures on SD were blindly annotated on the full recording and on the algorithm-labeled file and consequently compared to 25-channel EEG annotations. Patients or caregivers were asked to keep a seizure diary. Performance of the SD and seizure diary were measured using the F1 score. Results: We concomitantly recorded 284 absences on video-EEG and SD. Our absence detection algorithm had a sensitivity of.983 and false positives per hour rate of.9138. Blind reading of full SD data resulted in sensitivity of.81, precision of.89, and F1 score of.73, whereas review of the algorithm-labeled files resulted in scores of.83,89, and.87, respectively. Patient self-reporting gave sensitivity of.08, precision of 1.00, and F1 score of.15. Significance: Using the wearable SD, epileptologists were able to reliably detect typical absence seizures. Our automated absence detection algorithm reduced the review time of a 24-h recording from 1-2 h to around 5–10 min., SCOPUS: ar.j, info:eu-repo/semantics/published

Objective: Wearable seizure detection devices could provide more reliable seizure documentation outside the hospital compared to seizure self-reporting by patients, which is the current standard. Previously, during the SeizeIT1 project, we studied seizure detection based on behind-the-ear electroencephalography (EEG). However, the obtained sensitivities were too low for practical use, because not all seizures are associated with typical ictal EEG patterns. Therefore, in this paper, we aim to develop a multimodal automated seizure detection algorithm integrating behind-the-ear EEG and electrocardiography (ECG) for detecting focal seizures. In this framework, we quantified the added value of ECG to behind-the-ear EEG. Methods: This study analyzed three multicenter databases consisting of 135 patients having focal epilepsy and a total of 896 seizures. A patient-specific multimodal automated seizure detection algorithm was developed using behind-the-ear/temporal EEG and single-lead ECG. The EEG and ECG data were processed separately using machine learning methods. A late integration approach was applied for fusing those predictions. Results: The multimodal algorithm outperformed the EEG-based algorithm in two of three databases, with an increase of 11% and 8% in sensitivity for the same false alarm rate. Significance: ECG can be of added value to an EEG-based seizure detection algorithm using only behind-the-ear/temporal lobe electrodes for patients with focal epilepsy., Signal Processing Systems

Data fusion refers to the joint analysis of multiple datasets which provide complementary views of the same task. In this preprint, the problem of jointly analyzing electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) data is considered. Jointly analyzing EEG and fMRI measurements is highly beneficial for studying brain function because these modalities have complementary spatiotemporal resolution: EEG offers good temporal resolution while fMRI is better in its spatial resolution. The fusion methods reported so far ignore the underlying multi-way nature of the data in at least one of the modalities and/or rely on very strong assumptions about the relation of the two datasets. In this preprint, these two points are addressed by adopting for the first time tensor models in the two modalities while also exploring double coupled tensor decompositions and by following soft and flexible coupling approaches to implement the multi-modal analysis. To cope with the Event Related Potential (ERP) variability in EEG, the PARAFAC2 model is adopted. The results obtained are compared against those of parallel Independent Component Analysis (ICA) and hard coupling alternatives in both simulated and real data. Our results confirm the superiority of tensorial methods over methods based on ICA. In scenarios that do not meet the assumptions underlying hard coupling, the advantage of soft and flexible coupled decompositions is clearly demonstrated.

NMR spectra of Mg2+ ions in ionic liquids were recorded using a highly sensitive variant of NMR spectroscopy known as β-NMR. The β-NMR spectra of MgCl2 in EMIM-Ac and EMIM-DCA compare favourably with conventional NMR, and exhibit linewidths of ∼3 ppm, allowing for discrimination of species with oxygen and nitrogen coordination.

Filter bank-based multicarrier (FBMC) systems are currently being considered as a prevalent candidate for replacing the long established cyclic prefix (CP)-based orthogonal frequency division multiplexing (CP-OFDM) in the physical layer of next generation communications systems. In particular, FBMC/OQAM has received increasing attention due to, among other features, its potential for maximum spectral efficiency. It suffers, however, from an intrinsic self-interference effect, which complicates signal processing tasks at the receiver, including synchronization, channel estimation and equalization. In a multiple-input multiple-output (MIMO) configuration, the multi-antenna interference has also to be taken into account. (Semi-)blind FBMC/OQAM receivers have been little studied so far and mainly for single-antenna systems. The problem of joint channel estimation and data detection in a MIMO-FBMC/OQAM system, given limited or no training information, is studied in this paper through a tensor-based approach in the light of the success of such techniques in OFDM applications. Simulation-based comparisons with CP-OFDM are included, for realistic transmission models.

The growing use of neuroimaging technologies generates a massive amount of biomedical data that exhibit high dimensionality. Tensor-based analysis of brain imaging data has been proved quite effective in exploiting their multiway nature. The advantages of tensorial methods over matrix-based approaches have also been demonstrated in the characterization of functional magnetic resonance imaging (fMRI) data, where the spatial (voxel) dimensions are commonly grouped (unfolded) as a single way/mode of the 3-rd order array, the other two ways corresponding to time and subjects. However, such methods are known to be ineffective in more demanding scenarios, such as the ones with strong noise and/or significant overlapping of activated regions. This paper aims at investigating the possible gains from a better exploitation of the spatial dimension, through a higher- (4 or 5) order tensor modeling of the fMRI signal. In this context, and in order to increase the degrees of freedom of the modeling process, a higher-order Block Term Decomposition (BTD) is applied, for the first time in fMRI analysis. Its effectiveness is demonstrated via extensive simulation results.