Search

Learning from multiple modalities, such as audio and video, offers opportunities for leveraging complementary information, enhancing robustness, and improving contextual understanding and performance. However, combining such modalities presents challenges, especially when modalities differ in data structure, predictive contribution, and the complexity of their learning processes. It has been observed that one modality can potentially dominate the learning process, hindering the effective utilization of information from other modalities and leading to sub-optimal model performance. To address this issue the vast majority of previous works suggest to assess the unimodal contributions and dynamically adjust the training to equalize them. We improve upon previous work by introducing a multi-loss objective and further refining the balancing process, allowing it to dynamically adjust the learning pace of each modality in both directions, acceleration and deceleration, with the ability to phase out balancing effects upon convergence. We achieve superior results across three audio-video datasets: on CREMA-D, models with ResNet backbone encoders surpass the previous best by 1.9% to 12.4%, and Conformer backbone models deliver improvements ranging from 2.8% to 14.1% across different fusion methods. On AVE, improvements range from 2.7% to 7.7%, while on UCF101, gains reach up to 6.1%.

Objective: Most current wearable tonic-clonic seizure (TCS) detection systems are based on extra-cerebral signals, such as electromyography (EMG) or accelerometry (ACC). Although many of these devices show good sensitivity in seizure detection, their false positive rates (FPR) are still relatively high. Wearable EEG may improve performance; however, studies investigating this remain scarce. This paper aims 1) to investigate the possibility of detecting TCSs with a behind-the-ear, two-channel wearable EEG, and 2) to evaluate the added value of wearable EEG to other non-EEG modalities in multimodal TCS detection. Method: We included 27 participants with a total of 44 TCSs from the European multicenter study SeizeIT2. The multimodal wearable detection system Sensor Dot (Byteflies) was used to measure two-channel, behind-the-ear EEG, EMG, electrocardiography (ECG), ACC and gyroscope (GYR). First, we evaluated automatic unimodal detection of TCSs, using performance metrics such as sensitivity, precision, FPR and F1-score. Secondly, we fused the different modalities and again assessed performance. Algorithm-labeled segments were then provided to a neurologist and a wearable data expert, who reviewed and annotated the true positive TCSs, and discarded false positives (FPs). Results: Wearable EEG outperformed the other modalities in unimodal TCS detection by achieving a sensitivity of 100.0% and a FPR of 10.3/24h (compared to 97.7% sensitivity and 30.9/24h FPR for EMG; 95.5% sensitivity and 13.9 FPR for ACC). The combination of wearable EEG and EMG achieved overall the most clinically useful performance in offline TCS detection with a sensitivity of 97.7%, a FPR of 0.4/24 h, a precision of 43.0%, and a F1-score of 59.7%. Subsequent visual review of the automated detections resulted in maximal sensitivity and zero FPs.

Sleep abnormalities can have severe health consequences. Automated sleep staging, i.e. labelling the sequence of sleep stages from the patient's physiological recordings, could simplify the diagnostic process. Previous work on automated sleep staging has achieved great results, mainly relying on the EEG signal. However, often multiple sources of information are available beyond EEG. This can be particularly beneficial when the EEG recordings are noisy or even missing completely. In this paper, we propose CoRe-Sleep, a Coordinated Representation multimodal fusion network that is particularly focused on improving the robustness of signal analysis on imperfect data. We demonstrate how appropriately handling multimodal information can be the key to achieving such robustness. CoRe-Sleep tolerates noisy or missing modalities segments, allowing training on incomplete data. Additionally, it shows state-of-the-art performance when testing on both multimodal and unimodal data using a single model on SHHS-1, the largest publicly available study that includes sleep stage labels. The results indicate that training the model on multimodal data does positively influence performance when tested on unimodal data. This work aims at bridging the gap between automated analysis tools and their clinical utility., Comment: 10 pages, 4 figures, 2 tables, journal

We report $\beta$-detected nuclear magnetic resonance of ultra-dilute $^{8}$Li$^{+}$ implanted in highly oriented pyrolytic graphite (HOPG). The absence of motional narrowing and diffusional spin-lattice relaxation implies Li$^+$ is not appreciably mobile up to 400 K, in sharp contrast to the highly lithiated stage compounds. However, the relaxation is remarkably fast and persists down to cryogenic temperatures. Ruling out extrinsic paramagnetic impurities and intrinsic ferromagnetism, we conclude the relaxation is due to paramagnetic centers correlated with implantation. While the resulting effects are not consistent with a Kondo impurity, they also differ from free paramagnetic centers, and we suggest that a resonant scattering approach may account for much of the observed phenomenology.

Using ion-implanted $^8$Li $\beta$-detected NMR, we study the evolution of the correlated metallic state of LaNiO$_3$ in a series of LaNiO$_3$/LaAlO$_3$ superlattices as a function of bilayer thickness. Spin-lattice relaxation measurements in an applied field of 6.55 T reveal two equal amplitude components: one with metallic ($T$-linear) $1/T_{1}$, and a second with a more complex $T$-dependence. The metallic character of the slow relaxing component is only weakly affected by the LaNiO$_3$ thickness, while the fast component is much more sensitive, exhibiting the opposite temperature dependence (increasing towards low $T$) in the thinnest, most magnetic samples. The origin of this bipartite relaxation is discussed.

Functional Magnetic Resonance Imaging (fMRI) is a non-invasive technique for studying brain activity. During an fMRI session, the subject executes a set of tasks (task-related fMRI study) or no tasks (resting-state fMRI), and a sequence of 3-D brain images is obtained for further analysis. In the course of fMRI, some sources of activation are caused by noise and artifacts. The removal of these sources is essential before the analysis of the brain activations. Deep Neural Network (DNN) architectures can be used for denoising and artifact removal. The main advantage of DNN models is the automatic learning of abstract and meaningful features, given the raw data. This work presents advanced DNN architectures for noise and artifact classification, using both spatial and temporal information in resting-state fMRI sessions. The highest performance is achieved by a voting schema using information from all the domains, with an average accuracy of over 98% and a very good balance between the metrics of sensitivity and specificity (98.5% and 97.5% respectively)., Comment: EUSIPCO 2021 (presented)

Data fusion refers to the joint analysis of multiple datasets which provide complementary views of the same task. In this preprint, the problem of jointly analyzing electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) data is considered. Jointly analyzing EEG and fMRI measurements is highly beneficial for studying brain function because these modalities have complementary spatiotemporal resolution: EEG offers good temporal resolution while fMRI is better in its spatial resolution. The fusion methods reported so far ignore the underlying multi-way nature of the data in at least one of the modalities and/or rely on very strong assumptions about the relation of the two datasets. In this preprint, these two points are addressed by adopting for the first time tensor models in the two modalities while also exploring double coupled tensor decompositions and by following soft and flexible coupling approaches to implement the multi-modal analysis. To cope with the Event Related Potential (ERP) variability in EEG, the PARAFAC2 model is adopted. The results obtained are compared against those of parallel Independent Component Analysis (ICA) and hard coupling alternatives in both simulated and real data. Our results confirm the superiority of tensorial methods over methods based on ICA. In scenarios that do not meet the assumptions underlying hard coupling, the advantage of soft and flexible coupled decompositions is clearly demonstrated.

We report $\beta$-detected nuclear magnetic resonance ($\beta$-NMR) measurements in Bi$_{2}$Se$_{3}$:Ca (BSC) and Bi$_{2}$Te$_{3}$:Mn (BTM) single crystals using $^{8}$Li$^{+}$ implanted to depths on the order of 100 nm. Above $\sim 200$ K, spin-lattice relaxation (SLR) reveals diffusion of $^{8}$Li$^{+}$, with activation energies of $\sim 0.4$ eV ($\sim 0.2$ eV) in BSC (BTM). At lower temperatures, the nuclear magnetic resonance (NMR) properties are those of a heavily doped semiconductor in the metallic limit, with Korringa relaxation and a small, negative, temperature-dependent Knight shift in BSC. From this, we make a detailed comparison with the isostructural tetradymite Bi$_{2}$Te$_{2}$Se (BTS) [McFadden et al., Phys Rev. B 99, 125201 (2019)]. In the magnetic BTM, the effects of the dilute Mn moments predominate, but remarkably the $^{8}$Li signal is not wiped out through the magnetic transition at 13 K, with a prominent critical peak in the SLR that is suppressed in a high applied field. This detailed characterization of the $^{8}$Li NMR response is an important step towards using depth-resolved $\beta$-NMR to study the low-energy properties of the chiral topological surface state (TSS). With the bulk NMR response now established in several Bi$_{2}$Ch$_{3}$ tetradymite topological insulators (TIs), the prospect of directly probing their chiral TSS using the depth resolution afforded by $\beta$-NMR remains strong., Comment: 14 pages, 10 figures

We have studied a mosaic of 1T-CrSe$_2$ single crystals using $\beta$-detected nuclear magnetic resonance of $^{8}$Li from 4 to 300 K. We identify two broad resonances that show no evidence of quadrupolar splitting, indicating two magnetically distinct environments for the implanted ion. We observe stretched exponential spin lattice relaxation and a corresponding rate ($1/T_1$) that increases monotonically above 200 K, consistent with the onset of ionic diffusion. A pronounced maximum in $1/T_1$ is observed at the low temperature magnetic transition near 20 K. Between these limits, $1/T_1$ instead exhibits a broad minimum with a remarkable absence of strong features in the vicinity of structural and magnetic transitions between 150 and 200 K. Together, the results suggest $^{8}$Li$^{+}$ site occupation within the van der Waals gap between CrSe$_2$ trilayers. Possible origins of the two environments are discussed.

We demonstrate the application of implanted-ion $\beta$-detected NMR as a probe of ionic liquid molecular dynamics through the measurement of $^8$Li spin-lattice relaxation (SLR) and resonance in 1-ethyl-3-methylimidazolium acetate. The motional narrowing of the resonance, and the local maxima in the SLR rate, $1/T_1$, imply a sensitivity to sub-nanosecond Li$^+$ solvation dynamics. From an analysis of $1/T_1$, we extract an activation energy ${E_A = 74.8 \pm 1.5}$ meV and Vogel-Fulcher-Tammann constant ${T_{\mathrm{VFT}} = 165.8 \pm 0.9}$ K, in agreement with the dynamic viscosity of the bulk solvent. Near the melting point, the lineshape is broad and intense, and the form of the relaxation is non-exponential, reflective of our sensitivity to heterogeneous dynamics near the glass transition. The depth resolution of this technique may later provide a unique means of studying nanoscale phenomena in ionic liquids.

We report $\beta$-detected NMR of ion-implanted $^{8}$Li in a single crystal and thin film of the strongly correlated metal LaNiO$_{3}$. In both samples, spin-lattice relaxation measurements reveal two distinct local metallic environments, as is evident from $T$-linear Korringa $1/T_{1}$ below 200 K with slopes comparable to other metals. A small, approximately temperature independent Knight shift of $\sim 74$ ppm is observed, yielding a normalized Korringa product characteristic of substantial antiferromagnetic correlations, but, we find no evidence for a magnetic transition from 4 to 310 K. Two distinct, equally abundant $^{8}$Li sites is inconsistent with the widely accepted rhombohedral structure of LaNiO$_{3}$, but cannot be simply explained by either of the common alternative orthorhombic or monoclinic distortions., Comment: 10 pages, 8 figures

We report measurements of the diffusion rate of isolated ion-implanted $^{8}$Li$^{+}$ within $\sim$120 nm of the surface of oriented single-crystal rutile TiO$_2$ using a radiotracer technique. The $\alpha$-particles from the $^{8}$Li decay provide a sensitive monitor of the distance from the surface and how the depth profile of $^{8}$Li evolves with time. The main findings are that the implanted Li$^{+}$ diffuses and traps at the (001) surface. The T-dependence of the diffusivity is described by a bi-Arrhenius expression with activation energies of 0.3341(21) eV above 200 K, whereas at lower temperatures it has a much smaller barrier of 0.0313(15) eV. We consider possible origins for the surface trapping, as well the nature of the low-T barrier., Comment: 5 pages, 4 figures

We report measurements on the high temperature ionic and low temperature electronic properties of the 3D topological insulator Bi$_2$Te$_2$Se using ion-implanted $^8$Li $\beta$-detected nuclear magnetic relaxation and resonance. With implantation energies in the range 5-28 keV, the probes penetrate beyond the expected range of the topological surface state, but are still within 250 nm of the surface. At temperatures above ~150 K, spin-lattice relaxation measurements reveal isolated $^8$Li$^{+}$ diffusion with an activation energy $E_{A} = 0.185(8)$ eV and attempt frequency $\tau_{0}^{-1} = 8(3) \times 10^{11}$ s$^{-1}$ for atomic site-to-site hopping. At lower temperature, we find a linear Korringa-like relaxation mechanism with a field dependent slope and intercept, which is accompanied by an anomalous field dependence to the resonance shift. We suggest that these may be related to a strong contribution from orbital currents or the magnetic freezeout of charge carriers in this heavily compensated semiconductor, but that conventional theories are unable to account for the extent of the field dependence. Conventional NMR of the stable host nuclei may help elucidate their origin., Comment: 17 pages, 12 figures, submitted to Phys. Rev. B

We report measurements of the dynamics of isolated $^{8}$Li$^{+}$ in single crystal rutile TiO$_{2}$ using $\beta$-detected NMR. From spin-lattice relaxation and motional narrowing, we find two sets of thermally activated dynamics: one below 100 K; and one at higher temperatures. At low temperature, the activation barrier is $26.8(6)$ meV with prefactor $1.23(5) \times 10^{10}$ s$^{-1}$. We suggest this is unrelated to Li$^{+}$ motion, and rather is a consequence of electron polarons in the vicinity of the implanted $^{8}$Li$^{+}$ that are known to become mobile in this temperature range. Above 100 K, Li$^{+}$ undergoes long-range diffusion as an isolated uncomplexed cation, characterized by an activation energy and prefactor of $0.32(2)$ eV and $1.0(5) \times 10^{16}$ s$^{-1}$, in agreement with macroscopic diffusion measurements. These results in the dilute limit from a microscopic probe indicate that Li$^{+}$ concentration does not limit the diffusivity even up to high concentrations, but that some key ingredient is missing in the calculations of the migration barrier. The anomalous prefactors provide further insight into both Li$^{+}$ and polaron motion., Comment: 14 pages, 9 figures

We report a comparison of the $1/T_1$ spin lattice relaxation rates (SLR) for $^9Li$ and $^8Li$ in Pt and SrTiO$_{3}$, in order to differentiate between magnetic and electric quadrupolar relaxation mechanisms. In Pt, the ratio of the $1/T_{1}$ spin relaxation rates $R_{Pt}$ was found to be 6.82(29), which is close to but less than the theoretical limit of $\sim7.68$ for pure magnetic relaxation. In SrTiO$_{3}$ this ratio was found to be 2.7(3), which is close but larger than the theoretical limit of $\sim2.14$ expected for pure electric quadrupolar relaxation. These results bring new insight into the nature of the fluctuations in the local environment of implanted $^8Li$ observed by $\beta$-NMR., Comment: 8 pages, 6 figures, submitted for publication at Physical Review B

Extracting information from functional magnetic resonance (fMRI) images has been a major area of research for more than two decades. The goal of this work is to present a new method for the analysis of fMRI data sets, that is capable to incorporate a priori available information, via an efficient optimization framework. Tests on synthetic data sets demonstrate significant performance gains over existing methods of this kind., Comment: 5 pages, 2 figures

Filter bank-based multicarrier (FBMC) systems are currently being considered as a prevalent candidate for replacing the long established cyclic prefix (CP)-based orthogonal frequency division multiplexing (CP-OFDM) in the physical layer of next generation communications systems. In particular, FBMC/OQAM has received increasing attention due to, among other features, its potential for maximum spectral efficiency. It suffers, however, from an intrinsic self-interference effect, which complicates signal processing tasks at the receiver, including synchronization, channel estimation and equalization. In a multiple-input multiple-output (MIMO) configuration, the multi-antenna interference has also to be taken into account. (Semi-)blind FBMC/OQAM receivers have been little studied so far and mainly for single-antenna systems. The problem of joint channel estimation and data detection in a MIMO-FBMC/OQAM system, given limited or no training information, is studied in this paper through a tensor-based approach in the light of the success of such techniques in OFDM applications. Simulation-based comparisons with CP-OFDM are included, for realistic transmission models.

The growing use of neuroimaging technologies generates a massive amount of biomedical data that exhibit high dimensionality. Tensor-based analysis of brain imaging data has been proved quite effective in exploiting their multiway nature. The advantages of tensorial methods over matrix-based approaches have also been demonstrated in the characterization of functional magnetic resonance imaging (fMRI) data, where the spatial (voxel) dimensions are commonly grouped (unfolded) as a single way/mode of the 3-rd order array, the other two ways corresponding to time and subjects. However, such methods are known to be ineffective in more demanding scenarios, such as the ones with strong noise and/or significant overlapping of activated regions. This paper aims at investigating the possible gains from a better exploitation of the spatial dimension, through a higher- (4 or 5) order tensor modeling of the fMRI signal. In this context, and in order to increase the degrees of freedom of the modeling process, a higher-order Block Term Decomposition (BTD) is applied, for the first time in fMRI analysis. Its effectiveness is demonstrated via extensive simulation results.