Search

Robot welding penetration control is an important research topic in the field of robot welding online control. This study proposes a recognition control system for multi-modal signals with multiple welding parameters for robot welding penetration control. In this study, a filter bank suitable for welding environments was proposed for welding acoustic signals, and separation and contour extraction algorithms were designed for the welding image signals. Taking the acoustic and image signals as inputs, a multi-modal hybrid model and multi-parameter controller were established that can effectively classify and identify the welding penetration state. The penetration state was used as a reference input to the controller for controlling the welding speed and current in real time. In addition, a digital twin system was developed in this study and deployed on the main and edge computers. The test results show that the system and model can accurately identify the weld penetration, regulate the welding speed and welding current, control the width of the backside molten pool, and improve the welding quality. [ABSTRACT FROM AUTHOR]

Recent days have seen significant improvements in multi-modal learning made by Vision-Language Pre-training (VLP) models. However, most of them employ the coarse-grained global alignment to overcome semantic gap for generating common representations, which makes them inadequate to capture intrinsic semantic correlations in image-text retrieval and consequently degrading the accuracy. Moreover, it is expensive to fine-tune a VLP model to perform image-text retrieval due to its large number of parameters. In this paper, we propose a simple yet effective image-text retrieval method, termed Cross-Modality Interaction Reasoning for enhancing Vision-Language Pre-training (CMIR-VLP). Specifically, a Cross Modality Interaction Reasoning (CMIR) module, which is designed to inject fine-grained image-text associations into semantic correlations learning, integrates the patch cues into the word reasoning with a multi-modal interaction encoder. Besides, we propose a cross-interaction process to associate each local text semantics with local visual information for fine-grained image-text alignment. Extensive experiments demonstrate our method gains 52 and 97.5 over state-of-the-art non-pre-training methods on two widely used datasets, and it also outperforms several mainstream fine-tuned VIP models. The related code repository in https://github.com/PSYGIM/CMIR-VLP. [ABSTRACT FROM AUTHOR]

With the simultaneous acquisition of the infrared and optical remote sensing images of the same target becoming increasingly easy, using multi-modal data for high-performance object detection has become a research focus. In remote sensing multi-modal data, infrared images lack color information, it is hard to detect difficult targets with low contrast, and optical images are easily affected by illuminance. One of the most effective ways to solve this problem is to integrate multi-modal images for high-performance object detection. The challenge of fusion object detection lies in how to fully integrate multi-modal image features with significant modal differences and avoid introducing interference information while taking advantage of complementary advantages. To solve these problems, a new multi-modal fusion object detection method is proposed. In this paper, the method is improved in terms of two aspects: firstly, a new dual-branch asymmetric attention backbone network (DAAB) is designed, which uses a semantic information supplement module (SISM) and a detail information supplement module (DISM) to supplement and enhance infrared and RGB image information, respectively. Secondly, we propose a feature fusion pyramid network (FFPN), which uses a Transformer-like strategy to carry out multi-modal feature fusion and suppress features that are not conducive to fusion during the fusion process. This method is a state-of-the-art process for both FLIR-aligned and DroneVehicle datasets. Experiments show that this method has strong competitiveness and generalization performance. [ABSTRACT FROM AUTHOR]

Depth map super-resolution (DSR) is a technique aimed at restoring high-resolution (HR) depth maps from low-resolution (LR) depth maps. In this process, color images are commonly used as guidance to enhance the restoration procedure. However, the intricate degradation of LR depth poses a challenge, and previous image-guided DSR approaches, which implicitly model the degradation in the spatial domain, often fall short of producing satisfactory results. To address this challenge, we propose a novel approach called the Degradation-Guided Multi-modal Fusion Network (DMFNet). DMFNet explicitly characterizes the degradation and incorporates multi-modal fusion in both spatial and frequency domains to improve the depth quality. Specifically, we first introduce the deep degradation regularization loss function, which enables the model to learn the explicit degradation from the LR depth maps. Simultaneously, DMFNet converts the color images and depth maps into spectrum representations to provide comprehensive multi-domain guidance. Consequently, we present the multi-modal fusion block to restore the depth maps by leveraging both the RGB-D spectrum representations and the depth degradation. Extensive experiments demonstrate that DMFNet achieves state-of-the-art (SoTA) performance on four benchmarks, namely the NYU-v2, Middlebury, Lu, and RGB-D-D datasets. [ABSTRACT FROM AUTHOR]

Abstract Most of the sentiment analysis studies focus on the sentimental classification of pictures in the video, ignoring the spatio-temporal information of the sequence of picture frames as well as text and audio information. The multiple kernel learning is a new hotspot in the field of nuclear machine learning, capable of handling multiple modalities. For multiple kernel learning, it is easy to ignore the basic features that are not discriminative, and cannot make full use of the base features of different modes. This paper puts forward a novel multi-modal fusion model for sentiment analysis, in which a multiple kernel learning algorithm based on convolution margin-dimension constraint is proposed for feature fusion. Moreover, the 3D convolutional neural network is used to extract the features of visual information, and the multiple kernel learning algorithm based on margin-dimension constraint is used to fuse visual, text and audio sentiment features. Experiments conducted on the MOUD and IEMOCAP sentiment databases show that the proposed model outperforms existing models in the field of multi-modal sentiment analysis research.

Abstract Programmed death-ligand 1 (PD-L1) expressions play a crucial role in guiding therapeutic interventions such as the use of tyrosine kinase inhibitors (TKIs) and immune checkpoint inhibitors (ICIs) in lung cancer. Conventional determination of PD-L1 status includes careful surgical or biopsied tumor specimens. These specimens are gathered through invasive procedures, representing a risk of difficulties and potential challenges in getting reliable and representative tissue samples. Using a single center cohort of 189 patients, our objective was to evaluate various fusion methods that used non-invasive computed tomography (CT) and $$^{18}$$ 18 F-FDG positron emission tomography (PET) images as inputs to various deep learning models to automatically predict PD-L1 in non-small cell lung cancer (NSCLC). We compared three different architectures (ResNet, DenseNet, and EfficientNet) and considered different input data (CT only, PET only, PET/CT early fusion, PET/CT late fusion without as well as with partially and fully shared weights to determine the best model performance. Models were assessed utilizing areas under the receiver operating characteristic curves (AUCs) considering their 95% confidence intervals (CI). The fusion of PET and CT images as input yielded better performance for PD-L1 classification. The different data fusion schemes systematically outperformed their individual counterparts when used as input of the various deep models. Furthermore, early fusion consistently outperformed late fusion, probably as a result of its capacity to capture more complicated patterns by merging PET and CT derived content at a lower level. When we looked more closely at the effects of weight sharing in late fusion architectures, we discovered that while it might boost model stability, it did not always result in better results. This suggests that although weight sharing could be beneficial when modality parameters are similar, the anatomical and metabolic information provided by CT and PET scans are too dissimilar to consistently lead to improved PD-L1 status predictions.

Physiological signal recognition is crucial in emotion recognition, and recent advancements in multi-modal fusion have enabled the integration of various physiological signals for improved recognition tasks. However, current models for emotion recognition with hyper complexmulti-modal signals face limitations due to fusionmethods and insufficient attentionmechanisms, preventing further enhancement in classification performance. To address these challenges, we propose a new model framework named Signal Channel Attention Network (SCA-Net), which comprises three main components: an encoder, an attention fusion module, and a decoder. In the attention fusion module, we developed five types of attention mechanisms inspired by existing research and performed comparative experiments using the public dataset MAHNOB-HCI. All of these experiments demonstrate the effectiveness of the attention module we addressed for our baseline model in improving both accuracy and F1 score metrics. We also conducted ablation experiments within the most effective attention fusionmodule to verify the benefits of multi-modal fusion. Additionally, we adjusted the training process for different attention fusion modules by employing varying early stopping parameters to prevent model overfitting. [ABSTRACT FROM AUTHOR]

Programmed death-ligand 1 (PD-L1) expressions play a crucial role in guiding therapeutic interventions such as the use of tyrosine kinase inhibitors (TKIs) and immune checkpoint inhibitors (ICIs) in lung cancer. Conventional determination of PD-L1 status includes careful surgical or biopsied tumor specimens. These specimens are gathered through invasive procedures, representing a risk of difficulties and potential challenges in getting reliable and representative tissue samples. Using a single center cohort of 189 patients, our objective was to evaluate various fusion methods that used non-invasive computed tomography (CT) and 18 F-FDG positron emission tomography (PET) images as inputs to various deep learning models to automatically predict PD-L1 in non-small cell lung cancer (NSCLC). We compared three different architectures (ResNet, DenseNet, and EfficientNet) and considered different input data (CT only, PET only, PET/CT early fusion, PET/CT late fusion without as well as with partially and fully shared weights to determine the best model performance. Models were assessed utilizing areas under the receiver operating characteristic curves (AUCs) considering their 95% confidence intervals (CI). The fusion of PET and CT images as input yielded better performance for PD-L1 classification. The different data fusion schemes systematically outperformed their individual counterparts when used as input of the various deep models. Furthermore, early fusion consistently outperformed late fusion, probably as a result of its capacity to capture more complicated patterns by merging PET and CT derived content at a lower level. When we looked more closely at the effects of weight sharing in late fusion architectures, we discovered that while it might boost model stability, it did not always result in better results. This suggests that although weight sharing could be beneficial when modality parameters are similar, the anatomical and metabolic information provided by CT and PET scans are too dissimilar to consistently lead to improved PD-L1 status predictions. [ABSTRACT FROM AUTHOR]

In this study, we address the critical need for enhanced situational awareness and victim detection capabilities in Search and Rescue (SAR) operations amidst disasters. Traditional unmanned ground vehicles (UGVs) often struggle in such chaotic environments due to their limited manoeuvrability and the challenge of distinguishing victims from debris. Recognising these gaps, our research introduces a novel technological framework that integrates advanced gesture-recognition with cutting-edge deep learning for camerabased victim identification, specifically designed to empower UGVs in disaster scenarios. At the core of our methodology is the development and implementation of the Meerkat Optimization Algorithm--Stacked Convolutional Neural Network--Bi--Long Short Term Memory--Gated Recurrent Unit (MOASConv-Bi-LSTM-GRU) model, which sets a new benchmark for hand gesture detection with its remarkable performance metrics: accuracy, precision, recall, and F1-score all approximately 0.9866. This model enables intuitive, real-time control of UGVs through hand gestures, allowing for precise navigation in confined and obstacle-ridden spaces, which is vital for effective SAR operations. Furthermore, we leverage the capabilities of the latest YOLOv8 deep learning model, trained on specialised datasets to accurately detect human victims under a wide range of challenging conditions, such as varying occlusions, lighting, and perspectives. Our comprehensive testing in simulated emergency scenarios validates the effectiveness of our integrated approach. The system demonstrated exceptional proficiency in navigating through obstructions and rapidly locating victims, even in environments with visual impairments like smoke, clutter, and poor lighting. Our study not only highlights the critical gaps in current SAR response capabilities but also offers a pioneering solution through a synergistic blend of gesture-based control, deep learning, and purposebuilt robotics. The key findings underscore the potential of our integrated technological framework to significantly enhance UGV performance in disaster scenarios, thereby optimising life-saving outcomes when time is of the essence. This research paves the way for future advancements in SAR technology, with the promise of more efficient and reliable rescue operations in the face of disaster. [ABSTRACT FROM AUTHOR]

PURPOSES: To provide abdominal contrast-enhanced MR image synthesis, we developed an gradient regularized multi-modal multi-discrimination sparse attention fusion generative adversarial network (GRMM-GAN) to avoid repeated contrast injections to patients and facilitate adaptive monitoring. METHODS: With IRB approval, 165 abdominal MR studies from 61 liver cancer patients were retrospectively solicited from our institutional database. Each study included T2, T1 pre-contrast (T1pre), and T1 contrast-enhanced (T1ce) images. The GRMM-GAN synthesis pipeline consists of a sparse attention fusion network, an image gradient regularizer (GR), and a generative adversarial network with multi-discrimination. The studies were randomly divided into 115 for training, 20 for validation, and 30 for testing. The two pre-contrast MR modalities, T2 and T1pre images, were adopted as inputs in the training phase. The T1ce image at the portal venous phase was used as an output. The synthesized T1ce images were compared with the ground truth T1ce images. The evaluation metrics include peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and mean squared error (MSE). A Turing test and experts contours evaluated the image synthesis quality. RESULTS: The proposed GRMM-GAN model achieved a PSNR of 28.56, an SSIM of 0.869, and an MSE of 83.27. The proposed model showed statistically significant improvements in all metrics tested with p-values < 0.05 over the state-of-the-art model comparisons. The average Turing test score was 52.33%, which is close to random guessing, supporting the models effectiveness for clinical application. In the tumor-specific region analysis, the average tumor contrast-to-noise ratio (CNR) of the synthesized MR images was not statistically significant from the real MR images. The average DICE from real vs. synthetic images was 0.90 compared to the inter-operator DICE of 0.91. CONCLUSION: We demonstrated the function of a novel multi-modal MR image synthesis neural network GRMM-GAN for T1ce MR synthesis based on pre-contrast T1 and T2 MR images. GRMM-GAN shows promise for avoiding repeated contrast injections during radiation therapy treatment.

Parkinson's disease (PD) is a prevalent neurological disorder characterized by progressive dopaminergic neuron loss, leading to both motor and non-motor symptoms. Early and accurate diagnosis is challenging due to the subtle and variable nature of early symptoms. This study aims to address these diagnostic challenges by proposing a novel method, Localized Region Extraction and Multi-Modal Fusion (LRE-MMF), designed to enhance diagnostic accuracy through the integration of structural MRI (sMRI) and resting-state functional MRI (rs-fMRI) data. The LRE-MMF method utilizes the complementary strengths of sMRI and rs-fMRI: sMRI provides detailed anatomical information, while rs-fMRI captures functional connectivity patterns. We applied this approach to a dataset consisting of 20 PD patients and 20 healthy controls (HC), all scanned with a 3 T MRI. The primary objective was to determine whether the integration of sMRI and rs-fMRI through the LRE-MMF method improves the classification accuracy between PD and HC subjects. LRE-MMF involves the division of imaging data into localized regions, followed by feature extraction and dimensionality reduction using Principal Component Analysis (PCA). The resulting features were fused and processed through a neural network to learn high-level representations. The model achieved an accuracy of 75 %, with a precision of 0.8125, recall of 0.65, and an AUC of 0.8875. The validation accuracy curves indicated good generalization, with significant brain regions identified, including the caudate, putamen, thalamus, supplementary motor area, and precuneus, as per the AAL atlas. These results demonstrate the potential of the LRE-MMF method for improving early diagnosis and understanding of PD by effectively utilizing both sMRI and rs-fMRI data. This approach could contribute to the development of more accurate diagnostic tools.

Currently, most research on recognizing unsafe behaviors of underground personnel focuses on improving precision through computer vision. However, underground areas are prone to occlusion, unstable lighting, and reflection, making it difficult to accurately recognize unsafe behaviors using computer vision technology alone. Especially, similar actions such as climbing ladders and holding ladders during climbing operations are easily confused during the recognition process, posing safety hazards. In order to solve the above problems, a multi-modal detection method for holding ladders in underground climbing operations is proposed. This method analyzes surveillance video data from two modalities: visual and audio. In terms of visual modality, the YOLOv8 model is used to detect the presence of ladder. If there is a ladder, the position coordinates of the ladder are obtained, and the video segment is put into the OpenPose algorithm for pose estimation to obtain the features of various skeletal joint points of the human body. These skeletal joint point sequences are then placed into improved spatial attention temporal graph convolutional networks(SAT-GCN) to obtain human action labels and their corresponding probabilities. In terms of audio modality, the PaddlePaddle automatic language recognition system is used to convert speech into text, and the bidirectional encoder representations from transformers (BERT) model is used to analyze and extract the features of text information, so as to obtain the text label and its corresponding probability. Finally, the information obtained from the visual and audio modalities is fused at the decision-making level to determine whether there is a dpersonnel holding ladders for underground climbing operations. The experimental results show that in action recognition based on skeleton data, the optimized SAT-GCN model improves the recognition precision of three types of actions: holding, climbing, and standing by 3.36%, 2.83%, and 10.71%, respectively. The multi-modal detection method has a higher recognition accuracy than the single modal method, reaching 98.29%.

Abstract Arrhythmias are irregular heartbeat rhythms caused by various conditions. Automated ECG signal classification aids in diagnosing and predicting arrhythmias. Current studies mostly focus on 1D ECG signals, overlooking the fusion of multiple ECG modalities for enhanced analysis. We converted ECG signals into modal images using RP, GAF, and MTF, inputting them into our classification model. To optimize detail retention, we introduced a CNN-based model with FCA for multimodal ECG tasks. Achieving 99.6% accuracy on the MIT-BIH arrhythmia database for five arrhythmias, our method outperforms prior models. Experimental results confirm its reliability for ECG classification tasks.

Recently, medical image synthesis has attracted the attention of an increasing number of researchers. However, most of current approaches suffer from the loss of multi-modal complementary information and thus fail to preserve the property of each modality, resulting in image distortion and texture detail loss. To alleviate this issue, a multi-modal magnetic resonance (MR) image synthesis algorithm based on a mixed attention fusion module in hybrid generative adversarial network is proposed. Firstly, a novel mixed attention fusion (MAF) module aggregating an adaptive fusion strategy (AFS) and a soft attention module is proposed to fuse the high-level semantic information and the low-level fine-grained feature at different scales between different layers to exploit rich representative complementary information adaptively. Subsequently, Resnet-bottlenect attention mechanism (Res-BAM) is designed to perform adaptive optimization and exploit mutual information while preserving the original property of each modality. Thereafter, the attention weight is inferred by a 1D channel feature map and a 2D spatial feature map, and multiplied with the original feature map in order to get the adaptive feature map, which is integrated with the original feature map in a residual connection to preserve the original property of each modality and prevent network degradation. Finally, the structural similarity (SSIM) and L 1 -norm are point-wise combined by an optimal weighting impact factor to preserve the high frequency information, brightness, color and SSIM, which are viewed as the original property of each modality. The experimental results demonstrate the superiority of our model on the state of the art in quantitative measures, reasonable visual quality and clinic significance. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Mine Automation is the property of Industry & Mine Automation Editorial Department and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Traffic detection (including lane detection and traffic sign detection) is one of the key technologies to realize driving assistance system and auto drive system. However, most of the existing detection methods are designed based on single‐modal visible light data, when there are dramatic changes in lighting in the scene (such as insufficient lighting in night), it is difficult for these methods to obtain good detection results. In view of multi‐modal data can provide complementary discriminative information, based on the YoLoV5 model, this paper proposes a multi‐modal fusion YoLoV5 network, which consists of three key components: the dual stream feature extraction module, the correlation feature extraction module, and the self‐attention fusion module. Specifically, the dual stream feature extraction module is used to extract the features of each of the two modalities. Secondly, input the features learned from the dual stream feature extraction module into the correlation feature extraction module to learn the features with maximum correlation. Then, the extracted maximum correlation features are used to achieve information exchange between modalities through a self‐attention mechanism, and thus obtain fused features. Finally, the fused features are inputted into the detection layer to obtain the final detection result. Experimental results on different traffic detection tasks can demonstrate the superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Due to the well-known domain shift problem, directly deploying a trained multi-modal classifier to a new environment usually leads to poor performance. The existing multi-modal domain adaption methods not only lack the fine-grained information of cross-modal data distribution, but also lack the cross-modal correlation research. Therefore, this paper proposes a multi-modal domain adaption method based on parameter fusion and two-step alignment (PFTS) to solve the related problems. The consistency of network parameters is used to enhance the correlation among modalities, and a higher-order moment measurement is introduced to improve the alignment of data distribution at the fine-grained level. In addition, the weighting of each modality is further carried out to achieve focused transfer. Comprehensive experiments based on multi-modal datasets with different domain adaption settings have been conducted, the results show that the precision of PFTS is 5.38% higher than state-of-the-art multi-modal domain adaption methods. [ABSTRACT FROM AUTHOR]

Visual object tracking has been drawing increasing attention in recent years, as a fundamental task in computer vision. To extend the range of tracking applications, researchers have been introducing information from multiple modalities to handle specific scenes, with promising research prospects for emerging methods and benchmarks. To provide a thorough review of multi-modal tracking, different aspects of multi-modal tracking algorithms are summarized under a unified taxonomy, with specific focus on visible-depth (RGB-D) and visible-thermal (RGB-T) tracking. Subsequently, a detailed description of the related benchmarks and challenges is provided. Extensive experiments were conducted to analyze the effectiveness of trackers on five datasets: PTB, VOT19-RGBD, GTOT, RGBT234, and VOT19-RGBT. Finally, various future directions, including model design and dataset construction, are discussed from different perspectives for further research. [ABSTRACT FROM AUTHOR]

Physiological signals, such as EEG and eye movements, have emerged as promising research topics in emotion recognition due to their inherent advantages of objectivity, high recognition accuracy, and cost-effectiveness. However, most existing methods for fusing EEG and eye movement signals use concatenation or weighted summation, which may lead to information loss and limited ability to resist noise. To tackle this issue, in this paper, we propose a Coordinated-representation Decision Fusion Network (CoDF-Net) to efficiently fuse the representation of EEG and eye movement signals. Specifically, CoDF-Net first learns personalized information by maximizing the correlation between modalities. Next, the Decision-level Fusion Broad Learning System (DF-BLS) is developed to construct multiple sub-systems to obtain the final emotional states via the effective decision-making mechanism. To evaluate the performance of the proposed method, subject-dependent and subject-independent experiments are designed on two public datasets. Extensive experiments demonstrate that the proposed method has superior emotion recognition performance over traditional approaches and current state-of-the-art methods. The CoDF-Net achieves 94.09 and 91.62% in the subject-dependent setting and 87.04 and 83.87% in the subject-independent setting on the SEED-CHN and SEED-GER datasets, respectively. Moreover, it is found that the proposed method exhibits a more significant ability to resist noise by adding Gaussian noise with different standard deviations. [ABSTRACT FROM AUTHOR]

Semantic segmentation of high-resolution multispectral remote sensing image has been intensely studied. However, the shadow occlusions, or the similar color and textures, between the categories influence the segmentation accuracy. Concomitantly, the size of targets in the remote sensing images is diverse and the network cannot balance their segmentation. This paper introduces a network, Transformer-based Multi-modal Fusion Network (TMFNet), which fuses the multi-modal features and incorporates height features from the digital surface model (DSM) to supplement the extra different features between each category. Particularly, we introduce two parallel encoders to extract the features from different modalities, a Multi-Modal fusion model based on the Transformer (MMformer) to complete the multi-modal fusion, and a Border Region Attention based multi-level Fusion Module (BRAFM) to integrate the cross-level features and enhance the small target segmentation by utilizing the details around the border. The experiment results on the ISPRS Vaihingen and Potsdam benchmark datasets indicate that the proposed TMFNet outperforms the SOTA methods on the segmentation performance.

With the rise of social media and online interactions, there is a growing need for analytical models capable of understanding the nuanced, multi-modal communication inherent in platforms, especially for detecting sarcasm. Existing research employs multi-stage models along with extensive semantic information extractions and single-modal encoders. These models often struggle with efficient aligning and fusing multi-modal representations. Addressing these shortcomings, we introduce the Single-Stage Extensive Semantic Fusion (SSESF) model, designed to concurrently process multi-modal inputs in a unified framework, which performs encoding and fusing in the same architecture with shared parameters. A projection mechanism is employed to overcome the challenges posed by the diversity of inputs and the integration of a wide range of semantic information. Additionally, we design a multi-objective optimization that enhances the model’s ability to learn latent semantic nuances with supervised contrastive learning. The unified framework emphasizes the interaction and integration of multi-modal data, while multi-objective optimization preserves the complexity of semantic nuances for sarcasm detection. Experimental results on a public multi-modal sarcasm dataset demonstrate the superiority of our model, achieving state-of-the-art performance. The findings highlight the model’s capability to integrate extensive semantic information, demonstrating its effectiveness in the simultaneous interpretation and fusion of multi-modal data for sarcasm detection.

Walking as a unique biometric tool conveys important information for emotion recognition. Individuals in different emotional states exhibit distinct walking patterns. For this purpose, this paper proposes a novel approach to recognizing emotion during walking using electroencephalogram (EEG) and inertial signals. Accurate recognition of emotion is achieved by training in an end-to-end deep learning fashion and taking into account multi-modal fusion. Subjects wear virtual reality head-mounted display (VR-HMD) equipment to immerse in strong emotions during walking. VR environment shows excellent imitation and experience ability, which plays an important role in awakening and changing emotions. In addition, the multi-modal signals acquired from EEG and inertial sensors are separately represented as virtual emotion images by discrete wavelet transform (DWT). These serve as input to the attention-based convolutional neural network (CNN) fusion model. The designed network structure is simple and lightweight while integrating the channel attention mechanism to extract and enhance features. To effectively improve the performance of the recognition system, the proposed decision fusion algorithm combines Critic method and majority voting strategy to determine the weight values that affect the final decision results. An investigation is made on the effect of diverse mother wavelet types and wavelet decomposition levels on model performance which indicates that the 2.2-order reverse biorthogonal (rbio2.2) wavelet with two-level decomposition has the best recognition performance. Comparative experiment results show that the proposed method outperforms other existing state-of-the-art works with an accuracy of 98.73%.

Multi-modal sentiment analysis aims to recognize the sentiment expressed by the studied subjects utilizing textual, visual, and acoustic cues. Given the vast application prospects of multi-modal sentiment analysis in domains such as human-computer interaction and intelligent customer service systems, it has gained widespread attention in recent years. The design of fusion strategy significantly impacts analysis results. Different modalities varied on the extent of contributions to the sentiment analysis process, and the textual modality tends to play a crucial role. However, most existing fusion strategies have ignored the issue of uneven distribution of sentiment information across modalities. Therefore, in this study, we introduce a gated fusion network with progressive modality enhancement. To effectively address the challenge of imbalanced sentiment distribution, we propose a progressive modality enhancement approach that facilitates the interaction of sentiment cues. Furthermore, to overcome the limitations of insufficient inter-modal interactions observed in previous works, we introduce a gated multi-modal information interaction module. This module comprehensively exploits multi-modal data, ensuring that each modality can effectively utilize information from others. Experimental results demonstrate that the proposed method achieves highly competitive performance on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets. Furthermore, the ablation experiments confirm the dominant role of the textual modality, while indicating that the visual modality generally performs better than the acoustic modality across our experimental settings.

Knee osteoarthritis (OA) is a prevalent musculoskeletal condition affecting millions worldwide, posing significant health and economic burdens. Characterized by the degeneration of joint cartilage, the progression of knee OA varies significantly among individuals, making its prediction a complex issue. Previous studies on automated knee OA diagnosis have primarily relied on unimodal data, often overlooking the valuable information present in multi-modal data. Multi-modal learning, which integrates information from various modalities, is increasingly recognized for its potential to enhance diagnostic performance in medical applications. However, such models incur a higher computational load due to the additional data required. This research investigates the feasibility of multi-modal neural networks in knee OA diagnosis by integrating structural demographic data with unstructured imaging data. Three deep learning unimodal models (InceptionV3, DIKO, and EfficientNetv2) were transformed into multi-modal architectures (MF_InceptionNet, MF_DIKO, and MF_Eff) to compare their diagnostic capabilities. The proposed multi-modal models share a common architecture, with unimodal models acting as image feature extraction backbones and separate embedding layers for demographic data. The image features and demographic embeddings are combined into a unified vector before classification. Extensive experiments were conducted to evaluate the performance of these models across different class categories and dataset sizes. MF_DIKO and InceptionV3 emerged as the best multi-modal and unimodal neural networks, respectively, with overall accuracies of 0.67 and 0.75 for 3-class severity classification. Contrary to existing literature, our findings reveal that unimodal neural networks using only imaging features outperform multi-modal networks, suggesting unimodal models might suffice in certain applications.

Surface defects on the track may lead to accidents and pose a serious threat to the safety of passengers and staff. Regarding current detection methods, there are still shortcomings: manual detection is time-consuming and labor-intensive; vibration acceleration and machine vision methods rely solely on a single sensor data type, making them vulnerable to external interference, including coupled vibrations and oil contamination, leading to errors. The article proposes a new model for identifying the surface condition of steel rails using multimodal data of vibration and images: the multi-scale feature dynamic fusion network (MFDF-Net). Features are categorized into three scales: shallow, intermediate, and deep. An attention mechanism is incorporated to emphasize features of different scales, enabling dynamic weighting during the fusion of vibration and multi-scale features. Additionally, attention is given to fundamental features such as frequency and edge characteristics, as well as abstract features like the correlation between time series and image structure. MFDF-Net utilizes data collected by service vehicles to demonstrate that, in evaluating the model’s stability, the average accuracy of the test set across multiple training sessions reaches 99.333%. This represents a 2.67% improvement over the test accuracy of a single vibration mode and a 7.8% enhancement over that of a single image mode. These results indicate that the fusion network effectively discerns rail surface states, with multi-modal fusion offering higher precision and improved robustness compared to single-mode rail surface state analysis. This network holds promise for advancing intelligent diagnosis of rail surface defects.

Affective data is the basis of emotion recognition, which is mainly acquired through extrinsic elicitation. To investigate the enhancing effects of multi-sensory stimuli on emotion elicitation and emotion recognition, we designed an experimental paradigm involving visual, auditory, and olfactory senses. A multimodal emotional dataset (OVPD-II) that employed the video-only or video-odor patterns as the stimuli materials, and recorded the electroencephalogram (EEG) and electrooculogram (EOG) signals, was created. The feedback results reported by subjects after each trial demonstrated that the video-odor pattern outperformed the video-only pattern in evoking individuals’ emotions. To further validate the efficiency of the video-odor pattern, the transformer was employed to perform the emotion recognition task, where the highest accuracy reached 86.65% (66.12%) for EEG (EOG) modality with the video-odor pattern, which improved by 1.42% (3.43%) compared with the video-only pattern. What’s more, the hybrid fusion (HF) method combined with the transformer and joint training was developed to improve the performance of the emotion recognition task, which achieved classify accuracies of 89.50% and 88.47% for the video-odor and video-only patterns, respectively.

Color guided depth super-resolution (SR) aims to reconstruct a high-resolution (HR) depth image from a low-resolution (LR) one guided by its paired HR color image. However, when the sampling factor is large, color guided depth SR suffers from reconstructing accurate depth edges due to the severe loss of high frequency (HF) components. In this paper, we propose a latent edge guided depth SR network using attention-based hierarchical multi-modal fusion, named LEDSRNet. We extract the hierarchical multi-modal features from HR color and LR depth images, and perform selective fusion to estimate the residual map for depth SR. Firstly, we perform gradient map estimation to generate accurate depth edges from the input HR color image and the interpolated LR depth image, and filter out unnecessary edges in the HR color image while preventing texture copying artifacts in depth SR. Then, we perform depth upsampling to get depth edges from the input LR depth image and refine them guided by gradient features in the latent space. Moreover, we fuse the features extracted from gradient map estimation and depth upsampling to obtain the residual map for depth SR. Finally, we reconstruct SR depth image by adding the residual map to the interpolated LR depth image. We design an attention based multi-level residual block (AMRB) as the basic block for LEDSRNet to extract both shallow and deep features in color and depth images for hierarchical multi-modal fusion. In the loss function, we use a binarized gradient map from the ground truth depth image, i.e. mask map, to calculate the loss for edge and smooth areas separately, preventing excessive smoothing of edge regions in the reconstructed SR depth image. Extensive experiments show that LEDSRNet reconstructs accurate depth edges even in the large sampling factor and achieves the best performance in RMSE with low running time and small model parameters. They indicate that LEDSRNet outperforms state-of-the-art methods in terms of both visual quality and quantitative measurements.

Due to its potential use in evaluating mental health and enhancing patient care, emotion recognition using physiological signals, such as EEG, ECG, and EMG, has drawn a lot of attention in the healthcare industry. However, better neural network architecture and training parameter optimization is needed to achieve accurate and dependable emotion classification. This research aims to investigate the impact of various configurations of neural network architectures, proposed multimodal fusion network and training parameters on the classification performance of emotion recognition systems utilizing EEG, ECG, and EMG signals. Nine experiments were conducted, each representing a unique combination of hyperparameters, including EEG filters, kernel sizes, pooling sizes, ECG filters, EMG filters, LSTM units, dropout rates, epochs, and batch sizes. The experiments were designed to systematically explore the effects of these hyperparameters on classification accuracy, robustness, and computational efficiency. The experiments revealed that the choice of hyperparameters significantly influences the performance of emotion recognition systems. Proposed model achieved higher average accuracies, with Experiment 3 exhibiting the highest accuracy of 87.83%. However, the model also incurred higher computational costs in terms of training time and memory usage. Additionally, experiments with higher dropout rates demonstrated a slight decrease in accuracy compared to those with lower dropout rates, emphasizing the importance of balancing regularization and model complexity. The findings underscore the importance of fine-tuning hyperparameters and selecting appropriate neural network architectures to optimize the performance of emotion recognition systems. Advanced deep learning techniques and physiological signal processing, healthcare providers can develop robust and accurate emotion recognition systems to enhance patient care and support personalized interventions. Further research may focus on exploring novel architectures and refining training strategies to continue improving the efficacy and efficiency of AI-driven emotional intelligence solutions in healthcare delivery.

Jointing multi-modal image fusion and subsequent high-level tasks is attracting more researches to achieve both mutual promotions. However, owing the feature gap between the two tasks, complicated network structure and training strategies need to be redesigned for specific different datasets. To address these issues, this paper proposes an infrared and visible image fusion via general feature embedding from frozen CLIP and DINOv2 models. The core idea is that the general semantic features from CLIP model are injected into the fusion network with the DINOv2-based segmenter as a constraint. Specially, the feature merging module and injection strategies are design to generate the semantic features that are compatible with fusion features meanwhile aligned with DINOv2 features. Leveraging the generalization ability of these foundation models, the proposed network can be optimized mutually to promote the training process. Comprehensive experiments on the four public datasets demonstrate the effectiveness of our method.

Abstract Visual object tracking has been drawing increasing attention in recent years, as a fundamental task in computer vision. To extend the range of tracking applications, researchers have been introducing information from multiple modalities to handle specific scenes, with promising research prospects for emerging methods and benchmarks. To provide a thorough review of multi-modal tracking, different aspects of multi-modal tracking algorithms are summarized under a unified taxonomy, with specific focus on visible-depth (RGB-D) and visible-thermal (RGB-T) tracking. Subsequently, a detailed description of the related benchmarks and challenges is provided. Extensive experiments were conducted to analyze the effectiveness of trackers on five datasets: PTB, VOT19-RGBD, GTOT, RGBT234, and VOT19-RGBT. Finally, various future directions, including model design and dataset construction, are discussed from different perspectives for further research.

Many texture descriptors have been introduced in recent years to improve texture analysis and classification outcomes, which are important in many computer vision tasks including object recognition and detection, human detector, and especially in face recognition. Local pattern is a texture descriptor that can successfully extract distinctive texture features that possesses noise and illumination variance robustness. This paper focuses on making use of local pattern features in boosting object detection models in a multi-modal fusion paradigm to acquire reliable feature maps in forward propagation throughout the network regardless of variations in photo taking conditions. We propose an adaptive fusion architecture for RGB and Local Ternary Pattern information. This architecture leverage local pattern to enrich information of original feature maps and adapt to many object detection models. Our local pattern fusion network concentrates on backbone and neck modules with an simple and efficient operation. The notable accuracy advancement is 8.03% observed in Cascade R-CNN in KITTI Dataset. In difficult conditions, our fusion models significantly lift the original performance from 4.7% to 66.3% mAP score.

Facial action unit (AU) detection is a crucial step in the field of affective computing and plays a crucial role in applications such as human-computer interaction, psychology, and social robotics. Despite recent advances in the field, the problem of facial AU detection remains challenging, in particular in real-world scenarios with diverse lighting conditions and head poses. This paper first presents a new, realistically challenging multi-modal and multi-view AU dataset, captured in a real-world vehicle environment. Then we introduce a novel graph-based multi-modal multi-view fusion framework, tailored for challenging environments such as those encountered in Advanced Driver-Assistance Systems (ADAS), which significantly enhances AU detection performance under these difficult conditions. Our fusion model showcases significant advancements over current single-modality methods, achieving a marked improvement in F1 scores across most AUs. Specifically, the fusion approach demonstrated a 9.0% improvement in overall average F1 scores over the best-performing single-modality model. The results validate that integrating multiple modalities and viewpoints substantially boosts the model’s robustness and accuracy under diverse conditions, offering a meaningful advancement over the state-of-the-art.

With the simultaneous acquisition of the infrared and optical remote sensing images of the same target becoming increasingly easy, using multi-modal data for high-performance object detection has become a research focus. In remote sensing multi-modal data, infrared images lack color information, it is hard to detect difficult targets with low contrast, and optical images are easily affected by illuminance. One of the most effective ways to solve this problem is to integrate multi-modal images for high-performance object detection. The challenge of fusion object detection lies in how to fully integrate multi-modal image features with significant modal differences and avoid introducing interference information while taking advantage of complementary advantages. To solve these problems, a new multi-modal fusion object detection method is proposed. In this paper, the method is improved in terms of two aspects: firstly, a new dual-branch asymmetric attention backbone network (DAAB) is designed, which uses a semantic information supplement module (SISM) and a detail information supplement module (DISM) to supplement and enhance infrared and RGB image information, respectively. Secondly, we propose a feature fusion pyramid network (FFPN), which uses a Transformer-like strategy to carry out multi-modal feature fusion and suppress features that are not conducive to fusion during the fusion process. This method is a state-of-the-art process for both FLIR-aligned and DroneVehicle datasets. Experiments show that this method has strong competitiveness and generalization performance.

Efficient flotation beneficiation heavily relies on accurate flotation condition recognition based on monitored froth video. However, the recognition accuracy is hindered by limitations of extracting temporal features from froth videos and establishing correlations between complex multi-modal high-order data. To address the difficulties of inadequate temporal feature extraction, inaccurate online condition detection, and inefficient flotation process operation, this paper proposes a novel flotation condition recognition method named the multi-modal temporal hypergraph neural network (MTHGNN) to extract and fuse multi-modal temporal features. To extract abundant dynamic texture features from froth images, the MTHGNN employs an enhanced version of the local binary pattern algorithm from three orthogonal planes (LBP-TOP) and incorporates additional features from the three-dimensional space as supplements. Furthermore, a novel multi-view temporal feature aggregation network (MVResNet) is introduced to extract temporal aggregation features from the froth image sequence. By constructing a temporal multi-modal hypergraph neural network, we encode complex high-order temporal features, establish robust associations between data structures, and flexibly model the features of froth image sequence, thus enabling accurate flotation condition identification through the fusion of multi-modal temporal features. The experimental results validate the effectiveness of the proposed method for flotation condition recognition, providing a foundation for optimizing flotation operations. [ABSTRACT FROM AUTHOR]

Guided by information from RGB images, depth completion methods rebuild the dense depth from sparse depth input. However, the varying densities of valid pixels in sparse depth maps pose a significant challenge to the robustness of the completion model. To improve the robustness of depth completion, we propose a two-stage model called Semantic Aggregated Depth Completion (SADC) in this paper, comprising a coarse-grained completion stage and a fine-grained completion stage. In the coarse-grained completion stage, the Semantic Extraction Network (SEN) extracts RGB features and sends them to the Dynamic Semantic Aggregation (DSA) to predict the local semantic relationship (LSR) matrix. DSA aggregates the valid information based on the LSR matrix iteratively, resulting in coarse-grained completion results. In the fine-grained completion stage, SADC uses the Semantic Guidance Network (SGN) and Semantic Guidance Fusion (SGF) modules to refine the dense depth features from coarse-grained completion results by RGB features in multi-level and predict fine-grained completion results. We validate our method on NYU-v2 and KITTI with different valid pixel densities. The results demonstrate that SADC performs best results on benchmark tests and exhibits robustness to different densities without retraining. [ABSTRACT FROM AUTHOR]

This paper introduces 'SkeletonCLIP++', an extension of our prior work in human action recognition, emphasizing the use of semantic information beyond traditional label-based methods. The innovation, 'Weighted Frame Integration' (WFI), shifts video feature computation from averaging to a weighted frame approach, enabling a more nuanced representation of human movements in line with semantic relevance. Another key development, 'Contrastive Sample Identification' (CSI), introduces a novel discriminative task within the model. This task involves identifying the most similar negative sample among positive ones, enhancing the model's ability to distinguish between closely related actions. Incorporating the 'BERT Text Encoder Integration' (BTEI) leverages the pre-trained BERT model as our text encoder to refine the model's performance. Empirical evaluations on HMDB-51, UCF-101, and NTU RGB+D 60 datasets illustrate positive improvements, especially in smaller datasets. 'SkeletonCLIP++' thus offers a refined approach to human action recognition, ensuring semantic integrity and detailed differentiation in video data analysis. [ABSTRACT FROM AUTHOR]

Objective: To comprehensively capture intra-tumor heterogeneity in head and neck cancer (HNC) and maximize the use of valid information collected in the clinical field, we propose a novel multi-modal image–text fusion strategy aimed at improving prognosis. Method: We have developed a tailored diagnostic algorithm for HNC, leveraging a deep learning-based model that integrates both image and clinical text information. For the image fusion part, we used the cross-attention mechanism to fuse the image information between PET and CT, and for the fusion of text and image, we used the Q-former architecture to fuse the text and image information. We also improved the traditional prognostic model by introducing time as a variable in the construction of the model, and finally obtained the corresponding prognostic results. Result: We assessed the efficacy of our methodology through the compilation of a multicenter dataset, achieving commendable outcomes in multicenter validations. Notably, our results for metastasis-free survival (MFS), recurrence-free survival (RFS), overall survival (OS), and progression-free survival (PFS) were as follows: 0.796, 0.626, 0.641, and 0.691. Our results demonstrate a notable superiority over the utilization of CT and PET independently, and exceed the result derived without the clinical textual information. Conclusions: Our model not only validates the effectiveness of multi-modal fusion in aiding diagnosis, but also provides insights for optimizing survival analysis. The study underscores the potential of our approach in enhancing prognosis and contributing to the advancement of personalized medicine in HNC. [ABSTRACT FROM AUTHOR]

Currently, human action recognition has witnessed remarkable progress, and its achievements have been applied to daily life. However, most methods extract features from only a single view within each modality, which may not comprehensively capture the diversity and complexity of actions. Moreover, the ineffective removal of redundant information can result in an inconspicuous description of key information. These issues cloud affect the final action recognition accuracy. To address these issues, this paper proposes a novel method for single-subject routine action recognition, which combines multi-view key information representation and multi-modal fusion. Firstly, the energy of non-primary motion areas is reduced by motion mean normalization in the depth video sequence, thereby enhancing key information of action. Then, depth motion history map (DMHM) and depth spatio-temporal energy map (DSTEM) are extracted from planes and axes, respectively. The proposed DMHM effectively preserves the spatio-temporal information of actions, DSTEM preserves the motion contour and energy information. In terms of skeleton sequences, statistical features and motion contribution degree of each joint are extracted from the view of motion distribution and weights, respectively. Finally, depth and skeleton features are fused to achieve multi-modal fusion-based action recognition. The proposed method highlights the information of the main motion areas, and achieves recognition accuracies of 96.70 % on MSR-Action3D, 93.26 % on UTD-MHAD, and above 97.73 % on all tests of CZU-MHAD. The experimental results demonstrate that the proposed method effectively preserves action information and has better recognition accuracy than most existing methods. [ABSTRACT FROM AUTHOR]

Since camera and LiDAR sensors provide complementary information for the 3D semantic segmentation of intelligent vehicles, extensive efforts have been invested to fuse information from multi-modal data. Despite considerable advantages, fusion-based methods still have inevitable limitations: field-of-view disparity between two modal inputs, demanding precise paired data as inputs in both the training and inferring stages, and consuming more resources. These limitations pose significant obstacles to the practical application of fusion-based methods in real-world scenarios. Therefore, we propose a robust 3D semantic segmentation method based on multi-modal collaborative learning, aiming to enhance feature extraction and segmentation performance for point clouds. In practice, an attention based cross-modal knowledge distillation module is proposed to effectively acquire comprehensive information from multi-modal data and guide the pure point cloud network; then, a confidence-map-driven late fusion strategy is proposed to dynamically fuse the results of two modalities at the pixel-level to complement their advantages and further optimize segmentation results. The proposed method is evaluated on two public datasets (urban dataset SemanticKITTI and off-road dataset RELLIS-3D) and our unstructured test set. The experimental results demonstrate the competitiveness of state-of-the-art methods in diverse scenarios and a robustness to sensor faults. [ABSTRACT FROM AUTHOR]

High dynamic range imaging (HDRI) is an essential task in remote sensing, enhancing low dynamic range (LDR) remote sensing images and benefiting downstream tasks, such as object detection and image segmentation. However, conventional frame-based HDRI methods may encounter challenges in real-world scenarios due to the limited information inherent in a single image captured by conventional cameras. In this paper, an event-based remote sensing HDR imaging framework is proposed to address this problem, denoted as ERS-HDRI, which reconstructs the remote sensing HDR image from a single-exposure LDR image and its concurrent event streams. The proposed ERS-HDRI leverages a coarse-to-fine framework, incorporating the event-based dynamic range enhancement (E-DRE) network and the gradient-enhanced HDR reconstruction (G-HDRR) network. Specifically, to efficiently achieve dynamic range fusion from different domains, the E-DRE network is designed to extract the dynamic range features from LDR frames and events and perform intra- and cross-attention operations to adaptively fuse multi-modal data. A denoise network and a dense feature fusion network are then employed for the generation of the coarse, clean HDR image. Then, the G-HDRR network, with its gradient enhancement module and multiscale fusion module, performs structure enforcement on the coarse HDR image and generates a fine informative HDR image. In addition, this work introduces a specialized hybrid imaging system and a novel, real-world event-based remote sensing HDRI dataset that contains aligned remote sensing LDR images, remote sensing HDR images, and concurrent event streams for evaluation. Comprehensive experiments have demonstrated the effectiveness of the proposed method. Specifically, it improves state-of-the-art PSNR by about 30% and the SSIM score by about 9% on the real-world dataset. [ABSTRACT FROM AUTHOR]

Emotional recognition in conversations (ERC) is increasingly being applied in various IoT devices. Deep learning‐based multimodal ERC has achieved great success by leveraging diverse and complementary modalities. Although most existing methods try to adopt attention mechanisms to fuse different information, these methods ignore the complementarity between modalities. To this end, the joint cross‐attention model is introduced to alleviate this issue. However, multi‐scale feature information on different modalities is not utilized. Moreover, the context relationship plays an important role in feature extraction in the expression recognition task. In this paper, we propose a novel joint hierarchical graph convolution network (JHGCN) which exploits different layer features and context relationships for facial expression recognition based on audio‐visual (A‐V) information. Specifically, we adopt different deep networks to extract features from different modalities individually. For V modality, we construct V graph data based on patch embeddings which are extracted from the transformer encoder. Moreover, we embed the graph convolution which can leverage the intra‐modality relationships with the transformer encoder. Then, the deep feature from different layers is fed to the hierarchical fusion module to enhance feature representation. At last, we use the joint cross‐attention mechanism to exploit the complementary inter‐modality relationships. To validate the proposed model, we have conducted various experiments on the AffWild2 and CMU‐MOSI datasets. All results confirm that our proposed model achieves highly promising performance compared to the joint cross‐attention model and other methods. [ABSTRACT FROM AUTHOR]

Video action segmentation have been widely applied in many fields. Most previous studies employed video-based vision models for this purpose. However, they often rely on a large receptive field, LSTM or Transformer methods to capture long-term dependencies within videos, leading to significant computational resource requirements. To address this challenge, graph-based model was proposed. However, previous graph-based models are less accurate. Hence, this study introduces a graph-structured approach named Semantic2Graph, to model long-term dependencies in videos, thereby reducing computational costs and raise the accuracy. We construct a graph structure of video at the frame-level. Temporal edges are utilized to model the temporal relations and action order within videos. Additionally, we have designed positive and negative semantic edges, accompanied by corresponding edge weights, to capture both long-term and short-term semantic relationships in video actions. Node attributes encompass a rich set of multi-modal features extracted from video content, graph structures, and label text, encompassing visual, structural, and semantic cues. To synthesize this multi-modal information effectively, we employ a graph neural network (GNN) model to fuse multi-modal features for node action label classification. Experimental results demonstrate that Semantic2Graph outperforms state-of-the-art methods in terms of performance, particularly on benchmark datasets such as GTEA and 50Salads. Multiple ablation experiments further validate the effectiveness of semantic features in enhancing model performance. Notably, the inclusion of semantic edges in Semantic2Graph allows for the cost-effective capture of long-term dependencies, affirming its utility in addressing the challenges posed by computational resource constraints in video-based vision models. [ABSTRACT FROM AUTHOR]

In this study, we address the critical need for enhanced situational awareness and victim detection capabilities in Search and Rescue (SAR) operations amidst disasters. Traditional unmanned ground vehicles (UGVs) often struggle in such chaotic environments due to their limited manoeuvrability and the challenge of distinguishing victims from debris. Recognising these gaps, our research introduces a novel technological framework that integrates advanced gesture-recognition with cutting-edge deep learning for camera-based victim identification, specifically designed to empower UGVs in disaster scenarios. At the core of our methodology is the development and implementation of the Meerkat Optimization Algorithm—Stacked Convolutional Neural Network—Bi—Long Short Term Memory—Gated Recurrent Unit (MOA-SConv-Bi-LSTM-GRU) model, which sets a new benchmark for hand gesture detection with its remarkable performance metrics: accuracy, precision, recall, and F1-score all approximately 0.9866. This model enables intuitive, real-time control of UGVs through hand gestures, allowing for precise navigation in confined and obstacle-ridden spaces, which is vital for effective SAR operations. Furthermore, we leverage the capabilities of the latest YOLOv8 deep learning model, trained on specialised datasets to accurately detect human victims under a wide range of challenging conditions, such as varying occlusions, lighting, and perspectives. Our comprehensive testing in simulated emergency scenarios validates the effectiveness of our integrated approach. The system demonstrated exceptional proficiency in navigating through obstructions and rapidly locating victims, even in environments with visual impairments like smoke, clutter, and poor lighting. Our study not only highlights the critical gaps in current SAR response capabilities but also offers a pioneering solution through a synergistic blend of gesture-based control, deep learning, and purpose-built robotics. The key findings underscore the potential of our integrated technological framework to significantly enhance UGV performance in disaster scenarios, thereby optimising life-saving outcomes when time is of the essence. This research paves the way for future advancements in SAR technology, with the promise of more efficient and reliable rescue operations in the face of disaster.

Unexpected seizures significantly decrease the quality of life in epileptic patients. Seizure attacks are caused by hyperexcitability and anatomical lesions of special regions of the brain, and cognitive impairments and memory deficits are their most common concomitant effects. In addition to seizure reduction treatments, medical rehabilitation involving brain–computer interfaces and neurofeedback can improve cognition and quality of life in patients with focal epilepsy in most cases, in particular when resective epilepsy surgery has been considered treatment in drug-resistant epilepsy. Source estimation and precise localization of epileptic foci can improve such rehabilitation and treatment. Electroencephalography (EEG) monitoring and multimodal noninvasive neuroimaging techniques such as ictal/interictal single-photon emission computerized tomography (SPECT) imaging and structural magnetic resonance imaging are common practices for the localization of epileptic foci and have been studied in several kinds of researches. In this article, we review the most recent research on EEG-based localization of seizure foci and discuss various methods, their advantages, limitations, and challenges with a focus on model-based data processing and machine learning algorithms. In addition, we survey whether combined analysis of EEG monitoring and neuroimaging techniques, which is known as multimodal brain data fusion, can potentially increase the precision of the seizure foci localization. To this end, we further review and summarize the key parameters and challenges of processing, fusion, and analysis of multiple source data, in the framework of model-based signal processing, for the development of a multimodal brain data analyzing system. This article has the potential to be used as a valuable resource for neuroscience researchers for the development of EEG-based rehabilitation systems based on multimodal data analysis related to focal epilepsy.

Multimodal sentiment analysis is an important area of artificial intelligence. It integrates multiple modalities such as text, audio, video and image into a compact multimodal representation and obtains sentiment information from them. In this paper, we improve two modules, i.e., feature extraction and feature fusion, to enhance multimodal sentiment analysis and finally propose an attention-based two-layer bidirectional GRU (AB-GRU, gated recurrent unit) multimodal sentiment analysis method. For the feature extraction module, we use a two-layer bidirectional GRU network and connect two layers of attention mechanisms to enhance the extraction of important information. The feature fusion part uses low-rank multimodal fusion, which can reduce the multimodal data dimensionality and improve the computational rate and accuracy. The experimental results demonstrate that the AB-GRU model can achieve 80.9% accuracy on the CMU-MOSI dataset, which exceeds the same model type by at least 2.5%. The AB-GRU model also possesses a strong generalization capability and solid robustness.

Multi-modal fusion of remote sensing images poses challenges because of the intricate imaging mechanisms and variations in radiation across different modalities. Specifically, the fusion of visible-light and vegetation-sensitive images encounters similar difficulties. Traditional methods have seldom considered the varied imaging mechanisms and radiation difference between modalities, resulting in discrepancies in the correspond features. To address the issue, we propose the VAM-Net (Vegetation-Attentive Multi-modal deep Network) combining a radiometric correction mechanism and a lightweight multi-modal adaptive feature selection method for fusing multi-modal images. First, the vegetation index (VDVI) is integrated into visible-light images to mitigate the radiometric differences between visible-light images and vegetation-sensitive images (e.g., infrared and red edge images). Then, a two-branch network incorporating attention mechanisms is designed to independently capture the texture features and select similar features cross two different modalities of images. Last, a new loss function is presented to ensure the learned features are suitable for multi-modal fusion. The VAM-Net is evaluated by visible-light and vegetation-sensitive images in three different areas, and the experimental results show that VAM-Net attains an average precision of 67.02%, and recall of 35.49%, and an average RMSE of 2.191px, demonstrating the accuracy and robustness of VAM-Net in multi-modal image fusion.

Visual grounding aims to recognize and locate the target in the image according to human intention, which provides a new intelligent interaction idea and method for augmented reality (AR) and virtual reality (VR) devices. However, existing vision language grounding adopts language modals for visual grounding, but it performs ineffectively for images containing multiple similar objects. Gaze interaction is an important interaction mode in AR/VR devices, and it provides an advanced solution to the inaccurate vision language grounding cases. Based on the above questions and analysis, a vision language grounding framework fused with gaze intention is proposed. Firstly, we collect the manual gaze annotations using the AR device and construct a novel multi-modal dataset, RefCOCOg-Gaze, combining it with the proposed data augmentation methods. Secondly, an attention-based multi-modal feature fusion model is designed, providing a baseline framework for vision language grounding with gaze intention (VLG-Gaze). Through a series of precisely designed experiments, we analyze the proposed dataset and framework qualitatively and quantitatively. Comparing with the state-of-the-art vision language grounding model, our proposed scheme improves the accuracy by 5.3%, which indicates the significance of gaze fusion in multi-modal grounding tasks. [ABSTRACT FROM AUTHOR]