Search

Monitoring and managing Earth's surface resources is critical to human settlements, encompassing essential tasks such as city planning, disaster assessment, etc. To accurately recognize the categories and locations of geographical objects and reason about their spatial or semantic relations , we propose a multi-task framework named EarthVQANet, which jointly addresses segmentation and visual question answering (VQA) tasks. EarthVQANet contains a hierarchical pyramid network for segmentation and semantic-guided attention for VQA, in which the segmentation network aims to generate pixel-level visual features and high-level object semantics, and semantic-guided attention performs effective interactions between visual features and language features for relational modeling. For accurate relational reasoning, we design an adaptive numerical loss that incorporates distance sensitivity for counting questions and mines hard-easy samples for classification questions, balancing the optimization. Experimental results on the EarthVQA dataset (city planning for Wuhan, Changzhou, and Nanjing in China), RSVQA dataset (basic statistics for general objects), and FloodNet dataset (disaster assessment for Texas in America attacked by Hurricane Harvey) show that EarthVQANet surpasses 11 general and remote sensing VQA methods. EarthVQANet simultaneously achieves segmentation and reasoning, providing a solid benchmark for various remote sensing applications. Data is available at http://rsidea.whu.edu.cn/EarthVQA.htm [ABSTRACT FROM AUTHOR]

Image Salient Object Detection (SOD) is a fundamental research topic in the area of computer vision. Recently, the multimodal information in RGB, Depth (D), and Thermal (T) modalities has been proven to be beneficial to the SOD. However, existing methods are only designed for RGB-D or RGB-T SOD, which may limit the utilization in various modalities, or just finetuned on specific datasets, which may bring about extra computation overhead. These defects can hinder the practical deployment of SOD in real-world applications. In this paper, we propose an end-to-end U nified T riplet D ecoder Net work, dubbed UTDNet, for both RGB-T and RGB-D SOD tasks. The intractable challenges for the unified multimodal SOD are mainly two-fold, i.e., (1) accurately detecting and segmenting salient objects, and (2) preferably via a single network that fits both RGB-T and RGB-D SOD. First, to deal with the former challenge, we propose the multi-scale feature extraction unit to enrich the discriminative contextual information, and the efficient fusion module to explore cross-modality complementary information. Then, the multimodal features are fed to the triplet decoder, where the hierarchical deep supervision loss further enable the network to capture distinctive saliency cues. Second, as to the latter challenge, we propose a simple yet effective continual learning method to unify multimodal SOD. Concretely, we sequentially train multimodal SOD tasks by applying Elastic Weight Consolidation (EWC) regularization with the hierarchical loss function to avoid catastrophic forgetting without inducing more parameters. Critically, the triplet decoder separates task-specific and task-invariant information, making the network easily adaptable to multimodal SOD tasks. Extensive comparisons with 26 recently proposed RGB-T and RGB-D SOD methods demonstrate the superiority of the proposed UTDNet. [ABSTRACT FROM AUTHOR]

• Propose a multiple multi-head attentions network with random missing modalities. • Present a network to obtain missing modalities contextual information. • Achieve the best performance, and the method is robust and convergent. Multi-modal fusion is a hot topic in field of multi-modal learning. Most of the previous multi-modal fusion tasks are based on the complete modality. Existing researches on missing multi-modal fusion fail to consider the random missing of modalities, thereby lacking robustness. And most of methods are based on the correlation between missing and non-missing modalities, ignoring missing modalities contextual information. Considering the above two issues, we designed a multiple multi-head attentions network based on encoder with missing modalities (MMAN-M2). Firstly, the multi-head attention network is used to represent the single modality by extracting potential features based on the entire sequence, and then they are fused; Then, the missing modality context features are extracted by optimizing the result of multi-modal fusion including missing and non-missing features data, and the missing modalities are encoded through the encoding module; Finally, the Transformer encoder-decoder module is used to train the network model by mapping obtaining global information to multiple spaces and integrating our uncertain multi-modal encoding, and it realizes the classification of multi-modal fusion for evaluating model performance. Extensive experiments on multi-modal public datasets show that the proposed method has the best effect and can effectively improve the classification performance of multi-modal fusion. [ABSTRACT FROM AUTHOR]

RGBT person detection benefits numerous vital applications like surveillance, search, and rescue. Meanwhile, drones can capture images holding broad perspectives and large searching regions per frame, which can notably improve the efficacy of large-scale search and rescue missions. In this work, we leverage the advantages of drone-based vision for RGBT person detection. The drone-based RGBT person detection task brings interesting challenges to existing cross-modality object detectors, e.g. , tiny sizes of objects, modality-space imbalance, and position shifts. Observing that there is a lack of data and customized detectors for drone-based RGBT person detection, we contribute two new datasets and design a novel detector. The data contribution is two-fold. For one, we construct the first large-scale drone-based RGBT person detection benchmark RGBTDronePerson, which contains 6,125 pairs of RGBT images and 70,880 instances. Images are captured in various scenes and under various illumination and weather conditions. For another, we annotate the VTUAV tracking dataset and obtain its object detection version, named VTUAV-det. To tackle the challenges raised by this task, we propose a Quality-aware RGBT Fusion Detector (QFDet). Firstly, we design a Quality-aware Learning Strategy (QLS) to provide sufficient supervision for tiny objects while focusing on high-quality samples, in which a Quality-Aware Factor (QAF) is designed to measure the quality. Moreover, a Quality-aware Cross-modality Enhancement module (QCE) is proposed to predict a QAF map for each modality, which not only indicates the reliability of each modality but also highlights regions where objects are more likely to appear. Our QFDet remarkably boosts the detection performance over tiny and small objects, surpassing the strong baseline on mAP 50 tiny by 6.57 points on RGBTDronePerson and mAP s by 3.80 points on VTUAV-det. The datasets, codes, and pre-trained models are available at https://nnnnerd.github.io/RGBTDronePerson/. [ABSTRACT FROM AUTHOR]

Compared to ordinary news, fake news is characterized by faster dissemination and lower production cost and therefore causes a great social harm. For these reasons, the challenge to efficiently and accurately detect fake news has attracted a lot of attention in the research community. We propose a T wo- R ound I nconsistency-based M ulti-m o dal fusi o n N etwork (TRIMOON) for fake news detection, which consists of three main components: the multi-modal feature extraction module, the multi-modal feature fusion module and the classification module. To filter the noise generated in the fusion process, we perform a two-fold inconsistency detection, once before and once after the fusion process. Experimental results also prove this to be quite effective. Our proposed TRIMOON is evaluated on both the Chinese and the English datasets, and our model outperforms the state-of-the-art approaches on several classification evaluation metrics. • A two-round inconsistency-based multi-modal fusion network for fake news detection. • Effectively suppresses fusion noise when images and text are inconsistent. • A multi-modal fusion mechanism for strengthening the leading role of text modality. • Experimental results show that our method obtains the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

• Face-specific depth enhancement to refine the low-quality depth. • Iterative inter-modal feature interaction to fully exploit complementary information. • Feature re-calibration and weighted complementary feature aggregation to strengthen intra-modal representation. • Masked RGB-depth modeling mechanism to improve robustness towards missing-modality data. Existing RGB-based 2D face recognition approaches are sensitive to facial variations, posture, occlusions, and illumination. Current depth-based methods have been proved to alleviate the above sensitivity by introducing geometric information but rely heavily on high-quality depth from high-cost RGB-D cameras. To this end, we propose a Progressive Multi-modal Fusion framework to exploit enhanced and robust face representation for RGB-D facial recognition based on low-cost RGB-D cameras, which also deals with incomplete RGB-D modal data. Due to the defects such as holes caused by low-cost cameras, we first design a depth enhancement module to refine the low-quality depth and correct depth inaccuracies. Then, we extract and aggregate augmented feature maps of RGB and depth modality step-by-step. Subsequently, the masked modeling scheme and iterative inter-modal feature interaction module aim to fully exploit the implicit relations among these two modalities. We perform comprehensive experiments to verify the superior performance and robustness of the proposed solution over other FR approaches on four challenging benchmark databases. [ABSTRACT FROM AUTHOR]

At the heart of affective computing lies the crucial task of decoding human emotions, a field that expertly intertwines emotion identification with the nuances of sentiment analysis. This dynamic discipline harnesses an array of data sources, from the intricacies of textual information to the subtleties of auditory signals and the dynamic realm of visual cues. One of its primary challenges is discerning emotions from physical cues like facial expressions and vocal tones, especially when these emotions are subtly concealed. The precise information yielded by physiological signals is invaluable, yet the complexity of their acquisition in real-world settings remains a formidable challenge. Our comprehensive systematic review marks a significant foray into trimodal affective computing, integrating text, audio, and visual data to provide a holistic understanding. We analyzed over 410 research articles from prominent conferences and journals spanning the last two decades. This extensive study categorizes and critically evaluates a spectrum of affect recognition methods, from unimodal to multimodal approaches, including bimodal and trimodal, offering profound insights into their structural composition and practical effectiveness. In concluding our exploration, we highlight the pivotal aspects of affective computing and chart a course for future groundbreaking research. This includes refining data integration techniques, overcoming challenges in emotion recognition, and addressing the critical ethical dimensions inherent in this field. [ABSTRACT FROM AUTHOR]

In RGB-T salient object detection, effective utilization of the different characteristics of RGB and thermal modalities is essential to achieve accurate detection. Most of the previous methods usually only focus on reducing the differences between modalities, which may ignore the specific features that are crucial for salient object detection, leading to suboptimal results. To address the above issue, an RGB-T SOD network that simultaneously considers the reduction of modality differences and the preservation of specific features is proposed. Specifically, we construct a modality differences reduction and specific features preserving module (MDRSFPM) which aims to bridge the gap between modalities and enhance the specific features of each modality. In MDRSFPM, the dynamic vector generated by the interaction of RGB and thermal features is used to reduce modality differences, and then a dual branch is constructed to deal with the RGB and thermal modalities separately, employing a combination of channel-level and spatial-level operations to preserve their respective specific features. In addition, a multi-scale global feature enhancement module (MGFEM) is proposed to enhance global contextual information to provide guidance information for the subsequent decoding stage, so that the model can more easily localize the salient objects. Furthermore, our approach includes a fully fusion and gate module (FFGM) that utilizes dynamically generated importance maps to selectively filter and fuse features during the decoding process. Extensive experiments demonstrate that our proposed model surpasses other state-of-the-art models on three publicly available RGB-T datasets remarkably. Our code will be released at https://github.com/JOOOOKII/FRPNet. • The network introduces MDRSFPM, which minimizes modality differences while retaining key features. • Implementation of MGFEM enhances global contextual understanding, essential for improving saliency detection across various scenarios. • FFGM dynamically integrates features, reducing redundancy and focusing on the most relevant features for saliency. [ABSTRACT FROM AUTHOR]

Parkinson's disease (PD) is a prevalent neurological disorder characterized by progressive dopaminergic neuron loss, leading to both motor and non-motor symptoms. Early and accurate diagnosis is challenging due to the subtle and variable nature of early symptoms. This study aims to address these diagnostic challenges by proposing a novel method, Localized Region Extraction and Multi-Modal Fusion (LRE-MMF), designed to enhance diagnostic accuracy through the integration of structural MRI (sMRI) and resting-state functional MRI (rs-fMRI) data. The LRE-MMF method utilizes the complementary strengths of sMRI and rs-fMRI: sMRI provides detailed anatomical information, while rs-fMRI captures functional connectivity patterns. We applied this approach to a dataset consisting of 20 PD patients and 20 healthy controls (HC), all scanned with a 3 T MRI. The primary objective was to determine whether the integration of sMRI and rs-fMRI through the LRE-MMF method improves the classification accuracy between PD and HC subjects. LRE-MMF involves the division of imaging data into localized regions, followed by feature extraction and dimensionality reduction using Principal Component Analysis (PCA). The resulting features were fused and processed through a neural network to learn high-level representations. The model achieved an accuracy of 75 %, with a precision of 0.8125, recall of 0.65, and an AUC of 0.8875. The validation accuracy curves indicated good generalization, with significant brain regions identified, including the caudate, putamen, thalamus, supplementary motor area, and precuneus, as per the AAL atlas. These results demonstrate the potential of the LRE-MMF method for improving early diagnosis and understanding of PD by effectively utilizing both sMRI and rs-fMRI data. This approach could contribute to the development of more accurate diagnostic tools. • Developed a novel framework to integrate sMRI and fMRI for enhanced Parkinson's disease diagnosis. • Achieved classification model in distinguishing Parkinson's patients from healthy controls using multi-modal MRI data. • Significant brain regions associated with Parkinson's pathology identified, including the caudate and putamen. • Demonstrated the effectiveness of feature extraction and dimensionality reduction techniques in multi-modal neuroimaging analysis. • LRE-MMF offers a promising approach for improving the understanding of Parkinson's disease. [ABSTRACT FROM AUTHOR]

Referring image segmentation aims to generate a binary mask of the target object according to a referring expression. Some recent works argue that post-fusion paradigm may result in inconsistency and insufficiency issue and propose to integrate textual features during the visual encoding process. Although effective, they do fusion in a single way at each stage of encoder, e.g. utilizing cross attention mechanisms. This single fusion method ignores local and detailed image information correlated with language due to the incapability of attention in capturing high-frequencies information. To address this issue, we propose a Token-Word Mixer, which takes into consideration the characteristics of convolution and attention, and achieves more comprehensive interactions and alignments of multi-modal features through a mix operation. Furthermore, existing methods that rely solely on grid features lack perception of the target object and inference of relationships between objects, making it difficult to associate and align semantic information of target objects during multi-modal fusion when referring expressions or image scenes are complex. Therefore, we propose to incorporates object-level information by exploiting a DETR-based detector to provide region features, and the Object-Aware Transformer encoder with an additional learnable token is proposed to perceive effective information associated with the target object. Based on the enhanced cross-modal features and the aggregated token, we adopt query-based mask generation method instead of pixel classification framework for referring image segmentation. Extensive experiments and ablation studies indicate the effectiveness of our proposed methods. • To effectively explore interactions across multiple modalities, we propose a Token-Word Mixer, which enhances both local and global feature integration, thereby improving the alignment of multi-modal features. • To enhance the perception capability of the target object, we propose the Object-Aware Transformer, which integrates object-level information to align the semantic details and infer relationships between objects under the guidance of a referring expression. • We conduct extensive experiments on three prevalent referring image segmentation datasets, and our framework achieves consistent improvements over state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

RGB-T tracking takes visible and infrared images as inputs, which is an extended application of multi-modal fusion in the field of visual object tracking. The complementarity between visible and infrared modalities can enhance the robustness of tracker in complex scenes. Cross-modal interaction can facilitate the fusion and synergy of different modalities, but most previous methods lack clear target information in multi-modal fusion, leading to some undesired cross-relation in interaction. To reduce these undesired cross-relations, we propose a Multi-modal Interaction scheme Guided by Token Division strategy (MIGTD). This scheme divides the input multi-modal tokens into several categories and restricts the interaction between tokens by setting different rules. The above operation is implemented in parallel through an attention masking strategy. To accurately classify search tokens, an instance segmentation task with box-supervised loss is employed. We conduct extensive experiments on three popular benchmark datasets, RGBT234, LasHeR and VTUAV. The experimental results indicate that the tracker proposed in this article reach the world's advanced level in performance. • To our knowledge, our method sets new state-of-the-art on several benchmarks. • We propose a multi-modal interaction scheme based on token division strategy. • Input tokens are divided into eight categories based on consistency and complementarity. • We introduce an segmentation task to act as a token labeler for interaction scheme. • The token labeler is semi-supervised trained using box annotations. [ABSTRACT FROM AUTHOR]

Single-exposure High Dynamic Range Imaging (HDRI), as a typical ill-posed problem, has attracted extensive attention from researchers. However, restoration in real-world scenarios has always been an intractable task due to various exposures and noise artifacts. This work proposes an event-based HDRI framework that generalizes to scenes under various exposures by exploiting the high dynamic range of events. To address the challenge of processing diverse exposures, we propose an exposure-aware network incorporating the exposure attention fusion module, which facilitates the adaptive fusion of SDR image and event features. Moreover, the problem of noise in extremely under-exposed regions and events is effectively alleviated by introducing a self-supervised loss, namely EDDN, which effectively enhances the details of saturated areas while simultaneously decreasing noise. We conduct novel event-based HDRI datasets to evaluate our proposed method for benchmarking with diverse exposed images. Comprehensive experiments have demonstrated that our method outperforms the state-of-the-art. [ABSTRACT FROM AUTHOR]

Entity Alignment (EA) is a crucial task in knowledge fusion, which aims to link entities with the same real-world identity from different Knowledge Graphs (KGs). Existing methods have achieved satisfactory performance, however, they mainly focus on single modal KG, which is difficult to be effectively applied to multi-modal scenes. In this paper, we propose a Multi-modal Joint entity Alignment Framework (MultiJAF), which can effectively utilize the knowledge of various modalities. Concretely, we first learn the embeddings of different modalities, i.e., structure, attribute and image modalities. Next, we adopt an attention-based multi-modal fusion network to integrate these embeddings and use obtained joint embeddings to compute a joint embedding-based similarity matrix S J . Moreover, we design a Numerical Process Module (NPM) to infer a similarity matrix S N according to the numerical information of entities. In the end, we utilize a simple late fusion method to ensemble two similarity matrices for the final alignment. In addition, to reduce the cost of labeling data, we propose a novel NPM-based unsupervised multi-modal EA method. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed MultiJAF. [ABSTRACT FROM AUTHOR]

The goal of semantic segmentation is to segment the input image according to semantic information and predict the semantic category of each pixel from a given label set. With the gradual intellectualization of modern life, more and more applications need to infer relevant semantic information from images for subsequent processing, such as augmented reality, autonomous driving, video surveillance, etc. This paper reviews the state-of-the-art technologies of semantic segmentation based on deep learning. Because semantic segmentation requires a large number of pixel-level annotations, in order to reduce the fine-grained requirements of annotation and reduce the economic and time cost of manual annotation, this paper studies the works on weakly-supervised semantic segmentation. In order to enhance the generalization ability and robustness of the segmentation model, this paper investigates the works on domain adaptation in semantic segmentation. Many types of sensors are usually equipped in some practical applications, such as autonomous driving and medical image analysis. In order to mine the association between multi-modal data and improve the accuracy of the segmentation model, this paper investigates the works based on multi-modal data fusion semantic segmentation. The real-time performance of the model needs to be considered in practical application. This paper analyzes the key factors affecting the real-time performance of the segmentation model and investigates the works on real-time semantic segmentation. Finally, this paper summarizes the challenges and promising research directions of semantic segmentation tasks based on deep learning. [ABSTRACT FROM AUTHOR]

With artificial intelligence and deep learning development, crop disease recognition methods leverage deep networks for automatic feature learning but rely on the volume of training data. Addressing the scarcity of data in agriculture, few-shot learning (FSL) and multi-modal learning have become focal points. However, existing methods are confined to a single modality or insufficiently exploit cross-modal features. To address this, we propose a multi-modal contrastive learning approach integrating images and text to tackle the problem of small-sample recognition. This method combines CLIP multi-modal pre-training with cross-attention, termed ITIMCA. Experimental validation demonstrates the effectiveness of our approach in cassava leaf disease recognition tasks under natural conditions. Experimental results show that the proposed model achieved an accuracy of 78.00%, a precision of 88.48%, a recall rate of 80.00%, and an F1-Score of 79.00% on the cassava leaf disease identification and classification dataset. These results suggest that the proposed network effectively identifies cassava leaf diseases. • Rewriting high-quality cassava leaf description text using LLM models. • Constructing a multimodal dataset using collected cassava leaf images and rewritten text. • Proposing a novel framework ITIMCA that utilizes image-text contrastive learning and cross-attention. • ITIMCA demonstrates significant effectiveness in identifying cassava leaf diseases under limited data and complex natural conditions. [ABSTRACT FROM AUTHOR]

Multi-modal data fusion for effective feature representation in machine learning is challenging due to intrinsic biases present within and across different modalities. Existing multi-modal data fusion methods often face difficulties in learning generic features due to diverse noise patterns and variations in feature dynamics across different modalities. In this paper, we present a novel method called Uncertainty-guided Meta-Learning Multi-modal Fusion and Classification (UMLMC) to address these challenges. UMLMC dynamically transforms multi-modal feature spaces at both the pre- and post-fusion levels by incorporating uncertainty estimates from an auxiliary network. Our model is optimized using a meta-learning algorithm to enhance its generalization capabilities. Extensive experiments on multi-modal data from diverse domains, along with comparisons to state-of-the-art methods, demonstrate the effectiveness of UMLMC in improving classification performance. These results confirm that UMLMC, with its innovative uncertainty estimation and meta-learning framework, effectively learns informative intra- and inter-modal features, leading to superior classification outcomes. • An uncertainty-guided meta-fusion method for multi-modal fusion and classification. • Mitigating the impact of feature-level bias at both before and after fusion. • Meta-learning to generate less biased uncertainty estimation at the feature level. [ABSTRACT FROM AUTHOR]

RGB-Thermal Video Object Detection (RGBT VOD) is to localize and classify the predefined objects in visible and thermal spectrum videos. The key issue in RGBT VOD lies in integrating multi-modal information effectively to improve detection performance. Current multi-modal fusion methods predominantly employ middle fusion strategies, but the inherent modal difference directly influences the effect of multi-modal fusion. Although the early fusion strategy reduces the modality gap in the middle stage of the network, achieving in-depth feature interaction between different modalities remains challenging. In this work, we propose a novel hybrid fusion network called PTMNet, which effectively combines the early fusion strategy with the progressive interaction and the middle fusion strategy with the temporal-modal difference, for high performance RGBT VOD. In particular, we take each modality as a master modality to achieve an early fusion with other modalities as auxiliary information by progressive interaction. Such a design not only alleviates the modality gap but facilitates middle fusion. The temporal-modal difference models temporal information through spatial offsets and utilizes feature erasure between modalities to motivate the network to focus on shared objects in both modalities. The hybrid fusion can achieve high detection accuracy only using three input frames, which makes our PTMNet achieve a high inference speed. Experimental results show that our approach achieves state-of-the-art performance on the VT-VOD50 dataset and also operates at over 70 FPS. The code will be freely released at https://github.com/tzz-ahu for academic purposes. • A hybrid fusion strategy network for RGB-Thermal video object detection. • An early strategy for reducing modal disparities. • A novel differential method for modeling multimodal and temporal information. • The proposed PTMNet achieves SOTA performance on the VT-VOD50 dataset. [ABSTRACT FROM AUTHOR]

• A lightweight dual-stream backbone instance segmentation model named RTMFusion fusing RGB and Depth has been developed. • A Depth Enhanced Feature Fusion (DEFF) module for multi-modal complementary feature fusion is designed. • A fruit-bearing organ matching method based on distance constraints and conditional search method is proposed. • The mask mAP improved by 4.6 % compared to the baseline model RTMDet, reaching 72.1 %, with a speed of 17.1 ms. • The tomato organ matching method achieves a pedicel matching accuracy of 99.2 % and a stem matching accuracy of 97.3 %. Accurate and rapid acquisition of semantically annotated point clouds from tomato cultivation regions is crucial for autonomous tomato harvesting robots. This entails the robot's ability to perceive tomato point clouds from various locations and identify the stems and pedicels connecting the fruits. However, amidst background interference from similar stem and pedicel features, the heterogeneous growth postures of tomato clusters and the slender and short nature of pedicels pose challenges in semantic point cloud segmentation and matching of different tomato fruits, pedicels, and stems. Leveraging the salient feature representation of depth images in fine contours and distance dimensions, this paper introduces RTMFusion, a lightweight dual-stream backbone instance segmentation model that integrates color images and depth images. A Depth Enhanced Feature Fusion module is designed for multi-modal feature fusion. Fruit-bearing organ matching is facilitated by a method based on distance constraints and conditional search. Key characteristics are extracted from semantic point clouds, and an optimal condition search is conducted on candidates meeting constraint conditions. On the test dataset, our model improved the mask mAP by 4.6 % compared to the baseline, achieving 72.1 % with a speed of 17.1 ms. Our matching method achieved 99.2 % pedicel matching accuracy and 97.3 % stem matching accuracy on 150 greenhouse samples. These results demonstrate the effectiveness of accurately identifying and matching tomato organs in real-world scenarios, addressing the challenge from complex greenhouse environments. This integrated visual system enhances the success rate of visual identification and the precision of cutting position localization for tomato harvesting robots. [ABSTRACT FROM AUTHOR]

Depth completion is a low-level task rebuilding the dense depth from a sparse set of measurements from LiDAR sensors and corresponding RGB images. Current state-of-the-art depth completion methods used complicated network designs with much computational cost increase, which is incompatible with the realistic-scenario limited computational environment. In this paper, we explore a lightweight and efficient depth completion model named Light-SEF. Light-SEF is a two-stage framework that introduces local fusion and global fusion modules to extract and fuse local and global information in the sparse LiDAR data and RGB images. We also propose a unit convolutional structure named spatial efficient block (SEB), which has a lightweight design and extracts spatial features efficiently. As the unit block of the whole network, SEB is much more cost-efficient compared to the baseline design. Experimental results on the KITTI benchmark demonstrate that our Light-SEF achieves significant declines in computational cost (about 53% parameters, 50% FLOPs & MACs, and 36% running time) while showing competitive results compared to state-of-the-art methods. [Display omitted] • Light-SEF is a local-to-global two-stage depth completion framework. • Light-SEF benefits from fusion between local and global multi-modal features. • Spatial efficient block (SEB) allow the overall network to achieve cost-efficient. • Light-SEF showed significant computational cost declines with competitive results. [ABSTRACT FROM AUTHOR]

Fake news can mislead public perception and have a negative impact on people's lives. With the increase of multi-modal news content in social media, multi-modal fake news detection has received extensive attention from researchers. However, existing methods face difficulties in capturing temporal intensity and deep social graph representations. Considering the multi-modality of news content and the relationship between entities, this paper proposes a multi-modal fake news detection model based on dynamic propagation social graph (DPSG). Specifically, it first utilizes the temporal penetration fusion block to perform multi-modal dynamic fusion on various types of nodes in the propagation sequence, and then employs dynamic signed weighting to enhance the social graph features of the propagation graph. Finally, the Transformer captures news propagation patterns and fuses multi-domain information. Experimental results on three datasets demonstrate the effectiveness of the proposed model on the fake news detection task. • A dynamic propagation social graphs for multi-modal fake news detection. • Temporal similarity strength captures dynamic information. • Enhanced graph structure captures the dynamic semantics of propagation patterns. • Experimental results show that our method obtains the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Unmanned Aerial Vehicles (UAVs) exhibit superior maneuverability and large-scale movement capability in 3D space, making them ideal platforms for various autonomous applications. However, UAVs with cameras face challenges in 3D situational awareness, which limits their applications in 3D space, particularly in Lake Search and Rescue (LSAR) scenarios. In this paper, we propose a new LSAR Aerial 3D detection dataset (LA3D) and a pioneering end-to-end aerial Monocular-GPS Fusion 3D detector (MGF3D) to improve UAVs' 3D understanding capability and mitigate the gap between research and real-world application. LA3D is collected by a well-designed hybrid-wing UAV platform on lake surface scenes, which has high-quality annotated 3D bounding boxes, aligned GPS altitudes, and comprehensive metrics. Moreover, MGF3D aggregates multi-modal features with rich geometric and semantic information and feeds them into the 3D detection head for final predictions. To achieve this, we design the Altitude Encoder (AE) module to extract geometric GPS altitude features and introduce Image Encoder (IE) to encode semantic image features respectively. The plug-and-play Monocular-GPS Fusion (MGF) module is proposed to fuse GPS altitude features with image features, which adaptively learns 3D space information and enhances long-distance depth estimation capability. The extensive experiments on the LA3D dataset validate that MGF3D significantly improves aerial 3D detection accuracy. Furthermore, the MGF3D is deployed on a UAV embedded platform to perform the challenging real-world LSAR flight task, demonstrating the effectiveness and practicality of our LA3D and MGF3D. We plan to release the dataset and code in here. [ABSTRACT FROM AUTHOR]

As a hot topic in brain diseases, diagnostic and clinical score prediction of subjects based on multi-modal images helps assess pathological stages and estimate disease progression. Since brain atrophy occurs only in localized regions, previous patch-based deep learning methods require pre-determination of discriminative locations in the brain. In other words, the features extracted from the pre-determined potential atrophy locations are not fully adapted to the tasks in subsequent stages. Besides, most methods focus only on single-modal information with a single task, thus ignoring the intrinsic correlation between multi-modal information and multi-task variables. Furthermore, simply discarding subjects with incomplete clinical scores limits the number of available subjects. In this paper, we propose a multi-task joint learning network (MTJLN) for both brain disease diagnosis and clinical score prediction using multi-modal data and incomplete clinical scores. Specifically, we divided the brain images into 216 local patches covering all potential lesion locations. Then, an image patch pruning algorithm is designed for pruning the information-poor patches. The fine-grained multi-modal image features based on patches and coarse-grained non-image features are fused in the middle layer and used to predict multi-task variables. The ingenious design of the weighted loss function enables subjects with incomplete clinical scores to participate in network training. The experimental results of 842 subjects from the ADNI database demonstrate that the proposed method can effectively predict the pathological stage and clinical score of subjects. • Image patch pruning algorithm prunes information-poor local patches. • Reciprocity of categorical and scoring variables contributed to multitask prediction. • Multi-task weighted loss makes subjects with incomplete clinical scores available. • Hybrid fusion of fine-grained image features and coarse-grained non-image features. • Multi-modal features can model subjects more comprehensively. [ABSTRACT FROM AUTHOR]

RGB-T tracking, a vital downstream task of object tracking, has made remarkable progress in recent years. Yet, it remains hindered by two major challenges: (1) the trade-off between performance and efficiency; (2) the scarcity of training data. To address the latter challenge, some recent methods employ prompts to fine-tune pre-trained RGB tracking models and leverage upstream knowledge in a parameter-efficient manner. However, these methods inadequately explore modality-independent patterns and disregard the dynamic reliability of different modalities in open scenarios. We propose M3PT, a novel RGB-T prompt tracking method that leverages middle fusion and multi-modal and multi-stage visual prompts to overcome these challenges. We pioneer the use of the adjustable middle fusion meta-framework for RGB-T tracking, which could help the tracker balance the performance with efficiency, to meet various demands of application. Furthermore, based on the meta-framework, we utilize multiple flexible prompt strategies to adapt the pre-trained model to comprehensive exploration of uni-modal patterns and improved modeling of fusion-modal features in diverse modality-priority scenarios, harnessing the potential of prompt learning in RGB-T tracking. Evaluating on 6 existing challenging benchmarks, our method surpasses previous state-of-the-art prompt fine-tuning methods while maintaining great competitiveness against excellent full-parameter fine-tuning methods, with only 0.34 M fine-tuned parameters. Our code are available at https://github.com/rainbowsea123/M3PT. [ABSTRACT FROM AUTHOR]

Multi-modal emotion recognition in conversation is challenging because of the difficulty to jointly leverage the information from heterogeneous text, acoustic, and visual modalities. Recent context-aware methods usually design a graph structure to model dependencies of utterances and speakers, or integrate Multi-modal information. However, they typically lack a sufficient extraction of unimodal context, and rarely explore the emotion consensus prototypes among different samples with the same label. For solving these problems, in this paper, we propose a Graph Context extraction and Consensus-aware Learning (GCCL) framework to excavate context-sensitive fusion features and simulate the emotion evocation process during the emotion consensus learning. Specifically, GCCL contains a well-designed graph-based module to capture speaker, temporal and modality dependencies and integrate information from different modalities. Then, we design an emotion consensus learning unit to mine the most typical feature of each category in each modality. A speaker-guided contrastive learning loss is further proposed to guarantee the diversity between different individuals and the semantic consistency between distinct modalities. Moreover, we construct a consensus-aware unit with an attention-based memory mechanism to preserve semantic correlations among different samples on the category-level. Extensive experimental results on two conversational datasets demonstrate that the proposed GCCL outperforms the state-of-art methods. Code is available at https://github.com/gityider/GCCL. [ABSTRACT FROM AUTHOR]

• Multi-modal fusing algorithm with rice disease image and non-imaging spectral signals. • Image feature extraction with a new neural network root in MobilenetV2. • Linear SVM method with fused features obtained by effective fusion strategy for crop disease classification. Frequent occurrences of pests and diseases in farmland pose a severe threat to crop yield and quality. Various techniques utilized for the automatic detection of diseases and pests are mostly based on a single modality with limited diagnostic information. In this study, a multi-modal classification algorithm, ISMSFuse, was proposed to fuse the image and spectral information. This algorithm combines a MobilenNetV2 lightweight neural network with a traditional low-computational linear SVM method to adapt to edge computing terminals. The spectral information was extracted with non-imaging spectral signals to reduce the acquisition costs and hardware requirements. A proprietary multi-mode rice bacterial blight dataset, denoted as ImgS-RBB2022, was curated specifically for evaluating the algorithm. The effectiveness of the image features extracted by the algorithm model was evaluated by the t-SNE method and the Fisher scores. Finally, comparing the results with only using imaging data, the multi-mode fusion algorithm had a superior recognition accuracy of 98.14%. Through the SHAP method, it was shown that the added spectral features became the most critical feature affecting the output of the algorithm model. To further explore the application potential of the proposed algorithm, we deployed it on a board-based microcomputer Raspberry Pi to test its feasibility in an edge computing circumstance. The proposed method can be applied to various agricultural diseases and pest detection tasks, particularly in the case of foliar-type diseases and pests. [ABSTRACT FROM AUTHOR]

Nowadays, data is of paramount importance for artificial intelligence. However, collecting real-world hyperspectral images (HSIs) with desired characteristics and diversity can be prohibitively expensive and time-consuming, leading to the data scarcity issue in HSI, and further limiting the potential of deep learning-based HSI applications. Existing work to tackle this issue fails to generate abundant, diverse, and reliable synthetic HSIs. This work proposes a multi-modal scene fusion-based method that diffusion from the abundance perspective for HSI synthesis, termed MSF-Diff. Concretely, highlights involve: (1) Synthesis in low-dimensional abundance space, other than original high-dimensional HSI space, greatly releases the difficulty; (2) Integration of multi-modal data greatly enriches the diversity of spatial distribution that the model can perceive; (3) Incorporation of the unmixing concept ensures that the generated synthetic HSI has reliable spectral profiles. The proposed research can generate a vast amount of HSI with a rich diversity in various categories and scenes, closely resembling realistic data. It plays a pivotal role in ensuring that the model produces reliable results and can be trusted for real-world applications. The code is publicly available at https://github.com/EtPan/MSF-Diff. • Incorporation of the unmixing concept ensures that the generated data has reliable spectral profiles. • Generation in abundance space, other than the cube space, greatly releases the difficulty in HSI synthesis. • Integration of multi-modal data enriches the diversity of spatial distribution that the model can perceive. [ABSTRACT FROM AUTHOR]

Call centers are critical for gathering customer feedback, making them essential for business communication. Predicting the ongoing business process status accurately has become a focus in both academia and industry. However, current methods mainly analyze process sequence data from enterprise information systems, missing out on valuable data from other sources like call centers. Moreover, these methods often focus on a single task, ignoring the shared information across multiple tasks. This paper presents a novel method for business process prediction that fuses multi-modal data from both information systems and call centers. Specifically, the method combines sequence data from the enterprise information system and dialogue text data from the call center for a more enriched business process prediction. Additionally, to navigate the multi-task learning conundrum, we improve the existing MMoE algorithm and introduce a new multi-task learning architecture called Heterogeneous Multi-gate Mixture-of-experts. The experimental results over some current approaches like Transformer, CNN and LSTM show superior prediction performance compared to baseline models, demonstrating that our method can help call centers optimize their processes, improve customer service, and drive business success. • Innovates business process prediction in call centers using multi-modal fusion. • Proposes a multitask model for effective multi-source data integration. • Provides empirical validation of improved prediction performance in call center settings. [ABSTRACT FROM AUTHOR]

Depression has become a worldwide public health concern in recent years. The online social network can be served as a tool to predict latent depressed users. However, existing studies primarily use machine learning or deep learning methods on multi-modal information with simple concatenation, resulting in unsatisfactory prediction. Additionally, the lack of interpretability in extracted features is a challenge. Furthermore, the majority of these studies primarily concentrate on English social networks. To address these issues, we propose a novel unified multi-tasking training model named CrossAMF (Cross-modal Attention-based Multi-modal Fusion). CrossAMF extracts crucial multi-modal features with explainability and effectively fuses representations of different modalities. Our approach employs a sentiment filtering-based XLNet-CNN-BiGRU model and Reduce-VGGNet to extract textual and image features, respectively. We also introduce innovative user behavior metrics previously unexplored. We collect a dataset of Chinese depressed users from Sina microblog with their texts and images, including 5692 depressed users and 9121 normal users. Experimental results demonstrate the significance of each modality to our task, with the combination yielding the best performance. CrossAMF outperforms other state-of-the-art methods with 0.9097 in accuracy and 0.8915 in F1, showing the potential of application in screening out early depression automatically. [ABSTRACT FROM AUTHOR]

New product sales prediction is crucial for the digital economy as it enables businesses to make informed decisions about product development, inventory management, marketing strategies, and ultimately driving economic growth and innovation. In the digital economy era, traditional sales forecasting methods often struggle to address the unique challenges of forecasting demand for new products, primarily due to limited historical data and high levels of uncertainty. To address this challenge, we propose a multi-modal transform-based fusion model for new product sales prediction (M2TFM), which integrates multiple data sources (e.g., product images, attributes, text descriptions and context factors like holidays, weather and trends.) to predict new product sales with remarkable accuracy. The proposed method leverages diffusion embedding to fuse heterogeneous data modalities including images, text, and time series into a unified representation that models their complex interactions. By encoding multi modal data using Transformer self-attention, our approach is able to extract nuanced signals across modalities to make more accurate new product sales forecasts. We perform a comprehensive evaluation on a large e-commerce dataset with more than 10,000 fashion items, and the results demonstrate that the proposed method is more effective than existing state-of-the-art baselines for new product sales forecasting. [ABSTRACT FROM AUTHOR]

Depth completion is a crucial task in autonomous driving, aiming to convert a sparse depth map into a dense depth prediction. The sparse depth map serves as a partial reference for the actual depth, and the fusion of RGB images is frequently employed to augment the completion process owing to its inherent richness in semantic information. Image-guided depth completion confronts three principal challenges: (1) the effective fusion of the two modalities; (2) the enhancement of depth information recovery; and (3) the realization of real-time predictive capabilities requisite for practical autonomous driving scenarios. In response to these challenges, we propose a concise but high-performing network, named CHNet, to achieve high-performance depth completion with an elegant and straightforward architecture. Firstly, we use a fast guidance module to fuse the two sensor features, harnessing abundant auxiliary information derived from the color space. Unlike the prevalent complex guidance modules, our approach adopts an intuitive and cost-effective strategy. In addition, we find and analyze the optimization inconsistency problem for observed and unobserved positions. To mitigate this challenge, we introduce a decoupled depth prediction head, tailored to better discern and predict depth values for both valid and invalid positions, incurring minimal additional inference time. Capitalizing on the dual-encoder and single-decoder architecture, the simplicity of CHNet facilitates an optimal balance between accuracy and computational efficiency. In benchmark evaluations on the KITTI depth completion dataset, CHNet demonstrates competitive performance metrics and inference speeds relative to contemporary state-of-the-art methodologies. To assess the generalizability of our approach, we extend our evaluations to the indoor NYUv2 dataset, where CHNet continues to yield impressive outcomes. The code of this work will be available at https://github.com/lmomoy/CHNet. [ABSTRACT FROM AUTHOR]

Salient object refers to the conspicuous objects or regions within an image that stand out prominently from its surroundings. Depth maps are commonly utilized as supplementary inputs for salient object detection, referred to as RGB-D SOD. Due to the diverse acquisition sensors, such as infrared detectors and stereo cameras, the quality of acquired depth maps varies considerably. The low-quality depth introduces noise that seriously reduces detection accuracy. To tackle this problem, a triple attention architecture based on a 3D convolutional neural network tailored for quality-aware salient object detection is proposed in this paper, which capitalizes on the strengths across modality, channel, and spatial dimensions. The modality attention learns the quality factors based on the overall modal features. The channel attention highlights features in the dimension of channels, and the patch-level spatial attention establishes long-range dependencies. Thus, the quality factors, channel differences, and spatial contrast are combined to achieve global and local fusion. To enable the evaluations on low-quality depth maps, an assessment criterion is further introduced to categorize the RGB-D datasets. Experimental results of state-of-the-art methods on different quality levels demonstrate the proposed method's effectiveness, especially for the low-quality depth. • A triple attention framework for quality-aware salient object detection. • Modality, channel, and patch-level spatial attention are effective for saliency detection. • A novel quality assessment criterion for depth maps. [ABSTRACT FROM AUTHOR]

Gliomas are the most common intracranial primary tumors. Accurate grading of gliomas is crucial in determining treatment options and prognosis. Clinicians conventionally rely on multiple Magnetic Resonance Imaging (MRI) sequences for accurate glioma assessment. Traditional deep learning methods typically utilize the individual MRI sequence with the annotations of Region of Interest (ROIs), incurring substantial manual effort while losing the complementary information provided by other sequences. This task is inherently a standard multi-modal problem. However, existing methods often adopt complex multi-stream networks for feature extraction and fusion, leading to high resource demands and potential feature redundancy. To address the above challenge, a discrepancy-aware self-distillation method is proposed for multi-modal glioma grading, which requires only a single-stream network to concurrently analyze multiple MRI sequences without auxiliary ROIs. The first Modality Discrepancy-aware Fusion (MDF) module considers the MRI imaging differentiation and widens the inter-modal contrasts, allowing the modality-specific features to be highlighted and the modality-invariant features to be diminished. Furthermore, the proposed Class Activation Self-Distillation (CASD) strategy leverages the generated Class Activation Maps (CAMs) as dark knowledge for the distillation process. This guides shallow layers to focus on the category discriminative features specifically within the lesion region. Extensive experiments were conducted on the BraTS2018 and BraTS2019 datasets to evaluate the effectiveness of our method. The results show that our method outperforms other relevant glioma grading and multi-modal fusion methods on both datasets. • We propose an enhanced single-stream framework for multi-modal glioma grading. • Modality discrepancy-aware fusion module considers the MRI imaging differentiation. • Class activation self-distillation strategy transfers the CAMs as dark knowledge. • Our method achieves satisfactory results on the BraTS benchmarks. [ABSTRACT FROM AUTHOR]

Three dimensional (3D) object detection using the camera and light detection and ranging (LiDAR) fusion model has received much attention for meeting the environmental sensing requirement in autonomous vehicles. Due to the intrinsic data difference and the uneven distribution of LiDAR data, integrating the camera and LiDAR to solve the problem of the uneven spatial distribution of LiDAR data is still challenging. This paper proposes a novel 3D detector, which decorates the point cloud voxels with the semantic features at multi-scale and fuses the multi-source features by cross-attention to improve the detection performance. Specifically, the multi-modal fusion aims to maintain the cross-modal consistency by using the frustum model to construct the correspondence between point voxels and image pixels at multi-scale. To fully exploit the geometric constraints of voxels, we developed a adaptive sampling radius module to dynamically select the sampling radius of the voxel set abstraction module based on softmax. Moreover, we propose a multi-source fusion module which utilizes the cross-attention and takes the raw point cloud's spatial distribution as the clue to fuse the features of point clouds, bird's eye view information and the aggregated voxels to obtain the multi-scale and multi-source fusion features. Finally, region proposal network is adopted to generate and refine the 3D bbox and class prediction based on the fusion feature and bird's eye view feature. Extensive experiments are conducted on the two publicly available 3D object datasets proposed by Karlsruhe Institute of technology and Toyota Technological Institute (KITTI) and the nuScenes. The proposed model reaches 86.27% mean Average Precision (mAP) for car on KITTI and obtains 6.56% gains than PointPainting. Moreover, our model reaches 70.70% mAP on nuScens and improves 2.61% than Transfusion. Furthermore, comprehensive ablation experiments are conducted to validate the effects and contribution of the different components of the proposed model. [ABSTRACT FROM AUTHOR]

Affective computing, a field at the intersection of cognitive science, linguistics, and AI, seeks to enhance human–computer interactions. Recognizing the complexity of human emotions, which manifest across various channels, this paper advocates for a multi-modal approach to accurately recognize emotions. Such an approach enables the discernment of subtle emotional cues in multiple modalities, thus advancing the field of multi-modal affective computing. Focusing on a trimodal framework, this paper examines emotion recognition and sentiment analysis through text, voice, and visual data. It outlines key developments, current trends, and prominent datasets in trimodal emotional analysis. It also explores data fusion strategies across modalities and assesses various fusion techniques' effectiveness. The paper presents detailed emotion models, recent advancements, and key trimodal databases, while thoroughly addressing challenges like data processing and the complexities of TAC. Finally, it highlights potential future directions, underscoring the importance of benchmark databases and practical applications to deepen our understanding of the nuanced spectrum of human emotions. • Trimodal affective computing enhances emotion recognition. • Reviews and categorizes studies by type, results, and accuracy. • Discusses trimodal computing's use in human-machine interactions. [ABSTRACT FROM AUTHOR]

Visual dialog aims to accomplish multiple rounds of dialog by fusing information extracted from images, captions, and previous question–answer pairs. As a vision-language task, visual dialog encounters challenges related to language bias and vision bias. These biases create an imbalance in multi-modal fusion, resulting in shortcut learning and significantly compromising the model's robustness. Moreover, existing multi-modal fusion methods in visual dialog exhibit a low data interaction frequency, leading to insufficient fusion. To overcome the balance and sufficiency issues in multi-modal fusion, we propose a novel Parallel Attention Fusion visual dialog model with Counterfactual Sample debiasing (CS-PAF). Specifically, CS-PAF consists of two core ingredients: (i) a counterfactual sample generation module for model debiasing; and (ii) a parallel attention fusion network that enhances sufficiency in multi-modal data interaction. Notably, in contrast to other debiasing methods, our counterfactual sample generation applies contrastive learning to circumvent the high cost of manual annotations and ensure seamless integration with other models. Extensive comparisons with state-of-the-art approaches, along with comprehensive ablation and transferability studies across multiple datasets, substantiate the superiority and effectiveness of our CS-PAF. Our implement code is available at https://github.com/chenyulu2000/cspaf. • We propose a visual dialog model termed CS-PAF. • CS-PAF enhances fusion balance via counterfactual sample generation. • CS-PAF enhances fusion sufficiency by parallelly stacking fusion units. • Extensive experiments demonstrate the superiority of CS-PAF over other methods. [ABSTRACT FROM AUTHOR]

Multi-modal action recognition methods can be generally classified into two categories: (1) fusing multi-modal features with simple concatenation or fusing the classification scores of individual modalities without considering the interaction among the multi-modalities; (2) using one of the modalities as privileged information in training to boost the recognition on the other modalities in inference. The former approach usually is not able to deal with the cases where one of the modalities is missing. In the latter, the trained classifier does not work on the privileged modality. To address these shortcomings, this paper presents a novel end-to-end trainable deep fusion network (DFN) that is able to improve the performance not only in the cases where all modalities are available and also in the cases where there is a missing modality. The DFN is simple yet effective with the capability of retrieving an estimation of one modality by using another modality through a Multilayer Perceptron (MLP). In order to better preserve structure information, the DFN first maps the individual modality features to a high dimensional Kronecker-product space and subsequently learns a low-dimensional discriminative space for classification. The effectiveness of the proposed DFN has been verified on three benchmark datasets: the large NTU RGB+D, UTD-MHAD, and SYSU-3D datasets and it has achieved state-of-the-art results. • End-to-end trainable deep fusion network (DFN) for action recognition. • DFN outperforms the commonly used fusion methods. • DFN performs better than single modality cases when one modality is missing. • Competitive performance achieved by DFN on three commonly used datasets. [ABSTRACT FROM AUTHOR]

• Multi-modal bioelectrical signal fusion is reviewed. • Rationality and challenge of the acquisition equipment of EEG signal is analyzed. • The feasibility and challenges of the stimulus paradigm are analyzed. • Advantages, challenges and possible solutions of multi-modal fusion are summarized. • Fusion methods, evaluation index, and fusion application are reviewed. Multi-modal fusion combines multiple modal information to overcome the limitation of incomplete information expressed by a single modality, so as to realize the complementarity of modal information and enhance feature representation. Multi-modal medical signal fusion algorithm and extraction equipment play an important role in improving the recognition accuracy of brain diseases. This paper compared the existing data fusion methods and explored the fusion research of multi-modal bioelectrical signals, including: (1) the challenges and shortcomings in the signal acquisition phase are explored from the biological signal acquisition equipment and scene settings; (2) five multi-modal fusion forms are analyzed; (3) the fusion methods and evaluation indexes are briefly reviewed; (4) the research status and challenges of multi-modal fusion in the field of spatial cognitive impairment and biometrics are explored; (5) the advantages and challenges of multi-modal fusion are described. The conclusion of this review is that the research of multimodal medical signal fusion is in the initial stage, and some studies have proved that multi-modal fusion is meaningful for medical research. However, the fusion algorithm and fusion strategy need to be improved. While learning the relatively perfect image fusion algorithm, we need to develop the fusion algorithm and fusion strategy that is suitable for medical signal and strengthen its feasibility in clinical application. [ABSTRACT FROM AUTHOR]

Dense video captioning aims to automatically describe several events that occur in a given video, which most state-of-the-art models accomplish by locating and describing multiple events in an untrimmed video. Despite much progress in this area, most current approaches only encode visual features in the event location phase and they neglect the relationships between events, which may degrade the consistency of the description in the identical video. Thus, in the present study, we attempted to exploit visual–audio cues to generate event proposals and enhance event-level representations by capturing their temporal and semantic relationships. Furthermore, to compensate for the major limitation of not fully utilizing multimodal information in the description process, we developed an attention-gating mechanism that dynamically fuses and regulates the multi-modal information. In summary, we propose an event-centric multi-modal fusion approach for dense video captioning (EMVC) to capture the relationships between events and effectively fuse multi-modal information. We conducted comprehensive experiments to evaluate the performance of EMVC based on the benchmark ActivityNet Caption and YouCook2 data sets. The experimental results showed that our model achieved impressive performance compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• Gait, the 6th vital sign, needs multi-modal methodologies for robust measurement. • Contemporary wearables enable multi-modal patient assessment. • A novel gait fusion approach is presented for use in multiple environments. • This can inform holistic gait assessment beyond the clinic during free-living. Gait abnormalities are typically derived from neurological conditions or orthopaedic problems and can cause severe consequences such as limited mobility and falls. Gait analysis plays a crucial role in monitoring gait abnormalities and discovering underlying deficits can help develop rehabilitation programs. Contemporary gait analysis requires a multi-modal gait analysis approach where spatio-temporal, kinematic and muscle activation gait characteristics are investigated. Additionally, protocols for gait analysis are going beyond labs/clinics to provide more habitual insights, uncovering underlying reasons for limited mobility and falls during daily activities. Wearables are the most prominent technology that are reliable and allow multi-modal gait analysis beyond the labs/clinics for extended periods. There are established wearable-based algorithms for extracting informative gait characteristics and interpretation. This paper proposes a multi-layer fusion framework with sensor, data and gait characteristics. The wearable sensors consist of four units (inertial and electromyography, EMG) attached to both legs (shanks and thighs) and surface electrodes placed on four muscle groups. Inertial and EMG data are interpreted by numerous validated algorithms to extract gait characteristics in different environments. This paper also includes a pilot study to test the proposed fusion approach in a small cohort of stroke survivors. Experimental results in various terrains show healthy participants experienced the highest pace and variability along with slightly increased knee flexion angles (≈1°) and decreased overall muscle activation level during outdoor walking compared to indoor, incline walking activities. Stroke survivors experienced slightly increased pace, asymmetry, and knee flexion angles (≈4°) during outdoor walking compared to indoor. A multi-modal approach through a sensor, data and gait characteristic fusion presents a more holistic gait assessment process to identify changes in different testing environments. The utilisation of the fusion approach presented here warrants further investigation in those with neurological conditions, which could significantly contribute to the current understanding of impaired gait. [ABSTRACT FROM AUTHOR]

Multi-modal fusion has been proved to help enhance the performance of scene classification tasks. This paper presents a 2D–3D Fusion stage that combines 3D Geometric Features with 2D Texture Features obtained by 2D Convolutional Neural Networks. To get a robust 3D Geometric embedding, a network that uses two novel layers is proposed. The first layer, Multi-Neighbourhood Graph Convolution, aims to learn a more robust geometric descriptor of the scene combining two different neighbourhoods: one in the Euclidean space and the other in the Feature space. The second proposed layer, Nearest Voxel Pooling, improves the performance of the well-known Voxel Pooling. Experimental results, using NYU-Depth-V2 and SUN RGB-D datasets, show that the proposed method outperforms the current state-of-the-art in RGB-D indoor scene classification task. • Multi-Neighbourhood Graph Neural Network to extract enriched 3D Geometric features. • Nearest Voxel Pooling that improves the performance of the well-known Voxel Pooling. • 2D–3D multi-modal fusion stage that allows to exploit the benefits of 2D and 3D Networks simultaneously. • A novel 3D random crop that generates random crops of a point cloud. [ABSTRACT FROM AUTHOR]

RGBT tracking is a challenging task that requires robust fusion of visible images (RGB) and thermal infrared images (TIR) to handle various scenarios, such as illumination changes, occlusions, and camouflage. The current popular RGBT trackers mostly based on two stream Siamese trackers. They tend to separately extracts template and search images region features, which neglects relationship between target and background. Moreover, they do not fuse RGB and TIR features properly, limiting their ability to utilize complementary features. To address these issues, we introduce a highly compact adaptive transformer-based network that unifies the process of feature extracting and correlation between template and search images. In this way, the compact network has dual-branches. They can combine feature extraction and correlation for different modalities of template images and search images. Meanwhile, we introduce a cross-modal weight redistribution module (CMWR) for multi-modal fusion. This adaptive fusion scheme learns discriminative features of RGB and TIR data and assigns weights to them, enabling them to complement each other. Furthermore, to address the issue of tracking targets of different scales, we design scale-adaptive optimization pyramid module (SAOP) that adapt to objects of different sizes. Our method achieves exceptional performance on the GTOT, RGBT234 and LasHeR datasets, surpassing most of the existing methods. The results are consistent across multiple datasets, demonstrating the effectiveness and superiority of our approach. And our code is released at: https://github.com/ELOESZHANG/HCANet. • We propose highly compact adaptive network that employs the vision transformer for RGBT tracking. • We design a cross-modal weight redistribution module (CMWR) that achieves multimodal fusion with an adaptive optimization strategy. • We introduce scale-adaptive optimization pyramid module (SAOP) to handle target scale variations by using feature maps of different sizes. [ABSTRACT FROM AUTHOR]

Camera-based Semantic scene completion (SSC) aims to infer the 3D volumetric occupancy and semantic categories of a scene simultaneously from a single RGB image. The main challenge of camera-based SSC is the lack of geometry information compared with RGB-D SSC. Although the estimated depth from RGB image will help SSC to some extent, the depth prediction quality is far from the demand of SSC. To solve this problem, we propose a NAS-based multi-modal fusion method to incorporate the semantic and geometry information from other intermediate representations (predicted depth and predicted 2D segmentation) to form a more robust 2D feature representation. A key idea of this design is that explicit 2D semantic information could alleviate the misleading information of 3D distortions introduced by estimated depth. Specifically, we propose the Confidence-Block to automatically learn an optimal architecture for routing and obtaining the depth prediction confidence. We propose the two-level fusion search space by decomposing the fusion search space into fusion stage search space and fusion operation search space. Moreover, we propose a confidence-aware 2D–3D projection module to alleviate the 3D projection error. Extensive experiments show that our method outperforms the state-of-the-art method by a large margin using a single RGB image on NYU and NYUCAD datasets. [ABSTRACT FROM AUTHOR]

In the treatment of non-small cell lung cancer (NSCLC), radiotherapy is pivotal. However, manual segmentation of the gross target volume (GTV) is both time-consuming and prone to errors. This study presents an automated solution using deep learning-based multi-modal fusion for efficient GTV segmentation. The method integrates data from CT, and PET images, enhancing GTV segmentation precision. Key features include intensity normalization and random cropping, which ensure data integration and variety. The system's core is the 3D U-Net, extracting detailed features from CT images, and the DeepLab network managing multi-modal fusion. The 3D Unet-DeepLab model demonstrates superior performance with a Dice Similarity Coefficient (DSC) of 0.7502, outperforming the 3D ResSE-Unet. Additionally, the 3D Unet_4B model surpasses existing techniques, with a DSC of 0.7502 and a Hausdorff Distance 95 mm (HD95) of 19.78 mm. These results confirm the method's capability in improving GTV segmentation precision and spatial consistency. Automating segmentation through this innovative method not only refines tumor targeting but also minimizes potential damage to healthy tissues, offering implications for personalized radiotherapy planning and enhanced NSCLC treatment outcomes. [ABSTRACT FROM AUTHOR]

3D object detection, whose task is to perceive the surrounding environment, plays a significant role in autonomous driving. In this study, we propose a new BEV-FePNet 3D detection model, which can effectively fuse multi-modal information in deeply abstract features. The BEV-FePNet has been validated experimentally on the nuScenes dataset, and the findings demonstrate that the proposed approach enhances the performance of the detector and obtains 71.6 % mAP detection performance. In addition, with the rapid development of the autonomous driving market, collecting a large amount of data for autonomous driving has become one of the important means to enhance 3D detecting models' efficiency. However, local national data security policies have to be considered when autonomous driving manufacturers collect data in different countries, so it is difficult to transmit data abroad. To solve this problem, the DP-DeceFL framework has been proposed in this paper that utilizes differential privacy processing to enable information exchange between different countries without revealing sensitive information. Through the verification of nuScenes data, our proposed framework is superior to some selected federated learning frameworks. [ABSTRACT FROM AUTHOR]

Current LiDAR-only 3D detection methods inevitably suffer from the sparsity of point clouds and insufficient semantic information. To alleviate this difficulty, recent proposals densify LiDAR points by depth completion and then perform feature fusion with image pixels at the data-level or result-level. However, these methods often suffer from poor fusion effects and insufficient use of image information for voxel feature-level fusion. Meanwhile, noises brought by inaccurate depth completion significantly degrade detection accuracy. In this paper, we propose PVConvNet, a unified framework for multi-modal feature fusion that cleverly combines LiDAR points, virtual points and image pixels. Firstly, we develop an efficient Pixel-Voxel Sparse Convolution (PVConv) to perform voxel-wise feature-level fusion of point clouds and images. Secondly, we design a Noise-Resistant Dilated Sparse Convolution (NRDConv) to encode the voxel features of virtual points, which effectively reduces the impact of noise. Finally, we propose a unified RoI pooling strategy, namely Multimodal Voxel-RoI Pooling, for improving proposal refinement accuracy. We evaluate PVConvNet on the widely used KITTI dataset and the more challenging nuScenes dataset. Experimental results show that our method outperforms state-of-the-art multi-modal based methods, achieving a moderate 3D AP of 86.92% on the KITTI test set. • A unified multi-modal fusion framework is proposed to combine image, virtual points and points to achieve more accurate 3D detection. • A pixel-voxel sparse convolution is designed to perform feature-level fusion of point clouds and images, effectively realizing voxel-based backbone fusion. • A noise-resistant dilated submanifold convolution is constructed to encode voxel features of virtual points, which significantly reduces the impact of noise. • The object occlusion rate is used in cross-modal GT-Sampling to balance near-distant virtual objects while ensuring the consistency of point clouds and images. • Extensive experiments on the popular KITTI and nuScenes datasets validate the superiority of PVConvNet, with competitive performance compared to state-of-the-art 3D detection methods. [ABSTRACT FROM AUTHOR]

Various physiological signals can reflect the human's emotional states objectively. How to take advantage of the common as well as complementary properties of different physiological signals in representing the emotional states is an interesting problem. Although various models have been constructed to fuse multimodal physiological signals for emotion recognition, the possible incongruity existing among different physiological signals in representing the emotional states and the redundancy resulted from the fusion, which may affect the performance of the fusion schemes seriously, were seldom considered. To this end, a fusion model, which can eliminate the incongruity among different physiological signals and reduce the information redundancy to some extent, is proposed. First, one physiological signal is chosen as the primary modality due to its prominent performance in emotion recognition, and the remaining physiological signals are viewed as the auxiliary modalities. Secondly, the Cross Modal Transformer (CMT) is adopted to optimize the features of the auxiliary modalities by eliminating the incongruity among them, and then Low Rank Fusion (LRF) is performed to eliminate information redundancy caused by fusion. Thirdly, the modified CMT (MCMT) is constructed to enhance the feature of the primary modality by that of each optimized auxiliary modality feature. Fourthly, Self-Attention Transformer (SAT) is performed on the concatenation result of all the enhanced primary modality features to take full advantage of the common as well complementary properties among them in representing the emotional states. Finally, the enhanced primary modality feature and the optimized auxiliary features are fused by concatenation for emotion recognition. Extensive experimental results on DEAP and WESAD datasets demonstrate that (i) The incongruity does exist among different physiological signals, and the CMT-based auxiliary modality feature optimization strategy can eliminate the incongruity prominently; (ii) The emotion prediction accuracy of the primary modality can be enhanced by the auxiliary modality; (iii) All the key modules in the proposed model, CMT, LRF, and MCMT, contribute to the performance enhancement of the proposed model; iv) The proposed model outperforms State-Of-The-Art (SOTA) models in emotion recognition task. • Multimodal physiological signals are fused for emotion recognition. • Incongruity among auxiliary modalities is reduced by Cross Modal Transformer. • Primary modality is enhanced by auxiliary modality through Modified Cross Modal Transformer. • The proposed model works in various physiological signal classification tasks. [ABSTRACT FROM AUTHOR]

Continuous affect recognition has a huge potential in human computer interaction applications. How to efficiently fuse speech and facial information for inferring the affective state of a person from data captured in real-world conditions is a very important issue for continuous affect recognition. Currently, late fusion is usually used in multi-modal continuous affect recognition to improve system performance. However, late fusion ignores the complementarity and redundancy between multiple streams from the different modalities. In this work, we propose an efficient model-level fusion approach for audiovisual continuous affect recognition. First, we propose a LSTM based model-level fusion approach for audiovisual continuous affect recognition. Our approach considers the complementarity and redundancy between multiple streams from different modalities. In addition, our model can efficiently incorporate side information such as gender using adaptive weight network. At last, we design a deep supervision based effective optimization strategy for training the proposed audiovisual continuous affect recognition model. We demonstrate the effectiveness of our approach on the RECOLA dataset. Our experimental results show that the proposed adaptive weight network improves the performance compared to a plain neural network without adaptive weights. Our approach obtains remarkable improvements on both arousal and valence in terms of concordance correlation coefficient (CCC) compared to state-of-the-art early fusion and model-level fusion approaches. Therefore, we believe that our proposed approach gives a promising direction for further improving the performance of audiovisual continuous affect recognition. [ABSTRACT FROM AUTHOR]

Multi-view semi-supervised classification is a typical task to classify data using a small amount of supervised information, which has attracted a lot of attention from researchers in recent years. In practice, existing methods tend to focus on extracting spatial or spectral features using graph neural networks without considering the diversity and variability of graph structures and the contributions of different views. To address this challenge, a framework termed graph attention fusion network is proposed, which consists of two phases: view-specific feature embedding and graph embedding fusion. In the former feature extraction stage, the view-specific feature embedding module can flexibly focus on the neighborhood calculation operation to learn a weight for each neighboring node. In the latter feature fusion stage, the graph embedding fusion module is performed by complementarity and consistency to fuse these embeddings for semi-supervised classification tasks. We carry out comprehensive experiments in semi-supervised classification on real-world datasets to substantiate the effectiveness of the proposed approach compared to several existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Automatic segmentation of medical images plays an important role in the diagnosis of diseases. On single-modal data, convolutional neural networks have demonstrated satisfactory performance. However, multi-modal data encompasses a greater amount of information rather than single-modal data. Multi-modal data can be effectively used to improve the segmentation accuracy of regions of interest by analyzing both spatial and temporal information. In this study, we propose a dual-path segmentation model for multi-modal medical images, named TranSiam. Taking into account that there is a significant diversity between the different modalities, TranSiam employs two parallel CNNs to extract the features which are specific to each of the modalities. In our method, two parallel CNNs extract detailed and local information in the low-level layer, and the Transformer layer extracts global information in the high-level layer. Finally, we fuse the features of different modalities via a locality-aware aggregation block (LAA block) to establish the association between different modal features. The LAA block is used to locate the region of interest and suppress the influence of invalid regions on multi-modal feature fusion. TranSiam uses LAA blocks at each layer of the encoder in order to fully fuse multi-modal information at different scales. Extensive experiments on several multi-modal datasets have shown that TranSiam achieves satisfying results. • A new multi-modal fusion model, TranSiam, for medical image segmentation. • Aggregating multi-modal information using local attention mechanisms. • Combining CNN and Transformer to segment lesion regions. [ABSTRACT FROM AUTHOR]

Driving behavior classification plays an important role in many fields, such as Advanced Driving Assistance System (ADAS), traffic safety, and energy saving. In this paper, we propose a Global–local Multimodal Fusion Driving Behavior Classification Network (GLMDriveNet) which classifies driver behaviors into normal driving, aggressive driving, and drowsy driving. First of all, we design a Global–local Interaction Channel Attention Module (GLI-CAM) to extract effective features in both the roadside image and the spectrogram generated from the current prediction time and its previous four seconds of vehicle speeds. Furthermore, a learnable positional embedding is introduced to fuse the global and local information of the channels for better screening of the extracted features. Secondly, we propose a Multi-scale Feature Representation Fusion Module (MS-FRFM) to associate the high-scale and low-scale information of images and spectrograms and assign different importances for different modal information, making the network more inclined to useful modal information. Our model is evaluated on a public dataset UAH-DriveSet and achieves the best performance (98.4% F1-score on all roads, 97.4% F1-score on the motorway road, and 99.8% F1-score on the secondary road) compared to other state-of-the-art methods. Our model has a very fast speed (142 FPS) and strong generalization which has been verified through extensive experiments on multiple datasets. The code is available on https://github.com/liuwenzhuo1/GLMDrivenet. [ABSTRACT FROM AUTHOR]