Search

Currently, knowledge representation learning has been effectively utilized in knowledge graph completion tasks to predict missing entities or relationships within knowledge graphs, achieving considerable success. However, the structural features of triples after data construction are the only considerations for most traditional completion methods based on representation learning, the results in the rich textual, image, and other multimodal information in the original data not being effectively utilized. To address this problem, it is proposed that the energy function of triples should be defined as the sum of sub-energy functions, which are characterized by the textual features of relationships and the multimodal features of entities, to fully exploit the semantic information in the triple. In addition, a multimodal negative sampling strategy has been devised for diverse energy functions, aiming to align the disparate features of entities, enhance the quality of knowledge representation, and boost the performance of knowledge graph completion. Link prediction experiments were conducted on publicly available datasets DB15K and FB15K-237. The results show that the proposed model exhibits remarkable capabilities in knowledge graph completion. [ABSTRACT FROM AUTHOR]

Due to the limitations of single modality emotion recognition, many researchers have shifted their focus to the field of multimodal emotion recognition. Multimodal emotion recognition focuses on two problems: The optimal extraction of the features of each mode and the effective fusion of the extracted features. This study proposes an emotion recognition method based on multimodal cross interaction to capture the diversity of modality expressions. The editors of various modalities separately extract features with emotional information, and the stacked inter- action modules based on the attention mechanism between modalities model the potential relationship among vision, text and audio. Experiments are conducted on CMU MOSI and CMU MOSEI datasets for emotion recognition based on text, audio and visual. The results show that the method achieved the scores of 86.5%, 47.7%, 86.4%, 0.718, 0.776, and 83.4%, 51.5%, 83.4%, 0.566, 0.737 on five indicators, Acc2(Accuracy2)、Acc7 (Accuracy7) F1、MAE (Mean Absolute Error) and Corr(Correlation). This demonstrates that the proposed algorithm significantly improves performance, and also validates that the cross mapping mutual representation mechanism perform better than single - modal representation methods. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The article discusses the emerging trends and application prospects of current large models, using Tencent's Hunyuan large model as an example. It focuses mainly on innovations and implementations of large models in China. Companies like Google, Meta, and OpenAI have launched powerful models such as Google's Gemini and Meta's Llama 3, which have made significant progress in multi-modal applications and reasoning capabilities. China's large models have significantly improved performance and efficiency by adopting the MoE (Mixture of Experts) architecture. Specifically, with its self-developed MoE trillion-parameter large model and deep learning framework, Tencent has made breakthrough advancements in large model technology and achieved exceptional performance in muti-modal applications. Moreover, Tencent has launched a one-stop AI agent creation and distribution platform. Tencent understands that industry-wide, large models are key to implementing AI and strategies, and is actively supporting the application of large models across sectors such as retail, education, finance, healthcare, media, transportation, and government, helping these industries enhance quality and efficiency. [ABSTRACT FROM AUTHOR]

Multimodal sentiment analysis has become a hot research direction in affective computing by extending unimodal analysis to multimodal environments with information fusion. Word-level representation fusion is a key technique for modeling cross-modal interactions by capturing interplay between different modal elements. And word-level representation fusion faces two main challenges: local interactions between modal elements and global interactions along the temporal dimension. Existing methods often adopt attention mechanisms to model correlations between overall features across modalities when modeling local interactions, while ignoring interactions between adjacent elements and local features, and are computationally expensive. To address these issues, a multi-channel temporal convolution fusion (MCTCF) model is proposed, which uses 2D convolutions to obtain local interactions between modal elements. Specifically, local connections can capture associations between neighboring elements, multi-channel convolutions learn to fuse local features across modalities, and weight sharing greatly reduces computations. On the locally fused sequences, temporal LSTM networks further model global correlations along the temporal dimension. Extensive experiments on MOSI and MOSEI datasets demonstrate the efficacy and efficiency of MCTCF. Using just one convolution kernel (three channels, 28 weight parameters), it achieves state-of-the-art or competitive results on many metrics. Ablation studies confirm that both local convolution fusion and global temporal modeling are crucial for the superior performance. In summary, this paper enhances word-level representation fusion through feature interactions, and reduces computational complexity.

Achieving robust joint utilization of multidomain characteristics and deep-network features while maintaining a high jamming-recognition accuracy with limited samples is challenging. To address this issue, this paper proposes a multidomain characteristic-guided multimodal contrastive recognition method for active radar jamming. This method involves first thoroughly extracting the multidomain characteristics of active jamming and then designing an optimization unit to automatically select effective characteristics and generate a text modality imbued with implicit expert knowledge. The text modality and involved time-frequency transformation image are separately fed into text and image encoders to construct multimodal-feature pairs and map them to a high-dimensional space for modal alignment. The text features are used as anchors and a guide to time-frequency image features for aggregation around the anchors through contrastive learning, optimizing the image encoder’s representation capability, achieving tight intraclass and separated interclass distributions of active jamming. Experiments show that compared to existing methods, which involve directly combining multidomain characteristics and deep-network features, the proposed guided-joint method can achieve differential feature processing, thereby enhancing the discriminative and generalization capabilities of recognition features. Moreover, under extremely small-sample conditions (2～3 training samples for each type of jamming), the accuracy of our method is 9.84% higher than those of comparative methods, proving the effectiveness and robustness of the proposed method.

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The research of music emotion recognition has broad application prospects in the fields of music intelligent recommendation and music visualization. Aiming at the problem that only using low‑level audio features for emotion recognition has limited effectiveness and poor interpretability. Firstly, an emotion recognition model ERMSLM based on MIDI (musical instrument digital interface) data is constructed, which can learn the semantic information of notes. The features of this model are composed of melodic features extracted with skip‑gram and LSTM (long short‑term memory), tonal features extracted by pre‑trained MLP and manually constructed features. Secondly, an emotion recognition model ERMBT based on text data that integrates lyrics and social tags is constructed. The lyrics features are composed of emotional features extracted with BERT, emotional dictionary features constructed by using ANEW lists and TF-IDF features of lyrics. Finally, two multimodal fusion models of feature‑level fusion and decision‑level fusion are constructed based on MIDI and text data. The experimental results show that the ERMSLM and ERMBT models can achieve accuracies of 56. 93% and 72. 62% respectively. And the decision‑level multimodal fusion model is more effective. [ABSTRACT FROM AUTHOR]

Copyright of Computer Measurement & Control is the property of Magazine Agency of Computer Measurement & Control and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Journal of Technology & Chinese Language Teaching is the property of Middle Tennessee State University and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Petroleum Geology & Oilfield Development in Daqing is the property of Editorial Department of Petroleum Geology & Oilfield Development in Daqing and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Piezoelectrics & Acoustooptics is the property of Piezoelectric & Acoustooptic and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Journal of Hubei Open Vocational College is the property of Journal of Hubei Open Vocational College Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Aiming at the problems of unbalanced inter-modal optimization and inadequate fusion of multimodal features in multimodal sentiment analysis, a multimodal sentiment analysis model combining dynamic gradient mechanism and multi-view co-attention mechanism (DG-MCM) is proposed, which can effectively mine single-modal representation and fully integrate multimodal information. Firstly, the model uses pre-trained model BERT (bidirectional encoder representation from transformers) and stacked long short-term memory (SLSTM) to learn the features of text, audio and video, and proposes a dynamic gradient mechanism. By monitoring the contribution difference and learning speed of each mode, the feature learning of each mode is assisted. Secondly, the features of different modes obtained are fused using the multi-view co-attention mechanism. By projecting every two modes into multiple spaces for interaction, more adequate fusion features are obtained. Finally, fusion features and single-modal features are spliced together for sentiment prediction. Experimental results on CMU-MOSI and CMU-MOSEI datasets show that this model can fully learn information between single mode and different modes, and effectively improve the accuracy of multimodal sentiment analysis.

The construction of intelligent fully mechanized mining faces in China’s coal mines is in its early stages, and the automation technology of hydraulic support electro-hydraulic control has been widely applied. However, at present, the automation control logic of a single solidification of hydraulic supports is difficult to adapt to complex, ever-changing, and dynamic production scenarios. The actual control process in mine face still mostly adopts a collaborative approach of automation and manual intervention. In response to the demand for human-machine interaction and cooperation tasks in the complex scenarios of fully mechanized mining faces, a theoretical method and technical principle of a multimodal human-machine collaborative control system for hydraulic supports following the shearer in the middle range mining face (hereinafter referred to as hydraulic supports following) was proposed. Firstly, four human-machine collaborative modes, namely manual, division of labor, approval, and veto, were designed. Based on the factors such as coal seam geology, gas and dust, shearer speed, hydraulic support intelligence level, system status, job technical level, and task load, an AND-OR graph model for hydraulic support human-machine collaborative mode selection was constructed, which achieved a modal selection with manual or machine preferences. Then, a human-machine collaborative control decision-making mechanism for hydraulic supports following was designed. On this basis, an AI inference technology for the secondary control strategy of hydraulic supports following was proposed. Specifically, using on-site data to learn manual operation experience, a decision tree classification model for whether the hydraulic support should be secondary controlled and a Bayesian regression model for estimating the secondary control time of pulling hydraulic support were constructed. Based on the above model, a human-machine collaborative control decision-making program for the hydraulic support following was developed, which achieved human-machine collaborative control of hydraulic supports following based on AND/OR inference for modal selection and AI inference for secondary control strategy. Finally, using cloud edge end architecture software and hardware technology, a multimodal human-machine collaborative control system for hydraulic supports following was developed, which achieved the control functions such as model evolution, operational reasoning, and program execution. The system has undergone industrial trial operation on the 3404 mining face of Shaqu No.2 Mine. The result shows the efficiency of the hydraulic support following has increased by an average of 2% compared to that before. This paper forms an efficient and safe human-machine interactive decision-making mechanism for the comprehensive mining equipment group, which will provide practical theoretical methods and feasible technical paths for the development of intelligent comprehensive mining working faces.

With the rapid advancement of three-dimensional visual perception devices such as 3D scanners and LiDAR, the field of 3D model recognition is gradually gaining the attention of a growing number of researchers. This domain encompasses two core tasks: 3D model classification and retrieval. Since deep learning has already achieved significant success in two-dimensional visual tasks, its introduction into the realm of three-dimensional visual perception not only breaks free from the constraints of traditional methods but also makes notable strides in areas such as autonomous driving and intelligent robotics. However, the application of deep learning techniques to 3D model recognition tasks still faces several challenges. In light of this, there is a need for a comprehensive review of the application of deep learning in 3D model recognition. This review begins by discussing commonly used evaluation metrics and public datasets, providing relevant information and sources for each dataset. Subsequently, it delves into representative methods from various angles, including point clouds, views, voxels, and multimodal fusion. It also summarizes recent research development in the field. Through performance comparison on these datasets, the strengths and limitations of each method are analyzed. Finally, based on the merits and demerits of these approaches, the review outlines the challenges currently faced by 3D model recognition tasks and provides an outlook on future trends in this field.

Objective To achieve the classification of Alzheimer's disease (AD) by integrating information that utilize the complementary properties of multimodal data, and to provide references for clinical diagnosis. Methods A total of 872 subjects were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI), with both clinical information, structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) scans. They were divided into the training set (612 subjects) and test set (260 subjects). Based on three unimodal data and four multimodal combinations of different modalities, we constructed the sparse Partial Least Squares-Discriminant Analysis (sPLS-DA) classification models in the training set to achieve the multi-classification. The macro-averaged precision (Macro-P), macro-averaged recall (Macro-R), macro-averaged Fl value (Macro-F1), and accuracy were used to evaluate the model performance, with the optimal combination of modalities obtained explored for their applicability in the test set. Results The classification performance of clinical information (Macro-P = 0.781 8, Macro-R = 0.804 6, Macro-Fl = 0.791 2, Accuracy = 0.796 7) among the unimodal information was better than that of sMRI and fMRI modalities. The optimal number of potential components was 1, and the number of clinical information features was 9. Among the four multimodal combinations, the clinical information + fMRI combination had the strongest classification ability (Macro-P = 0.806 2, Macro-R = 0.800 6, Macro-FI = 0.797 6, Accuracy = 0.813 2), with the optimal number of potential components selected as 1, and the number of features were 5, while the sMRI + fMRI had the worst classification ability (Macro-P = 0.401 7, Macro-R = 0.398 3, Macro-F1 = 0.349 9, Accuracy = 0.565 9). Applying the best modal combination to the test set, the model performance metrics achieved were 0.791 8 for Macro-P, 0.734 5 for Macro-R, 0.758 4 for Macro-F1, and 0.766 4 (0.646 0, 0.846 5) for Accuracy. Conclusion The performance of the sPLS-DA classification model constructed based on each multimodal combination was higher than that of the unimodal, among which the combination of clinical information + fMRI modality had the best performance, which couldgreatly facilitate the formulation of scientific and reasonable clinical diagnosis plans. [ABSTRACT FROM AUTHOR]

Complementary product recommendations can provide complementary matching products for the convenience of users. However, existing work using graph neural networks ignores the multimodal information of products, and the performance of multimodal models is affected when the modal information is missing. In addition, existing multimodal models simply concatenate the modalities, ignoring the connections between the modalities. For these reasons, a complementary product recommendation using graph neural network (CPRUG) model is proposed. The model combines the graph neural network with multimodal information to strengthen the product representation. Then, it uses the graph attention network to deal with the multimodal absence problem, maintain the performance of the model, and improve the robustness of the model. Finally, it uses the common attention mechanism and matrix factorized bilinear pooling method to fuse multimodal features and learn the complementary relationship of products. Experiments are conducted on the Amazon dataset, and the experimental results show that the model outperforms other baseline models. [ABSTRACT FROM AUTHOR]

Existing methods for detecting Android malware only consider a single kind of features, which do not fully describe the features of Android software. In order to solve the above problems, this study presents a multimodal Android malware detection method based on the permissions, byte code probability matrix and function call graph. At the same time, in order to solve the problem of feature representation of function nodes, a new node feature generation method is presented in this study in the generation of function call graph. In order to enrich the semantic information of opcode, a byte probability matrix generation method based on 2-gram is presented. The experiment proves that the method described the characteristics of Android software more comprehensively than other methods, and the detection accuracy rate reached 95.2%. Compared with the existing methods, the accuracy of this method has been improved by 22% on average, effectively improving the detection ability of Android malware. [ABSTRACT FROM AUTHOR]

3D visual understanding aims to intelligently perceive and interpret 3D scenes, achieving a profound understanding and analysis of objects, environment, and dynamic changes. As its core technology, 3D object detection plays an indispensable role. For the problem of low detection accuracy of distant targets and small targets in current 3D detection algorithms, this paper proposed a 3D object detection method called MIFPR, which was oriented towards multimodal interactive fusion and progressive refinement. In the feature extraction stage, this algorithm introduced an adaptive gated information fusion module firstly. Incorporating the geometric features of the point cloud into the image features results in a more discriminative image representation for handling variations in lighting conditions. Subsequently, the proposed voxel centroid-based deformable cross-modal attention module was to drive the fusion of rich semantic features and contextual information from images into the point cloud features. During the proposal refinement stage, this algorithm introduced a progressive attention module. By learning and aggregating features from different stages, it continuously enhanced the model’s ability to extract and model fine-grained features, progressively refining bounding boxes. This gradual refinement of the proposal helps improve the detection accuracy of distant and small objects, thereby enhancing the overall capability of visual scene understanding. The proposed method shows significant improvement in the detection accuracy of small objects like pedestrian and cyclist on the KITTI dataset compared to the state-of-the-art baseline. This confirms the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

In the field of deep learning, solving problems often requires combining multiple modal information for reasoning and decision-making, among which visual and language information are two important modalities in the interaction process. In many application scenarios, processing multi-modal tasks often faces problems such as complex model architecture organization and inefficient training methods. Based on the above problems, this paper reviewed the representative achievements in the field of multimodal image text in the past five years. This paper first started from the mainstream multi-modal tasks and introduced related text and image multi-modal datasets and pre-training targets. Secondly, considering the visual language model based on Transformer and the feature extraction method, this paper analyzed from the perspectives of multi-modal organization architecture and cross-modal fusion methods, and summarized and compared the commonalities and differences of different processing strategies. Then it introduced the lightweight method of the model from data input, structural components and other aspects. Finally, it prospected the future research direction of multimodal methods based on image text. [ABSTRACT FROM AUTHOR]

Pig farming can be greatly promoted by automatic behavior recognition for lactating sows. However, the recognition accuracies have been confined to behaviors with similar visual characteristics. In this study, an audio-video fusion-based model was proposed for the behavior classification of lactating sows in pig farming. A three-branch deep neural network (AVSlowFast) was employed as the backbone. The gaussian context transformer (GCT) attention mechanism was introduced to optimize the model without increasing the number of parameters. The experiment was conducted in Lihua Pig Farm of Changzhou City, Jiangsu Province, China, from August 1, 2023 to September 10, 2023. Ten long white sows were randomly selected as the research objects with significant differences in their litter environment and farrowing houses. All of these sows were within three days postpartum. The camera and sound recorder were used to collect video and audio data in the experiment, respectively. The dataset was constructed from the captured video and audio data. The sow behaviors were then manually labelled into six groups: breastfeeding, eating, drinking, sleeping, fence-hitting, and daily activities. Three models of behavior recognition verified the vision-audio fusion with different feature models. Specifically, MFCC-Vision Transformer was tested with audio features, SlowFast was with vision features, and AVSlowFast was with vision-audio multimodal features. The results showed that the outstandingly higher accuracies of multimodal models (AVSlowFast) were achieved to identify six types of sow behaviors, compared with two single-modal models, Vision Transfomer and Slowfast. Notably, AVSlowFast demonstrated superior performance in the behaviors with similar visual features among lactating sows, such as feeding, drinking, and fence-hitting. Nevertheless, there was a relatively smaller decrease in the recognition accuracy of sleeping behavior with a multimodal approach, compared with the single-vision. The reason was that the distinct audio features of sleep behavior were often lacking in the inclusion of audio information. The attention mechanisms (such as SENet and GCT) were then introduced to improve the recognition performance, especially in sleep behavior. After that, the accuracy of sleeping behavior recognition increased with the improved model. The attention mechanisms effectively adjusted the weight values of feature channels during iterative training, thus mitigating the interference caused by audio signals. GCT-AVSlowFast had achieved an accuracy of 94.3% precision and 94.6% recall, compared with SENet-AVSlowFast. The average F1-score of behavior recognition was significantly improved by 12.7 percentage points, compared with the single-modal (SlowFast). Finally, the superior performance of GCT-AVSlowFast without additional model parameters was suitable for deployment in resource-limited pig farm environments. The finding can also provide an effective approach to implementing multi-modal behavior monitoring in livestock and poultry. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Frontiers of Computer Science & Technology is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Journal of Guangxi Normal University - Natural Science Edition is the property of Gai Kan Bian Wei Hui and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Electric Machines & Control / Dianji Yu Kongzhi Xuebao is the property of Electric Machines & Control and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In response to the dynamic and multimodal characteristics problems in industrial processes, a dynamic weighted difference principal component analysis approach (DWDPCA) was proposed. Firstly, the timing characteristics of the system were described by setting a reasonable time window. Then, the first nearest neighbor and the nearest neighbor set of the first nearest neighbor were searched for the samples within the time window. The weighted difference method was used to process the data to solve the problem of data center drift problem. Finally, a principal component analysis(PCA) model was built for fault detection using the processed data. The data dynamic and center drift issue can be resolved with this method. By employing the method for numerical examples and the Tennessee-Eastman (TE) process for fault detection, the effectiveness of the proposed method was confirmed. [ABSTRACT FROM AUTHOR]

Copyright of Control Theory & Applications / Kongzhi Lilun Yu Yinyong is the property of Editorial Department of Control Theory & Applications and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Jamming recognition is a prerequisite for radar antijamming and actual radar deception jamming recognition; however, there is a problem of insufficient samples. To address this issue, we propose a multimodal radar active deception jamming recognition method based on small samples in this paper. This method is based on two modal information—feature parameters and time-frequency images extracted from radar signals—and utilizes prototype networks to train multimodal features. Furthermore, the model adopts the image denoising method and weighted Euclidean distance to improve the recognition performance at low signal-to-noise ratios. Thus, radar deception jamming recognition can be achieved under small sample conditions. Simulation results reveal that the proposed method achieves an average recognition accuracy of over 97% across 10 types of radar deception jamming when the jamming-to-signal ratio is 3 dB. Moreover, the test results from the simulator data verify the good generalization performance of the proposed method.

Although deep learning methods have achieved better results than traditional trajectory prediction algorithms, there are still problems such as information loss, interaction and uncertainty difficulties in modelling, and lack of interpretability of predictions when implementing multimodal high-precision prediction for autonomous vehicles in heterogeneous, highly dynamic and complex changing environments.The newly developed Transformer's long-range modelling capability and parallel computing ability make it a great success not only in the field of natural language processing, but also in solving the above problems when extended to the task of multimodal trajectory prediction for autonomous driving.Based on this, the aim of this paper is to provide a comprehensive summary and review of past deep neural network-based approaches, in particular the Transformer-based approach.The advantages of Transformer over traditional sequential network, graphical neural network and generative model were also analyzed and classified in relation to existing challenges, simultaneously.Transformer models can be better applied to multimodal trajectory prediction tasks, and that such models have better generalisation and interpretability.Finally, the future directions of multimodal trajectory prediction were presented.

[Purpose/Significance] Argument mining, a research hotspot in the field of computational linguistics, provides machine processable structured data for computational models of argument. Argument mining tasks are closely related to artificial intelligence (AI) technologies, such as natural language processing and knowledge representation. There are numerous systematic studies in academia and a clear technical realization route has come into being. New research results continue to emerge as a result of rich resources and rapid development and iteration of deep learning, large language models (LLMs), and other technologies. This study, which reviews the research status and progress of argument mining, can serve as a resource for future research and application development. [Method/Process] Through literature review, this paper systematically reviews the relevant research basis (including foundational techniques and semantic representation models), summarizes the related technical system in terms of task framework, influencing factors of technological complexity, and method classification, and then introduces the argument mining practice and application cases for specific fields and research objectives and makes a comparative analysis. Most importantly, the overall development and characteristics of this research field are summarized, with a focus on tracking the progress of multimedia argument mining in the context of the new AI environment. [Results/Conclusions] Relevant research has experienced the development of "machine learning - deep learning" and "text only - multimodal", and the levels of development and application of various fields vary much. Future research may focus on how to achieve multigranularity and multimodal content generalization, as well as how to promote its application and implementation in practice. Possible research directions include: 1) the use of LLMs in argument mining, because they exhibit significant benefits in downstream applications such as natural language processing and multimodal learning, and can also provide certain technical conditions for the generation of argument content; 2) the use of domain knowledge organization systems such as vocabulary, knowledge base and knowledge graph: with these systems, researchers can combine domain-specific argument mining models with rich knowledge structure, to strengthen semantic representation and organization improve the systematization and dig deeper into argument mining model research in the domain; 3) promoting the application research and practice of argument mining in more fields or across disciplines, and improving the retrieval and visualization of argument information, such as combining information retrieval methods with argument mining to build the next generation of argument search engines.

Multimodal emotion recognition in conversations（MERC） is a task to predict the emotional category of the discourse in a dialogue based on its textual, audio, and visual modality. Existing studies focus on multimodal feature extraction and fusion of discourse context without fully considering the utilization of emotional features of different speakers. Therefore, this paper proposed a model of multimodal dialogue emotion recognition based on a consistent graph convolutional network. The model first constructed a graph convolutional network of multimodal feature learning and fusion, and obtained the context features of each discourse. On this basis, the average features of the speaker in the complete dialogue as the constraint to make the model learn more reasonable discourse features, so as to improve the performance of predicting emotion class. The paper compared with other baseline models on two benchmark datasets IEMOCAP and MELD. And the results show that the proposed model is superior to the other models. In addition, the paper verifies the consistency constraints and other components of the model through ablation experiments. [ABSTRACT FROM AUTHOR]

Copyright of China Medical Education Technology is the property of China Medical Education Technology Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of China Terminology is the property of China Terminology Magazine and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Experimental Technology & Management is the property of Experimental Technology & Management Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Chinese Journal of Liquid Crystal & Displays is the property of Chinese Journal of Liquid Crystal & Displays and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Chinese Medical Equipment Journal is the property of Chinese Medical Equipment Journal Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Chinese Journal of Intelligent Science & Technology is the property of Beijing Xintong Media Co., Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Automotive Digest is the property of Automotive Digest Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In recent years, image caption generation, as a multimodal task in the field of artificial intelligence, integrates the related research of computer vision and natural language processing, and can realize the modal conversion from image to text. It plays an important role in visual assistance and image understanding, and has attracted extensive attention from researchers. Firstly, this paper describes the task of image caption generation, and introduces three image caption generation methods: template-based method, retrieval-based method and encode-decode method. Their respective method ideas, representative research and advantages and disadvantages are also introduced. Secondly, from the model structure, the research progress of image understanding phase and caption generation phase, this paper expounds in detail the method based on encoding-decoding, and summarizes the research over years into the research of image understanding and caption generation. Image understanding research includes attention mechanism and semantic aspects. The research of caption generation is divided into traditional caption generation, dense caption generation and stylish caption generation. The performance, advantages and disadvantages of the model are summarized, and the datasets and evaluation index of the performance evaluation of the image captioning model are introduced. Finally, the challenges and difficulties in the field of image captioning are pointed out.

In recent years, the generative adversarial network (GAN) has attracted widespread attention with its unique adversarial training mechanism. Its applications have gradually extended to the field of medical imaging, and much excellent research has emerged. This paper reviews the research progress of the popular application for GAN in medical image translation. It starts with an introduction to the basic concepts of GAN and its typical variants, emphasizing on several GANs related to medical image translation. Then, the recent progress is summarized and analyzed from the perspectives of different target tasks and training modes. Finally, the remaining challenges of GAN in medical image translation and the directions of future development are discussed.

The layout of document images is complex and distribution of object sizes is uneven, currently, most of detection methods ignore multimodal information and global dependencies. Therefore, this paper proposed a multi modal document object detection method based on vision and text. Firstly, this method explored the fusion strategy of multimodal features. In order to utilize textual features, it converted text sequence information of the image into two-dimensional representation. After the initial fusion of text features and visual features, it input the fused features to backbone network to extract multiscale features, and repeatedly integrated textual features during the extraction process, so as to realize deep fusion of multimodal features. Next, to ensure the detection accuracy of small and large objects, this paper designed a pyramid network. The lateral connection could concatenate feature maps of the same spatial size from the bottom-up pathway and the top-down pathway in channel, so as to achieve the propagation between high-level semantic information and low-level feature information. The experimental results on large public dataset PubLayNet show that the detection accuracy of this method reaches 95. 86%,and it has a higher accuracy than other methods. This method not only realizes the deep fusion of multi modal features, but also enriches the fused multimodal feature information, and it has good detection performance. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Data Acquisition & Processing / Shu Ju Cai Ji Yu Chu Li is the property of Editorial Department of Journal of Nanjing University of Aeronautics & Astronautics and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Teacher Education Research is the property of Teacher Education Research Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Experimental Technology & Management is the property of Experimental Technology & Management Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Facial expression is an important basis for judging human emotion and human-computer interaction. The development of traditional machine learning and deep learning has brought many opportunities and challenges to facial expression recognition and analysis. First, this paper analyzes the internal relationship and difference between expression recognition and emotion analysis, and points out that expression recognition focuses on identifying facial expression and emotion. Then, it summarizes the advantages and disadvantages of expression recognition tech-nology based on single mode dataset and traditional machine learning method, and introduces the expression recognition technology based on single mode dataset and deep learning method. Then, it points out that the expres-sion recognition technology based on single mode data has certain limitations, such as insufficient quantity and quality of datasets, generally low recognition accuracy, mostly staying in the laboratory research stage. Then, it introduces the expression recognition and inter-modal fusion methods based on multimodal datasets, and introduces the commonly used multimodal expression datasets. The expression recognition technology based on multimodal dataset and the fusion technology between modes are analyzed, including feature level fusion, decision level fusion and hybrid fusion. Finally, the expression recognition analysis technology is summarized and prospected: considering the problem of dataset, more high-quality expression datasets in natural environment can be constructed; multimodal data-sets can also be constructed combined with physiological signals such as posture and EEG; GAN network can be used to enhance data; pay attention to the extraction of micro-expression, and study multimodal fusion algorithm, etc.

Multimodal emotion recognition aims to recognize human emotional states through different modalities related to human emotion expression such as audio, vision, text, etc. This topic is of great importance in the fields of human-computer interaction, a.pngicial intelligence, affective computing, etc., and has attracted much attention. In view of the great success of deep learning methods developed in recent years in various tasks, a variety of deep neural networks have been used to learn high-level emotional feature representations for multimodal emotion recog-nition. In order to systematically summarize the research advance of deep learning methods in the field of multi-modal emotion recognition, this paper aims to present comprehensive analysis and summarization on recent multi-modal emotion recognition literatures based on deep learning. First, the general framework of multimodal emotion recognition is given, and the commonly used multimodal emotional dataset is introduced. Then, the principle of representative deep learning techniques and its advance in recent years are briefly reviewed. Subsequently, this paper focuses on the advance of two key steps in multimodal emotion recognition: emotional feature extraction methods related to audio, vision, text, etc., including hand-crafted feature extraction and deep feature extraction; multi-modal information fusion strategies integrating different modalities. Finally, the challenges and opportunities in this field are analyzed, and the future development direction is pointed out.

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Objective To analyze the diagnostic value of multimodal MRI in H3 K27M mutant diffuse midline glioma (DMG). Methods The clinical and imaging data of 14 patients with H3 K27M mutant DMG confirmed by pathology were collected. The characteristics of MRI were analyzed and summarized. Results Among the 14 cases, 9 were located intracranially, with long T1 and slightly long T2 signals, 8 showed cystic change and 2 showed bleeding in the lesions. DWI showed limited diffusion in the solid portion. The other 5 cases occurred in the spinal cord and showed long T1 and slightly long T2 signals. All the cases showed inhomogeneous enhancement on enhancement scan. Conclusion The MRI finding of H3 K27M mutant DMG has certain characteristics, which are of value for its accurate preoperative diagnosis. [ABSTRACT FROM AUTHOR]