Search

Background: Accurately classifying primary bone tumors is crucial for guiding therapeutic decisions. The National Comprehensive Cancer Network guidelines recommend multimodal images to provide different perspectives for the comprehensive evaluation of primary bone tumors. However, in clinical practice, most patients' medical multimodal images are often incomplete. This study aimed to build a deep learning model using patients' incomplete multimodal images from X-ray, CT, and MRI alongside clinical characteristics to classify primary bone tumors as benign, intermediate, or malignant., Methods: In this retrospective study, a total of 1305 patients with histopathologically confirmed primary bone tumors (internal dataset, n = 1043; external dataset, n = 262) were included from two centers between January 2010 and December 2022. We proposed a Primary Bone Tumor Classification Transformer Network (PBTC-TransNet) fusion model to classify primary bone tumors. Areas under the receiver operating characteristic curve (AUC), accuracy, sensitivity, and specificity were calculated to evaluate the model's classification performance., Results: The PBTC-TransNet fusion model achieved satisfactory micro-average AUCs of 0.847 (95% CI: 0.832, 0.862) and 0.782 (95% CI: 0.749, 0.817) on the internal and external test sets. For the classification of benign, intermediate, and malignant primary bone tumors, the model respectively achieved AUCs of 0.827/0.727, 0.740/0.662, and 0.815/0.745 on the internal/external test sets. Furthermore, across all patient subgroups stratified by the distribution of imaging modalities, the PBTC-TransNet fusion model gained micro-average AUCs ranging from 0.700 to 0.909 and 0.640 to 0.847 on the internal and external test sets, respectively. The model showed the highest micro-average AUC of 0.909, accuracy of 84.3%, micro-average sensitivity of 84.3%, and micro-average specificity of 92.1% in those with only X-rays on the internal test set. On the external test set, the PBTC-TransNet fusion model gained the highest micro-average AUC of 0.847 for patients with X-ray + CT., Conclusions: We successfully developed and externally validated the transformer-based PBTC-Transnet fusion model for the effective classification of primary bone tumors. This model, rooted in incomplete multimodal images and clinical characteristics, effectively mirrors real-life clinical scenarios, thus enhancing its strong clinical practicability., (© 2024. The Author(s).)

Background: To assess the capability of multimodal apparent diffusion (MAD) weighted magnetic resonance imaging (MRI) to distinguish between malignant and benign breast lesions, and to predict Ki-67 expression level in breast cancer., Methods: This retrospective study was conducted with 93 patients who had postoperative pathology-confirmed breast cancer or benign breast lesions. MAD images were acquired using a 3.0 T MRI scanner with 16 b values. The MAD parameters, as flow (f F , D F ), unimpeded (fluid) (f UI ), hindered (f H , D H , and α H ), and restricted (f R , D R ), were calculated. The differences of the parameters were compared by Mann-Whitney U test between the benign/malignant lesions and high/low Ki-67 expression level. The diagnostic performance was assessed by the area under the receiver operating characteristic curve (AUC)., Results: The f R in the malignant lesions was significantly higher than in the benign lesions (P = 0.001), whereas the f UI and D H were found to be significantly lower (P = 0.007 and P < 0.001, respectively). Compared with individual parameter in differentiating malignant from benign breast lesions, the combination parameters of MAD (f R , D H , and f UI ) provided the highest AUC (0.851). Of the 73 malignant lesions, 42 (57.5%) were assessed as Ki-67 low expression and 31 (42.5%) were Ki-67 high expression. The Ki-67 high status showed lower D H , higher D F and higher α H (P < 0.05). The combination parameters of D H , D F , and α H provided the highest AUC (0.691) for evaluating Ki-67 expression level., Conclusions: MAD weighted MRI is a useful method for the breast lesions diagnostics and the preoperative prediction of Ki-67 expression level., (© 2024. The Author(s).)

Microbes have been demonstrated to be closely linked to diseases that pose a major threat to human health. Computing technologies can help researchers find potential microbe-drug associations more quickly and precisely. In this study, we introduced a novel computational prediction model called GLNNMDA based on global and local features of microbes and drugs to infer possible microbe-drug correlations. In GLNNMDA, we first constructed a heterogeneous network based on known microbe-drug relationships by integrating multiple similarity metrics of drugs and microbes. Subsequently, low-dimensional features will be extracted for nodes in the heterogeneous network by adopting the graph attention encoder. Next, based on combining these low-dimensional features with multiple properties of microbes and drugs to form a new comprehensive feature matrix, we would utilize the GLF module to extract the global and local features for microbes and drugs respectively, and then, we would further fuse these global and local features to come up with predictions of possible microbe-drug associations. Moreover, in order to evaluate the prediction performance of GLNNMDA, under the framework of fivefold cross-validation, intensive comparative experiments and case studies were done on different well-known public databases. The results showed that GLNNMDA obtained the highest AUC values as well as AUPR values of 0.9802 ± 0.0011, 0.9773 ± 0.0021 and 0.8586 ± 0.0004, 0.8008 ± 0.0031 in the two databases, MDAD and aBiofilm, respectively, compared to the state-of-the-art competing prediction methods. In addition, case studies of well-known microorganisms and drugs have demonstrated the effectiveness of GLNNMDA in inferring potential microbial drug associations, which implies that GLNNMDA may be a useful tool for microbe-drug association prediction in the future. The source code is available at: "https://github.com/KuangHaiYue/GLNNMDA.git". [ABSTRACT FROM AUTHOR]

Representation learning aims to extract meaningful features from medical images that are often multimodal, i.e., captured using multiple imaging modalities, to provide a more comprehensive understanding of the underlying anatomy and pathology. The primary objective is to improve the quality of features that can help in accurate diagnosis, disease detection, and treatment planning. With more than one modality, the extracted features must be integrated efficiently to capture the complex relationships between them. Integrating features in medical imaging-based classification problems, such as those involving magnetic resonance imaging-based diagnosis, remains a significant challenge. Current approaches to multimodal medical imaging classification often focus on features from a single modality or concatenate them into a high-dimensional feature vector, resulting in overfitting. Intermediate methods like deep fusion that combine features at a mid-level or deeper level have shown promise in offering a better representation. Depthwise convolution is a promising alternative in the deep learning paradigm that has demonstrated strong performance while requiring less computation. This paper presents a novel method in deep fusion based on depthwise 1D convolution. The proposed architecture is compared with other methods based on different fusion techniques and machine learning classifiers. The effectiveness of this method is demonstrated through experiments involving the brain tumor segmentation (BraTs-21) competition, Task-2, which involves predicting O(6)-methylguanine-DNA methyltransferase promoter status, a biomarker for glioblastoma, as a binary classification problem using multimodal magnetic resonance images. The proposed approach involves calculating radiomic features directly from magnetic resonance images and deep features from pre-trained deep learning models, selecting valuable features from both sets and then deep feature fusion using depthwise 1D convolution followed by fully connected layers for classification. In this approach, the deep fusion and classifier part is trainable. To make the proposed framework generalizable, we have trained and tested the model with the public BraTs-21 dataset and externally tested the model on a public Lumiere dataset. The experimental results show the promising performance of this approach. The proposed method got the best AUC score of 0.748 with minimum BCE loss of 0.62 on BraTs-21 as compared to other methods. [ABSTRACT FROM AUTHOR]

Multimodal sentiment analysis faces two challenges: modality representation and modality fusion. Most of the existing models rely only on the feature extraction network to learn modality representation, and the fusion mechanism adopted by some models does not perform well. These factors are not conducive to the model learning rich emotional information and further affect the model's predictive ability. To solve these problems, we propose a multimodal sentiment analysis model based on two-stage contrastive learning and feature hierarchical fusion network (TSCL-FHFN). First, we apply the idea of contrastive learning to unimodal feature representation and multimodal fusion feature representation respectively. By designing a two-stage contrastive learning task, TSCL-FHFN learns similar features for data with the same emotion category and learns distinguishable features for data with different emotion categories. This enables the model to better learn the features of emotional differences. Second, in order to further explore the deep semantic association of multimodal data, we propose a multimodal feature hierarchical fusion network (FHFN). The core idea is to design an attention-based directional cross-modal transformer so that one modality can receive information from the other modality, thereby obtaining complementary information between two modalities. Then, FHFN uses the low-rank tensor fusion method to further learn interactive information between multiple modalities. Finally, we conduct a series of comparative experiments on CMU-MOSI and CMU-MOSEI datasets. Compared with the current representative models, the TSCL-FHFN model achieves better experimental results. In addition, ablation experiments further verify the effectiveness of the improved TSCL-FHFN model. [ABSTRACT FROM AUTHOR]

Medication recommendation aims to provide clinicians with safe medicine combinations for the treatment of patients. Existing medication recommendation models are built based on the temporal structured code data of electronic health records (EHRs) in the medical database; nevertheless, unstructured data in EHRs, such as textual data containing rich information, are underexploited. To fill the gap, a novel multimodal interactive fusion network (MIFNet) is proposed for medication recommendation, which integrates both structured code information and unstructured text information in EHRs. Our model first extracts a series of informative feature representations to encode comprehensive patient health history and control potential drug–drug interactions (DDI), including medical code, clinical notes, and the DDI knowledge graph. Next, a novel cross-modal interaction extraction block is proposed to capture the intricate interaction information between the two modalities. Finally, a multimodal fusion block is adopted to fuse the constructed features and generate a medication combination list. Experiments are conducted on the public MIMIC-III dataset, and the results demonstrate that the proposed model outperforms the state-of-the-art medication recommendation methods on main evaluation metrics. [ABSTRACT FROM AUTHOR]

Abstract Despite the growing recognition of the importance of multimodal input and digital virtual reality (VR) games in enhancing EFL learners’ productive language skills, a significant gap remains in empirical research examining their impact on multimodal output—particularly writing and speaking—within content and language integrated learning (CLIL) science education. This quasi-experimental study addresses this gap by investigating the potential benefits of using VR games to enhance fourth-grade CLIL students’ productive language skills, specifically writing and speaking, through the analysis of their ability to convey scientific concepts in multimodal output. Grounded in self-regulated learning (SRL) theory, the study compares the effects of multimodal input embedded in VR games with those of traditional PowerPoint (PPT)-led games on students’ English poster designs (writing) and oral presentations (speaking), using the 4Cs (Content, Communication, Cognition, and Culture) framework in multimodal assessments. The study involved 81 fourth-grade students from three Taiwanese public elementary schools, divided into an experimental group (EG = 40) using VR-based games and a control group (CG = 41) using PPT-led games for content review. A mixed-methods approach was employed, combining quantitative evaluations with rubrics based on the 4Cs framework and qualitative rater reflections to provide a comprehensive understanding of how different review methods influenced student performance and creative output. Quantitative findings revealed that students using VR review games significantly outperformed those using traditional PPT games in aspects of Content and Cognition for both poster designs and presentations, demonstrating greater depth, accuracy, and application of scientific concepts and higher-order cognitive skills. In terms of Communication, the EG showed higher target vocabulary usage and sentence complexity in presentations, but no significant differences were found in Culture outcomes between the groups or in Communication in posters. Expert raters’ reflections further highlighted that students using VR games exhibited more innovative and integrated use of scientific content, critical thinking, and multimodal expressions, reflecting deeper engagement with the material. This study empirically demonstrates that game-based virtual reality learning environments (VRLEs) significantly enhance students’ multimodal output in content and cognitive skills. Theoretically, it extends the application of SRL in CLIL contexts by highlighting the potential of VRLEs to foster advanced cognitive skills and emphasizes the importance of multimodal assessments in capturing comprehensive student learning outcomes. Future research should explore integrating cultural content into VR environments to enhance students’ cultural awareness and sensitivity.

Few-shot meta-learning involves training a model on multiple tasks to enable it to efficiently adapt to new, previously unseen tasks with only a limited number of samples. However, current meta-learning methods assume that all tasks are closely related and belong to a common domain, whereas in practice, tasks can be highly diverse and originate from multiple domains, resulting in a multimodal task distribution. This poses a challenge for existing methods as they struggle to learn a shared representation that can be easily adapted to all tasks within the distribution. To address this challenge, we propose a meta-learning framework that can handle multimodal task distributions by conditioning the model on the current task, resulting in a faster adaptation. Our proposed method learns to encode each task and generate task embeddings that modulate the model's activations. The resulting modulated model become specialized for the current task and leads to more effective adaptation. Our framework is designed to work in a realistic setting where the mode from which a task is sampled is unknown. Nonetheless, we also explore the possibility of incorporating auxiliary information, such as the task-mode-label, to further enhance the performance of our method if such information is available. We evaluate our proposed framework on various few-shot regression and image classification tasks, demonstrating its superiority over other state-of-the-art meta-learning methods. The results highlight the benefits of learning to embed task-specific information in the model to guide the adaptation when tasks are sampled from a multimodal distribution. [ABSTRACT FROM AUTHOR]

Gait recognition has become an increasingly promising area of research in the search for noninvasive and effective methods of person identification. Its potential applications in security systems and medical diagnosis make it an exciting field with wide-ranging implications. However, precisely recognizing and assessing gait patterns is difficult, particularly in changing situations or from multiple perspectives. In this study, we utilized the widely used CASIA-B dataset to observe the performance of our proposed gait recognition model, with the aim of addressing some of the existing limitations in this field. Fifty individuals are randomly selected from the dataset, and the resulting data are split evenly for training and testing purposes. We begin by excerpting features from gait photos using two well-known deep learning networks, MobileNetV1 and Xception. We then combined these features and reduced their dimensionality via principal component analysis (PCA) to improve the model's performance. We subsequently assessed the model using two distinct classifiers: a random forest and a one against all support vector machine (OaA-SVM). The findings indicate that the OaA-SVM classifier manifests superior performance compared to the others, with a mean accuracy of 98.77% over eleven different viewing angles. This study is conducive to the development of effective gait recognition algorithms that can be applied to heighten people's security and promote their well-being. [ABSTRACT FROM AUTHOR]

A new framework for explainable artificial intelligence in the context of multimodal triage for autism spectrum disorders (ASD) using a fuzzy approach-based multi-criteria decision-making (MCDM) is proposed in this study. The framework consists of five phases. In the first phase, a real ASD dataset of 538 autistic patients is obtained and diagnosed based on 42 medical and sociodemographic criteria. In the second phase, an ASD methodology for triaging the 538 autistic patients into three levels (i.e., minor, moderate, and urgent) is presented using fuzzy approach-based MCDM techniques, namely, fuzzy Delphi method and fuzzy-weighted zero-inconsistency, followed by the processes for triaging autism patients. In the third phase, cost-sensitive learning is employed to balance two ASD datasets: one labeled based on the ASD triage methodology and the other labeled by specialized psychologists. Two multimodal of artificial intelligence are developed in the fourth phase using nine machine-learning algorithms for the balanced ASD datasets. The evaluation of the two multimodal setups is carried out using nine metrics. In the fifth phase, the local interpretable model-agnostic explanations (LIME) model is used to interpret the models using two scenarios. Four new algorithms are presented in the developed framework. [ABSTRACT FROM AUTHOR]

This paper develops a theoretical model of determinants influencing multimodal fake review generation using the theories of signaling, actor-network, motivation, and human–environment interaction hypothesis. Applying survey data from users of China's three leading E-commerce platforms (Taobao, Jingdong, and Pinduoduo), we adopt structural equation modeling, machine learning technique, and Bayesian complex networks analysis to perform factor identification, path analysis, feature factor importance ranking, regime division, and network centrality analysis of full sample, male sample, and female sample to reach the following conclusions: (1) platforms' multimodal recognition and governance capabilities exert significant negative moderating effects on merchants' information behavior, while it shows no apparent moderating effect on users' information behavior; users' emotional venting, perceived value, reward mechanisms, and subjective norms positively influence multimodal fake review generation through perceptual behavior control; (2) feature factors of multimodal fake review generation can be divided into four regimes, i.e., regime 1 includes reward mechanisms and perceived social costs, indicating they are key feature factors of multimodal fake review generation; merchant perception impact is positioned in regime 2, signifying its pivotal role in multimodal fake review generation; regime 3 includes multimodal recognition and governance capabilities, supporting/disparaging merchants, and emotional venting; whereas user perception impact is positioned in regime 4, indicating its weaker influence on multimodal fake review generation; (3) both in full sample, male sample, and female sample, reward mechanisms play a crucial role in multimodal fake review generation; perceived value, hiring review control agency, multimodal recognition and governance capabilities exhibit a high degree of correlation; however, results of network centrality analysis also exhibit heterogeneity between male and female samples, i.e., male sample has different trends in closeness centrality values and betweenness centrality values than female sample. This indicates that determinants influencing multimodal fake review generation are complex and interconnected. [ABSTRACT FROM AUTHOR]

Recently, vision transformer (ViT) based multimodal learning methods have been proposed to improve the robustness of face anti-spoofing (FAS) systems. However, there are still no works to explore the fundamental natures (e.g., modality-aware inputs, suitable multimodal pre-training, and efficient finetuning) in vanilla ViT for multimodal FAS. In this paper, we investigate three key factors (i.e., inputs, pre-training, and finetuning) in ViT for multimodal FAS with RGB, Infrared (IR), and Depth. First, in terms of the ViT inputs, we find that leveraging local feature descriptors (such as histograms of oriented gradients) benefits the ViT on IR modality but not RGB or Depth modalities. Second, in consideration of the task (FAS vs. generic object classification) and modality (multimodal vs. unimodal) gaps, ImageNet pre-trained models might be sub-optimal for the multimodal FAS task. Finally, in observation of the inefficiency on direct finetuning the whole or partial ViT, we design an adaptive multimodal adapter (AMA), which can efficiently aggregate local multimodal features while freezing majority of ViT parameters. To bridge these gaps, we propose the modality-asymmetric masked autoencoder (M 2 A 2 E) for multimodal FAS self-supervised pre-training without costly annotated labels. Compared with the previous modality-symmetric autoencoder, the proposed M 2 A 2 E is able to learn more intrinsic task-aware representation and compatible with modality-agnostic (e.g., unimodal, bimodal, and trimodal) downstream settings. Extensive experiments with both unimodal (RGB, Depth, IR) and multimodal (RGB+Depth, RGB+IR, Depth+IR, RGB+Depth+IR) settings conducted on multimodal FAS benchmarks demonstrate the superior performance of the proposed methods. One highlight is that the proposed method is robust under various missing-modality cases where previous multimodal FAS models suffer serious performance drops. We hope these findings and solutions can facilitate the future research for ViT-based multimodal FAS. [ABSTRACT FROM AUTHOR]

Pepper diseases and pests typically exhibit small target proportions, diverse shapes and sizes, complex imaging backgrounds, and similarities with the background. Existing detection methods perform poorly in identifying targets of different sizes and shapes within the same scene, and they lack adequate noise suppression capabilities. To address the practical needs of detecting pepper diseases and pests in complex scenarios, we have constructed the first multimodal pepper diseases and pests object detection dataset (PDD). This dataset includes a wide variety of diseases and pests images, along with detailed natural language descriptions of their attributes. Locating the described targets in complex scenes with similar disease symptoms and leaf occlusion presents a significant challenge. To tackle this issue, we propose the PepperNet model for object detection in pepper diseases and pests images using natural language descriptions. This model decomposes complex multimodal features of language and images into explicit attribute features and employs fine-grained multimodal attribute contrast learning strategies. This approach effectively distinguishes subtle local differences between similar objects, achieving fine-grained mapping from language to vision in complex scenarios. Our detection results show a mAP@0.5 of 91.93% and a detection speed of 121.8 frames per second. Visualizations indicate that the model maintains high robustness under varying noise levels and occlusion conditions, demonstrating superior performance and stability across diverse complex scenarios. [ABSTRACT FROM AUTHOR]

Conventional Human–Machine Interaction (HMI) interfaces have predominantly relied on GUI and voice commands. However, natural human communication also consists of non-verbal communication, including hand gestures like pointing. Thus, recent works in HMI systems have tried to incorporate pointing gestures as an input, making significant progress in recognizing and integrating them with voice commands. However, existing approaches often treat these input modalities independently, limiting their capacity to handle complex multimodal instructions requiring intricate reasoning of language and gestures. On the other hand, multimodal tasks requiring complex reasoning are being challenged in the language and vision domain, but these typically do not include gestures like pointing. To bridge this gap, we explore one of the challenging multimodal tasks, called Referring Expression Comprehension (REC), within multimodal HMI systems incorporating pointing gestures. We present a virtual setup in which a robot shares an environment with a user and is tasked with identifying objects based on the user's language and gestural instructions. Furthermore, to address this challenge, we propose a hybrid neuro-symbolic model combining deep learning's versatility with symbolic reasoning's interpretability. Our contributions include a challenging multimodal REC dataset for HMI systems, an interpretable neuro-symbolic model, and an assessment of its ability to generalize the reasoning to unseen environments, complemented by an in-depth qualitative analysis of the model's inner workings. [ABSTRACT FROM AUTHOR]

Objective To evaluate the effectiveness of preemptive multimodal analgesiain elderly patients undergoing laparoscopic colorectal surgery. Methods A prospective randomized controlled study was conducted in the Department of Gastrointestinal Surgery of the Second Hospital of Hebei Medical University from January 2022 to December 2022. A total of 133 patients were included according to the criteria and randomly divided into preemptive analgesia (PRA) group (test group, 67patients) and postoperative analgesia (POA) group (control group, 66patients). Results The Visual Analog Scale (VAS)scores of PRA group 24 h, 48 h, and 72 h after operation were lower than those of POA group, and the difference was statistically significant, P < 0.001.The incidences of postoperative gastrointestinal dysfunction (POGD) and postoperative delirium (POD)in PRA group were 13.43% and 8.98%, respectively, which were significantly lower than those in POA group (31.82% and 24.24%), P < 0.05. The levels of IL-6 and IL-10 in PRA group after the operation were 17.54 ± 2.13 ng/L and 15.57 ± 1.71 ng/L respectively, which were lower than those in POA group (25.45 ± 2.95 ng/L and 23.45 ± 1.88 ng/L), P < 0.05. The level of acetylcholinesterase(AchE) was 56.34 ± 5.62 nmol/L in the POA group, which was significantly higher than that in the POA group (49.59 ± 5.52 nmol/L), P < 0.001. Conclusion Preemptive multimodal analgesia can reduce the incidence of POGD and POD in elderly patients undergoing laparoscopic gastrocolic surgery, improve the recovery process of postoperative gastrointestinal function, increase the concentrations of propionic acid and butyric acid in short chain fatty acids (SCFAs) and the number of beneficial intestinal bacteria. [ABSTRACT FROM AUTHOR]

The rupture of vulnerable plaque (VP) are significant pathogenic factors leading to cardiovascular and cerebrovascular diseases. This study aims to construct a vulnerable plaque prediction model (VPPM) by combining multimodal vascular ultrasound parameters and clinical risk factors, and to validate it. A total of 196 atherosclerotic patients who underwent carotid endarterectomy (CEA) from January 2017 to December 2023 were collected and divided into a modeling group (n = 137) and a validation group (n = 59). Clinical information including: hypertension, diabetes, smoking history, and body mass index (BMI) was included in the analysis. All patients underwent carotid ultrasound and contrast-enhanced ultrasound (CEUS) examination after admission, with main ultrasound parameters including thickness, echogenicity types, stenosis degree, and CEUS neovascularization grading of plaques. Independent risk factors for VP in CEA patients were screened through binary Logistic regression analysis, and a prediction model was established along with a nomogram. The calibration curve, receiver-operating characteristic curve (ROC), and decision curve analysis (DCA) were employed to assess the calibration, diagnostic efficacy, and clinical utility of the VPPM model. There were no significant statistical differences in multimodal vascular ultrasound parameters and clinical risk factors between the modeling and validation groups (P > 0.05). Binary Logistic regression analysis identified plaque thickness, echo type, CEUS neovascularization grading, BMI, and smoking history as 5 variables entering the prediction model. The VPPM model showed good diagnostic efficacy, with an area under the ROC curve of 0.959 (95% CI 0.915–0.999). Using the nomogram with a VPPM risk assessment score of 135.42 as the diagnostic cutoff value in the modeling group, the sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and Youden index were 88.1%, 94.1%, 14.98, 0.126, and 82.2%, respectively. In the DCA curve, the VPPM model curve was significantly better than two extreme lines, indicating good clinical utility. The VPPM model constructed by integrating multimodal ultrasound parameters and clinical key risk factors has high diagnostic efficacy and is expected to be an auxiliary tool for clinical diagnosis of vulnerable plaques. [ABSTRACT FROM AUTHOR]

Capsules applied in large-volume and unstructured organs, such as stomach and colon, should perform multimodal motion to effectively accomplish gastrointestinal tract diagnosis. For this purpose, a magnetic navigated dual-hemisphere capsule robot (DHCR) actuated by the spatial universal rotating magnetic field (SURMF) is proposed, utilizing its passive and active modes for fixed-point posture adjustment and rolling locomotion, respectively. The DHCR exhibits a distinctive physical property—self-standing characteristics, which can specify vertically upward orientation as the starting posture and benchmark for multimodal motion after the DHCR undergoes complex multimodal (pseudo-active, active, and passive modes) conversion. On the basis of the momentum moment theorem, a general dynamic model is devised to reveal the mechanism of multimodal transition through stability and posture response. Results show that the DHCR can flip spontaneously after applying SURMF and is finally in a stable passive mode with the vertically upward DHCR axis. Finally, the multimodal conversion of the DHCR is tested in turning navigation within the curved intestine, providing a new method of endoscopic diagnosis. [ABSTRACT FROM AUTHOR]

Stagnation detection has been proposed as a mechanism for randomized search heuristics to escape from local optima by automatically increasing the size of the neighborhood to find the so-called gap size, i. e., the distance to the next improvement. Its usefulness has mostly been considered in simple multimodal landscapes with few local optima that could be crossed one after another. In multimodal landscapes with a more complex location of optima of similar gap size, stagnation detection suffers from the fact that the neighborhood size is frequently reset to 1 without using gap sizes that were promising in the past. In this paper, we investigate a new mechanism called radius memory which can be added to stagnation detection to control the search radius more carefully by giving preference to values that were successful in the past. We implement this idea in an algorithm called SD-RLS m and show compared to previous variants of stagnation detection that it yields speed-ups for linear functions under uniform constraints and the minimum spanning tree problem. Moreover, its running time does not significantly deteriorate on unimodal functions and a generalization of the Jump benchmark. Finally, we present experimental results carried out to study SD-RLS m and compare it with other algorithms. [ABSTRACT FROM AUTHOR]

Multimodal sentiment analysis focuses on the fusion of multiple modalities. However, modality representation learning is a key step for better modality fusion, so how to fully learn the sentiment information of non-text modalities is a problem worth exploring. In addition, how to further improve the accuracy of sentiment polarity prediction is also a work to be studied. To solve the above problems, we propose a multimodal sentiment analysis model with effective context semantic modality fusion and sentiment polarity correction (CSMF-SPC). Firstly, we design a low-rank multimodal fusion network based on context semantic modality (CSM-LRMFN). CSM-LRMFN uses the bi-directional long short-term memory network to extract the context semantic features of non-text modalities, and the BERT to extract the features of text modality. Then, CSM-LRMFN adopts a low-rank multimodal fusion method to fully extract the interaction information among modalities with contextual semantics. Different from previous studies, to improve the accuracy of sentiment polarity prediction, we design a weight self-adjusting sentiment polarity penalty loss function, which makes the model learn more sentiment features that are conducive to model prediction through backpropagation. Finally, a series of comparative experiments are conducted on the CMU-MOSI and CMU-MOSEI datasets. Compared with the current representative models, CSMF-SPC achieves better experimental results. Among them, the Acc-2 (including zero) metric is increased by 1.41% and 1.58% on the word-aligned and unaligned CMU-MOSI datasets respectively; it is improved by 1.50% and 2.14% respectively on the CMU-MOSEI dataset, which indicates that the improvement of CSMF-SPC is effective. [ABSTRACT FROM AUTHOR]

Traditional sentiment analysis focuses on text-level sentiment mining, transforming sentiment mining into classification or regression problems, resulting in a sentiment analysis low accuracy rate. Sentiment analysis refers to the use of natural language processing, text analysis, and computational linguistics to systematically identify, extract, quantify, and study sentimental states. Therefore, more scholars have begun to focus on speech recognition and facial expression recognition research, and extracting and analysing people's sentiment tendencies can improve sentiment recognition accuracy. Traditional single-modal sentiment analysis can no longer meet people's needs. Therefore, this paper proposes a multimodal sentiment analysis method based on the multimodal sentiment analysis method that can obtain more sentimental information sources and help people make better decisions. The experimental results in this paper show that the highest recognition rates of CNN-SVM, RNN-SVM, and CRNN-SVM were 76.8%, 71.2%, and 93.5%, respectively. It can be seen that CRNN-SVM has the highest sentiment tendency recognition rate in deep learning, so it is suitable to apply CRNN-SVM to sentiment tendency analysis system design in this paper. The average accuracy rate of the system designed in this paper was 91%, and the stability was also very strong, which shows that the system designed in this paper is meaningful. The main contribution of this paper is based on the limitations of single-mode emotion analysis. It proposes a multimode emotion analysis method and introduces a convolutional neural network to help people obtain more emotional information sources to meet their needs. [ABSTRACT FROM AUTHOR]

The core premise of cancelable biometrics lies in the creation of a distinct biometric template for every individual, which can be either canceled or regenerated as needed. This process requires the use of a uniquely-defined key during the generation of such template. The generated templates are tailored to be key-specific. This ensures that each distinct key will generate a unique template, while preserving the integrity and security of the original biometric data, ensuring that it remains uncompromised. In this paper, a cancelable biometric system based on electrocardiography (ECG) and photoplethysmography (PPG) signals is introduced. A signal fusion process is implemented for the two traits to generate a single template per user. In order to enhance the security of generated templates, a well-designed permutation stage is implemented according to a user-specific key. The permutation key is obtained through a well-designed look-up table created by the authors. The user verification is conducted on the cancelable template, without the need for any inversion processes. The user verification scheme depends on a two-pronged approach: robust feature extraction followed by the application of a machine learning (ML) classifier. The mel-frequency cepstral coefficients (MFCCs) extraction algorithm is employed for feature extraction due to the low frequency range of the adopted biometric signals and the nonlinearity of the filter bank used for MFCC extraction. Several ML classifiers are adopted to validate the system with cancelable templates without any inversion process. Simulation results with multilayer perceptron (MLP) and logistic regression (LR) classifiers demonstrated superior effectiveness of the proposed authentication framework, with accuracy rates up to 100% and 99.7% on the pulse transit time PPG and BIDMC datasets, respectively. Hence, the proposed system proves effective access control and user verification in the Internet-of-Things (IoT) applications. [ABSTRACT FROM AUTHOR]

With the increasing popularity of watching baseball videos, there is a growing desire among fans to enjoy the highlights of these videos. However, the extraction of the highlights from lengthy baseball videos faces a significant challenge due to its time-consuming and labor-intensive nature. To address this challenge, this paper proposes a novel mechanism, called Hierarchical Multimodal Transformer for Video query (HMTV). The proposed HMTV incorporates a two-phase involving Coarse-Grained clipping for candidate videos and Fine-Grained identification for highlights. In the Coarse-Grained phase, a pitching detection model is employed to extract relevant candidate videos from baseball videos, encompassing the features of pitch deliveries and pitching. In the Fine-Grained phase, Transformer encoder and pre-trained Bidirectional Encoder Representations from Transformers (BERT) are utilized to capture relationship features between frames of candidate videos and words from users’ questions, respectively. These relationship features are then fed into the Video Query (VideoQ) model, implemented by the Text Video Attention (TVA). The VideoQ model identifies the start and end positions of the highlights mentioned in the query within the candidate videos. Simulation results demonstrate that the proposed HMTV significantly improves accuracy of highlights identification in terms of precision, recall, and F1-score. [ABSTRACT FROM AUTHOR]

This paper aims to design a multimodal teaching platform to facilitate users by learning oral Chinese as a Foreign language. The designed system intends to improve the oral learning experience, efficiency and fluency. Firstly, this paper describes the overall design of Chinese as a Foreign language system for speaking using a multimodal teaching mode. Secondly, it analyzes the architecture of the oral Chinese language as a Foreign language system by introducing specific functional modules and system processes. Thirdly, it uses the optimization algorithm of Support Vector Machine (SVM) based on fruit fly optimization to optimize the parameters by enhancing the degree of oral fluency of Chinese as a Foreign language, gradually accumulate the rules of oral learning, and complete the design of oral Chinese as a Foreign language system based on multimodal Teaching mode. Finally, it incorporates SVM and the improved SVM for Drosophila Algorithm (ISVM-DA) to optimize language learning by accurately classifying vocabulary and enhancing learning. The experimental results showed that the proposed system has the maximum Stability Ratio of 90%, substantial Sensitivity of 85%, and Recall values of 88% compared to existing systems by proving the good robustness of the system. The system designed in this paper can stimulate users' interest in learning oral Chinese as a Foreign language and improve the efficiency and level of users' learning oral Chinese as a Foreign language. [ABSTRACT FROM AUTHOR]

Abstract International English-graded readers have been increasingly adapted, rewritten, and localised to cater to the specific needs of learners in different regions. This cross-cultural adaptation of materials has not attracted much scholarly attention, although it is an essential dimension of research on materials for language learning and teaching. The present study explores the textual transformations that occur in the bilingual adaptation of Cambridge English Readers, a series of original fiction books for learners of English, by a Chinese publisher. The analysis is underpinned by social semiotic multimodal discourse theory. Based on the analysis, the study evaluates the pedagogic potential of the bilingual edition of Cambridge English Readers as materials for English language learning and teaching. In doing so, it demonstrates a novel approach to evaluating bilingual adaptations of graded readers. It also shows that the original and bilingually adapted editions reflect different cultures of learning.

Multimodal sentiment analysis (MSA) is crucial for constructing the complex relationship across modalities and learning effective representations of multimodal information. However, current methods primarily focus on consistent information across modalities, neglecting the specific information unique to each modality. Additionally, most MSA datasets lack annotations for individual modality, limiting the thorough exploration of modality-specific information. Therefore, for the aim of modeling multimodal representations and capturing modality-specific information effectively, we propose a novel approach for MSA based on dual representation and self-supervised strategy (DRSS). Firstly, each modality input is decomposed into modality-consistent and modality-specific representation using a modality-dual-representation (MDR) learning network, which employs a shared consistency learning network to capture modality-consistent features and distinct specificity learning networks to extract modality-specific features. By contrasting modality-consistent and -specific representations within the same sample, the model’s representational capacity is further enhanced. Furthermore, the unimodal label generation module (ULGM) based on self-supervised learning is employed to generate unimodal labels for each modality, preserving the differentiation between modalities through unimodal label prediction. Finally, a multitask prediction loss is introduced to jointly train multimodal, unimodal, and contrastive learning tasks. The optimization of modality consistency and specificity losses is integrated into a unified loss function. Additionally, a self-adjusting strategy is employed in unimodal tasks to dynamically update the loss weights, guiding the model to effectively learn modality specificity. Experimental results on the CMU-MOSI and CMU-MOSEI datasets, which lack unimodal annotations, demonstrate that DRSS achieves competitive performance across a range of metrics. [ABSTRACT FROM AUTHOR]

Multimodal aspect-based sentiment analysis aims to predict the sentiment polarity of all aspect targets from text-image pairs. Most existing methods fail to extract fine-grained visual sentiment information, leading to alignment issues between the two modalities due to inconsistent granularity. In addition, the deep interaction between syntactic structure and semantic information is also ignored. In this paper, we propose an Aspect-aware Semantic Feature Enhancement Network (ASFEN) for multimodal aspect-based sentiment analysis to learn aspect-aware semantic and sentiment information in images and texts. Specifically, images are converted into textual information with fine-grained emotional cues. We construct dependency syntax trees and multi-layer syntax masks to fuse syntactic and semantic information through graph convolution. Extensive experiments on two multimodal Twitter datasets demonstrate the superiority of ASFEN over existing methods. The code is publicly available at . [ABSTRACT FROM AUTHOR]

Objectives: This study aimed to analyze the clinical and multimodal imaging manifestations of adult-onset neuronal intranuclear inclusion disease (NIID) patients and to investigate NIID-specific neuroimaging biomarkers. Methods: Forty patients were retrospectively enrolled from the Qilu Hospital of Shandong University. We analyzed the clinical and imaging characteristics of 40 adult-onset NIID patients and investigated the correlation between these characteristics and genetic markers and neuropsychological scores. We further explored NIID-specific alterations using multimodal imaging indices, including diffusion tensor imaging (DTI), magnetic resonance spectroscopy (MRS), and brain age estimation. In addition, we summarized the dynamic evolution pattern of NIID by examining the changes in diffusion weighted imaging (DWI) signals over time. Results: The NIID patients' ages ranged from 31 to 77 years. Cognitive impairment was the most common symptom (30/40, 75.0%), while some patients (18/40, 45.0%) initially presented with episodic symptoms such as headache (10/40, 25.0%). Patients with cognitive impairment symptoms had more cerebral white matter damage (χ2 = 11.475, P = 0.009). The most prevalent imaging manifestation was a high signal on DWI in the corticomedullary junction area, which was observed in 80.0% (32/40) of patients. In addition, the DWI dynamic evolution patterns could be classified into four main patterns. Diffusion tensor imaging (DTI) revealed extensive thinning of cerebral white matter fibers. The estimated brain age surpassed the patient's chronological age, signifying advanced brain aging in NIID patients. Conclusions: The clinical manifestations of NIID exhibit significant variability, usually leading to misdiagnosis. Our results provided new imaging perspectives for accurately diagnosing and exploring this disease's neuropathological mechanisms. [ABSTRACT FROM AUTHOR]

Multimodal sentiment analysis (MSA) seeks to understand human affection by leveraging signals from multiple modalities. A core challenge in MSA is the effective extraction of sentimental relations between these signals, as this can enhance a model's consistency and accuracy. Existing studies typically use multimodal matching tasks to learn all semantic relations between modalities and then use downstream task to obtain the specific semantics from the multimodal representation. However, there are multiple semantics between modalities, such as action semantics, scene semantics and sentiment semantics. Relying solely on specific tasks to filter these semantics often results in a surplus of redundant information in the multimodal representation, potentially degrading MSA accuracy. In addition, the unimodal semantic expression is also important. In this paper, we propose a semantic-specific multimodal relation learning method to correlate modalities with specific semantics. Specifically, with smaller computational resources, we enhance unimodal sentimental semantic expression while diminishing non-sentimental semantic information in the multimodal representation. We conducted experiments on multimodal sentiment analysis datasets, CMU-MOSI, CMU-MOSEI and CH-SIMS. The results show that our method outperforms the current state-of-the-art. Notably, on the Acc2 evaluation metric, our approach exhibits an average accuracy improvement of 0.75 compared to the best baseline. [ABSTRACT FROM AUTHOR]

A novel hybrid fusion scheme is proposed by employing non-subsampled shearlet transform (NSST) and stationary wavelet transform (SWT). In the preliminary stage, multimodal input images are putrefied comprehensively by NSST. The regional energy as an activity parameter is used to fuse the high-frequency sub-band coefficients of NSST. The approximation sub-band of NSST is fused using SWT. The maximum entropy of the squared coefficients and regional energy as the activity parameters are employed to fuse the LF and HF sub-bands of SWT, respectively. This step is explicitly performed for preserving contrast, edges, texture and brightness information within the image. The final output is taken by employing the inverse NSST. The research is verified both qualitatively and quantitatively from the fused images. It is seen that the suggested methodology obtained enriched results of the textural details such as divergence, boundaries, consistency and brightness of any of the brain intracranial masses (tumor or stroke or haemorrhage or even fungal infection). It is further observed that the morphology of the associated intracranial mass and the minimum path distance during invasive surgery along with the bone are prerequisites for the radiologist and are better depicted by the present method (as per the radiologist's perception). The fusion performance parameters of the suggested technique are related to seven existing methods. The quantitative evaluation of the proposed algorithm is done using mutual information, edge information index, entropy, standard deviation and mean. The parametric values obtained in any of the publicly available datasets or real-time datasets gave assuring outcomes. [ABSTRACT FROM AUTHOR]

Multimodal, single-cell genomics technologies enable simultaneous measurement of multiple facets of DNA and RNA processing in the cell. This creates opportunities for transcriptome-wide, mechanistic studies of cellular processing in heterogeneous cell populations, such as regulation of cell fate by transcriptional stochasticity or tumor proliferation through aberrant splicing dynamics. However, current methods for determining cell types or 'clusters' in multimodal data often rely on ad hoc approaches to balance or integrate measurements, and assumptions ignoring inherent properties of the data. To enable interpretable and consistent cell cluster determination, we present meK-means (mechanistic K-means) which integrates modalities through a unifying model of transcription to learn underlying, shared biophysical states. With meK-means we can cluster cells with nascent and mature mRNA measurements, utilizing the causal, physical relationships between these modalities. This identifies shared transcription dynamics across cells, which induce the observed molecule counts, and provides an alternative definition for 'clusters' through the governing parameters of cellular processes., (© 2024. The Author(s), under exclusive licence to Springer Nature America, Inc.)

Recent studies show that the use of multimodality can effectively enhance the understanding of social media content. The relations between texts and images become an important basis for developing multimodal data and models. Some studies have attempted to label image-text relation (ITR) and build supervised learning models. However, manually labeling ITR is a challenging task and incurs many controversial labels because of disagreements among the annotators. In this paper, we present a novel unsupervised multimodal method called ITR pseudo-labeling (ITRp) that learns multimodal representations for various ITR types using different finetuning strategies. Our ITRp method generates pseudo-labels by clustering and uses them as supervision to train the classifier and encoders. We evaluate the ITRp method on the ITR dataset and the effects of the samples with incorrect labels on both the supervised and unsupervised models. The code and data are available on the website https://github.com/SuYindu/ITRp. [ABSTRACT FROM AUTHOR]

Major Depressive Disorder (MDD) is an affective disorder that can lead to persistent sadness and a decline in the quality of life, increasing the risk of suicide. Utilizing multimodal data such as electroencephalograms and patient interview audios can facilitate the timely detection of MDD. However, existing depression detection methods either consider only a single modality or do not fully account for the differences and similarities between modalities in multimodal approaches, potentially overlooking the latent information inherent in various modal data. To address these challenges, we propose EMO-GCN, a multimodal depression detection method based on an adaptive multi-graph neural network. By employing graph-based methods to model data from various modalities and extracting features from them, the potential correlations between modalities are uncovered. The model's performance on the MODMA dataset is outstanding, achieving an accuracy (ACC) of 96.30%. Ablation studies further confirm the effectiveness of the model's individual components.The experimental results of EMO-GCN demonstrate the application prospects of graph-based multimodal analysis in the field of mental health, offering new perspectives for future research. [ABSTRACT FROM AUTHOR]

Traditional English corpora mainly collect information from a single modality, but lack information from multimodal information, resulting in low quality of corpus information and certain problems with recognition accuracy. To solve the above problems, this paper proposes to introduce depth information into multimodal corpora, and studies the construction method of English multimodal corpora that integrates electronic images and depth information, as well as the speech recognition method of the corpus. The multimodal fusion strategy adopted integrates speech signals and image information, including key visual information such as the speaker's lip movements and facial expressions, and uses deep learning technology to mine acoustic and visual features. The acoustic model in the Kaldi toolkit is used for experimental research.Through experimental research, the following conclusions were drawn: Under 15-dimensional lip features, the accuracy of corpus A under monophone model was 2.4% higher than that of corpus B under monophone model when the SNR (signal-to-noise ratio) was 10dB, and the accuracy of corpus A under the triphone model at the signal-to-noise ratio of 10dB was 1.7% higher than that of corpus B under the triphone model at the signal-to-noise ratio of 10dB. Under the 32-dimensional lip features, the speech recognition effect of corpus A under the monophone model at the SNR of 10dB was 1.4% higher than that of corpus B under the monophone model at the SNR of 10dB, and the accuracy of corpus A under the triphone model at the SNR of 10dB was 2.6% higher than that of corpus B under the triphone model at the SNR of 10dB. The English multimodal corpus with image and depth information has a high accuracy, and the depth information helps to improve the accuracy of the corpus. [ABSTRACT FROM AUTHOR]

Immersive virtual reality (VR) is anticipated to peak in development this decade bringing new opportunities for 3D multimodal designing across all levels of education. The need for students to gain capabilities with multimodal texts—texts that combine two or more modes, such as spoken, written, and visual—is emphasised at all levels of education from P-12 in the Australian Curriculum. Likewise, the use of technology-supported pedagogies is increasing worldwide, rendering multimodal texts ubiquitous across all knowledge domains. This original, qualitative classroom research investigated students' 3D designing of multimodal texts using an immersive VR head-mounted display. Upper primary students (ages 10–12 years, n = 48) transferred their knowledge of ancient Rome through 2D drawing, writing, speaking, and 3D multimodal designing with VR. The application of multimodal analysis to video data, screen recordings, and think-aloud protocols, and the thematic coding of student and teacher interviews yielded four key findings: (i) VR gaming supported 3D multimodal designing through haptic and embodied experience, (ii) VR improved performance through creative redesigning, (iii) VR-supported knowledge application, consolidation, and transfer, and (iv) pedagogical strengths of VR were situated and transformed practice. This research is timely and significant given the increasing accessibility and affordability of VR and the need to connect research and pedagogical practice to support students' advanced knowledge and capabilities with multimodal learning across the curriculum. [ABSTRACT FROM AUTHOR]

Multimodal named entity recognition (MNER) is an emerging foundational task in natural language processing. However, existing methods have two main limitations: 1) previous methods have focused on the visual representation of the entire image or target objects. However, they overlook the fine-grained semantic correspondence between entities and visual target objects, or ignore the visual cues of the overall scene and background details in the image. 2) Existing methods have not effectively overcome the semantic gap between different modalities due to the heterogeneity between text and images. To address these issues, we propose a novel multimodal heterogeneous graph entity-level fusion method for MNER (HGMVG) to achieve cross-modal feature interaction from coarse to fine between text and images under the guidance of visual information at different granularities, which can improve the accuracy of named entity recognition. Specifically, to resolve the first issue, we cascade cross-modal semantic interaction information between text and vision at different visual granularities to obtain a comprehensive and effective multimodal representation. For the second issue, we describe the precise semantic correspondences between entity-level words and visual target objects via multimodal heterogeneous graphs, and utilize heterogeneous graph attention networks to achieve cross-modal fine-grained semantic interactions. We conduct extensive experiments on two publicly available Twitter datasets, and the experimental results demonstrate that HGMVG outperforms the current state-of-the-art models in the MNER task. [ABSTRACT FROM AUTHOR]

Social media platforms have seen an influx of multimodal data, leading to heightened attention on image-text multimodal classification. Existing methods for multimodal classification primarily focus on multimodal fusion from different modalities. However, owing to the heterogeneity and high-dimensionality of multimodal data, the fusion process frequently introduces redundant information and noise limiting the accuracy and generalization. To resolve the limitation, we propose a Compact Bilinear pooling and Multi-Loss network (CBMLNet). Compact bilinear pooling is used for feature fusion to learn low-dimensional and expressive multimodal representations efficiently. Furthermore, a multi-loss function is proposed to import the specific information carried by each single modality. Therefore, CBMLNet simultaneously considers the correlation between multimodality and the specificity of single modality for image-text classification. We evaluate the proposed CBMLNet on two publicly available datasets, Twitter-15 and Twitter-17, and on a private dataset, AIFUN. CBMLNet is compared with the advanced methods such as multimodal BERT with Max Pooling, Multi-Interactive Memory Network, Multi-level Multi-modal Cross-attention Network, Image-Text Correlation model (ITC), Target-oriented multimodal BERT and multimodal hierarchical attention model (MHA). Experimental results demonstrate that CBMLNet averagely improves F1_score by 0.28% and 0.44% compared with the best fine-grained baseline, MHA and the best coarse-grained baseline, ITC. It illustrates that CBMLNet is practical for real-world fuzzy applications as a coarse-grained model. [ABSTRACT FROM AUTHOR]

Most multimodal recommender systems utilize multimodal content of user-interacted items as supplemental information to capture user preferences based on historical interactions without considering user-uninteracted items. In contrast, multimodal recommender systems based on causal inference counterfactual learning utilize the causal difference between the multimodal content of user-interacted and user-uninteracted items to purify the content related to user preferences. However, existing methods adopt a unified multimodal channel, which treats each modality equally, resulting in the inability to distinguish users’ tastes for different modalities. Therefore, the differences in users’ attention and perception of different modalities' content cannot be reflected. To cope with the above issue, this paper proposes a novel recommender system based on multi-channel counterfactual learning (MCCL) networks to capture user fine-grained preferences on different modalities. First, two independent channels are established based on the corresponding features for the content of image and text modalities for modality-specific feature extraction. Then, leveraging the counterfactual theory of causal inference, features in each channel unrelated to user preferences are eliminated using the features of the user-uninteracted items. Features related to user preferences are enhanced and multimodal user preferences are modeled at the content level, which portrays the users' taste for the different modalities of items. Finally, semantic entities are extracted to model semantic-level multimodal user preferences, which are fused with historical user interaction information and content-level user preferences for recommendation. Extensive experiments on three different datasets show that our results improve up to 4.17% on NDCG compared to the optimal model. [ABSTRACT FROM AUTHOR]

Documents contain abundant information available for managerial decision-making. However, manual methods of screening document information lack accuracy due to the heterogeneity of documents. To address the above issue, we propose a multimodal network combining multivariate semantic association graphs, MMIE, for accurately extracting information from documents. Firstly, the multivariate semantic graphs between multimodal data within each modality are constructed based on the semantic association of text contents, followed by the semantic relationships in the graphs to lead the fusion and embedding of the extracted multimodal data and improve the feature representation capability. Subsequently, the semantically linked multimodal information is fed into the newly constructed multimodal self-attention module to better establish inter-modal associations. Finally, a supervised comparison learning loss function is employed to reduce further the information loss due to sample imbalance. The experimental results on three real datasets show that the proposed model can extract feature information of different modal data more accurately, and the F1 scores reach 87.28 % , 82.53 % , and 81.17 % , respectively. [ABSTRACT FROM AUTHOR]

Abstract The article deals with the visual and linguistic representation of alleged perpetrators through personalization in news reporting from The Telegraph’s online media platform. The analysis shows how visual message, along with verbal labelling in the text, represent different groups of alleged perpetrators as more or less ‘dangerous’ in news reports. Agency is analyzed through a focus on the lens range and its influence on perceived social distance, the angle of the shot and its role in the understanding of social relations, as well as the direction of an alleged perpetrator’s gaze in images as a way of social interaction with the viewer. The photos of alleged perpetrators appear in The Telegraph with the aim of informing the public about the danger these individuals pose, as well as legitimizing the actions of law enforcement institutions. It appears that verbal and visual identification of these individuals is done with the intention of ‘excluding’ the most dangerous social actors, such as ‘murderers’, from ingroups.

Multimodal sentiment recognition has obtained increasing attention in recent years due to its potential to improve sentiment recognition accuracy by integrating information from multiple modalities. However, the heterogeneity issue caused by the differences in modalities poses a significant challenge for multimodal sentiment recognition. In this paper, we propose a novel framework, Cross-Modal Contrastive Learning (CMCL), which integrates multiple contrastive learning methods and multimodal data augmentation to address the heterogeneity issue. Specifically, we establish a cross-modal contrastive learning framework by leveraging diversity contrastive learning, consistency contrastive learning and sample-level contrastive learning. Through diversity contrastive learning, we constrain modality features to different feature spaces, capturing the complementary nature of modality-specific features. Additionally, through consistency contrastive learning, we map the representations of different modalities into a shared feature space, capturing the consistency of modality-specific features. We also introduce two data augmentation techniques, namely random noise and modal combination, to improve the model's robustness. The experimental results show that our approach achieves state-of-the-art performance on three benchmark datasets and outperforms the existing baseline models. Our work demonstrates the effectiveness of cross-modal contrastive learning and data augmentation in multimodal sentiment recognition, and provides valuable insights for future research in this area. [ABSTRACT FROM AUTHOR]

Task-oriented multimodal dialogue systems have important application value and development prospects. Existing methods have made significant progress, but the following challenges still exist: (1) Most existing methods focus on improving the accuracy of dialogue state tracking and dialogue act prediction. However, the essential to leverage knowledge in the knowledge base to supplement textual responses in multi-turn dialogues is ignored. (2) One feature that distinguishes multimodal dialogue from plain text dialogue is the usage of visual information. However, existing methods ignore the importance of accurately providing visual information to improve user satisfaction. (3) For multimodal dialogue systems, most existing methods ignore the classification of response types to assign appropriate response generators automatically. To address the issues above, we present a user-satisfactory multimodal dialogue system, USMD for short. Specifically, USMD is designed as four modules. The general response generator is based on generative pre-training 2.0 (GPT-2) to generate dialogue acts and general textual responses. The knowledge-enriched response generator is designed to leverage a structured knowledge base under the guidance of a knowledge graph. The image recommender pays attention to both latent and explicit visual cues, a deep multimodal fusion model to obtain informative image representations. Finally, the response classifier automatically selects the appropriate generators to answer the user based on user and agent actions. Extensive experiments on the benchmark multimodal dialogue datasets show that the proposed USMD model achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]