Search

Robot welding penetration control is an important research topic in the field of robot welding online control. This study proposes a recognition control system for multi-modal signals with multiple welding parameters for robot welding penetration control. In this study, a filter bank suitable for welding environments was proposed for welding acoustic signals, and separation and contour extraction algorithms were designed for the welding image signals. Taking the acoustic and image signals as inputs, a multi-modal hybrid model and multi-parameter controller were established that can effectively classify and identify the welding penetration state. The penetration state was used as a reference input to the controller for controlling the welding speed and current in real time. In addition, a digital twin system was developed in this study and deployed on the main and edge computers. The test results show that the system and model can accurately identify the weld penetration, regulate the welding speed and welding current, control the width of the backside molten pool, and improve the welding quality. [ABSTRACT FROM AUTHOR]

Emotion recognition in conversation (ERC) represents a pivotal research domain within affective computing, concentrating on discerning the emotional nuances embedded within individual utterances during conversational exchanges. The majority of current research focuses on modeling situational cues, with relatively little attention paid on affective tendencies inherent in emotional expression. Moreover, ensuring the fair representation of diverse modalities in emotional expression presents a significant challenge in effectively extracting synergies and insights from multi-modal data sources. To tackle these challenges, this study proposes a novel approach termed the Uni-Modal and Cross-Modal Encoding Network based on Multi-Head Attention (UCEMA) for ERC. The framework leverages two distinct encoding techniques, namely Uni-Modal Encoding based on Multi-Head Attention (UEMA) and Cross-Modal Encoding based on Multi-Head Attention (CEMA), to extract distinct emotional features from individual modes and facilitate the fusion of emotional attributes within the multi-modal context. Particular emphasis is placed on textual input as the primary mode of interaction. Additionally, this study employs Context Modeling (CM) to analyze the outcomes of emotion recognition in conversational contexts. A comprehensive comparative analysis of the UCEMA was conducted on two publicly available datasets, IEMOCAP and MELD. The results demonstrated that the UCEMA exhibited superior efficacy. It is noteworthy that the proposed framework effectively balances intra-modal emotional orientation information, inter-modal emotional association information, and context-related cues, thereby demonstrating superior performance in recognition accuracy compared to current state-of-the-art (SOTA) models. [ABSTRACT FROM AUTHOR]

Developing agents to follow language instructions is a compelling yet challenging research topic. Recently, vision-and-language navigation in continuous environments has been proposed to explore the multi-modal pattern analysis and mapless navigation abilities of intelligent agents. However, current waypoint-based methods still have shortcomings, such as the coupled decision process and the possible shortest path-instruction misalignment. To address these challenges, we propose an instruction-aligned hierarchical waypoint planner (IA-HWP) that ensures fine-grained waypoint prediction and enhances instruction alignment. Our HWP architecture decouples waypoint planning into a coarse view selection phase and a refined waypoint location phase, effectively improving waypoint quality and enabling specialized training supervision for different phases. In terms of instruction-aligned model design, we introduce the global action-vision co-grounding and local text-vision co-grounding modules to explicitly improve the understanding of visual landmarks and actions, thereby enhancing the alignment between instructions and trajectories. In terms of instruction-aligned model optimization, we employ reference-waypoint-oriented supervision and direction-aware loss to optimize the model for enhanced instruction following and waypoint execution capabilities. Experiments on the standard benchmark demonstrate the effectiveness of our approach, with improved success rate compared to existing methods. [ABSTRACT FROM AUTHOR]

Recent days have seen significant improvements in multi-modal learning made by Vision-Language Pre-training (VLP) models. However, most of them employ the coarse-grained global alignment to overcome semantic gap for generating common representations, which makes them inadequate to capture intrinsic semantic correlations in image-text retrieval and consequently degrading the accuracy. Moreover, it is expensive to fine-tune a VLP model to perform image-text retrieval due to its large number of parameters. In this paper, we propose a simple yet effective image-text retrieval method, termed Cross-Modality Interaction Reasoning for enhancing Vision-Language Pre-training (CMIR-VLP). Specifically, a Cross Modality Interaction Reasoning (CMIR) module, which is designed to inject fine-grained image-text associations into semantic correlations learning, integrates the patch cues into the word reasoning with a multi-modal interaction encoder. Besides, we propose a cross-interaction process to associate each local text semantics with local visual information for fine-grained image-text alignment. Extensive experiments demonstrate our method gains 52 and 97.5 over state-of-the-art non-pre-training methods on two widely used datasets, and it also outperforms several mainstream fine-tuned VIP models. The related code repository in https://github.com/PSYGIM/CMIR-VLP. [ABSTRACT FROM AUTHOR]

With the simultaneous acquisition of the infrared and optical remote sensing images of the same target becoming increasingly easy, using multi-modal data for high-performance object detection has become a research focus. In remote sensing multi-modal data, infrared images lack color information, it is hard to detect difficult targets with low contrast, and optical images are easily affected by illuminance. One of the most effective ways to solve this problem is to integrate multi-modal images for high-performance object detection. The challenge of fusion object detection lies in how to fully integrate multi-modal image features with significant modal differences and avoid introducing interference information while taking advantage of complementary advantages. To solve these problems, a new multi-modal fusion object detection method is proposed. In this paper, the method is improved in terms of two aspects: firstly, a new dual-branch asymmetric attention backbone network (DAAB) is designed, which uses a semantic information supplement module (SISM) and a detail information supplement module (DISM) to supplement and enhance infrared and RGB image information, respectively. Secondly, we propose a feature fusion pyramid network (FFPN), which uses a Transformer-like strategy to carry out multi-modal feature fusion and suppress features that are not conducive to fusion during the fusion process. This method is a state-of-the-art process for both FLIR-aligned and DroneVehicle datasets. Experiments show that this method has strong competitiveness and generalization performance. [ABSTRACT FROM AUTHOR]

Depth map super-resolution (DSR) is a technique aimed at restoring high-resolution (HR) depth maps from low-resolution (LR) depth maps. In this process, color images are commonly used as guidance to enhance the restoration procedure. However, the intricate degradation of LR depth poses a challenge, and previous image-guided DSR approaches, which implicitly model the degradation in the spatial domain, often fall short of producing satisfactory results. To address this challenge, we propose a novel approach called the Degradation-Guided Multi-modal Fusion Network (DMFNet). DMFNet explicitly characterizes the degradation and incorporates multi-modal fusion in both spatial and frequency domains to improve the depth quality. Specifically, we first introduce the deep degradation regularization loss function, which enables the model to learn the explicit degradation from the LR depth maps. Simultaneously, DMFNet converts the color images and depth maps into spectrum representations to provide comprehensive multi-domain guidance. Consequently, we present the multi-modal fusion block to restore the depth maps by leveraging both the RGB-D spectrum representations and the depth degradation. Extensive experiments demonstrate that DMFNet achieves state-of-the-art (SoTA) performance on four benchmarks, namely the NYU-v2, Middlebury, Lu, and RGB-D-D datasets. [ABSTRACT FROM AUTHOR]

With the emergence and development of social networks, people can stay in touch with friends, family, and colleagues more quickly and conveniently, regardless of their location. This ubiquitous digital internet environment has also led to large-scale disclosure of personal privacy. Due to the complexity and subtlety of sensitive information, traditional sensitive information identification technologies cannot thoroughly address the characteristics of each piece of data, thus weakening the deep connections between text and images. In this context, this paper adopts the CLIP model as a modality discriminator. By using comparative learning between sensitive image descriptions and images, the similarity between the images and the sensitive descriptions is obtained to determine whether the images contain sensitive information. This provides the basis for identifying sensitive information using different modalities. Specifically, if the original data does not contain sensitive information, only single-modality text-sensitive information identification is performed; if the original data contains sensitive information, multimodality sensitive information identification is conducted. This approach allows for differentiated processing of each piece of data, thereby achieving more accurate sensitive information identification. The aforementioned modality discriminator can address the limitations of existing sensitive information identification technologies, making the identification of sensitive information from the original data more appropriate and precise. [ABSTRACT FROM AUTHOR]

Abstract Most of the sentiment analysis studies focus on the sentimental classification of pictures in the video, ignoring the spatio-temporal information of the sequence of picture frames as well as text and audio information. The multiple kernel learning is a new hotspot in the field of nuclear machine learning, capable of handling multiple modalities. For multiple kernel learning, it is easy to ignore the basic features that are not discriminative, and cannot make full use of the base features of different modes. This paper puts forward a novel multi-modal fusion model for sentiment analysis, in which a multiple kernel learning algorithm based on convolution margin-dimension constraint is proposed for feature fusion. Moreover, the 3D convolutional neural network is used to extract the features of visual information, and the multiple kernel learning algorithm based on margin-dimension constraint is used to fuse visual, text and audio sentiment features. Experiments conducted on the MOUD and IEMOCAP sentiment databases show that the proposed model outperforms existing models in the field of multi-modal sentiment analysis research.

Abstract Programmed death-ligand 1 (PD-L1) expressions play a crucial role in guiding therapeutic interventions such as the use of tyrosine kinase inhibitors (TKIs) and immune checkpoint inhibitors (ICIs) in lung cancer. Conventional determination of PD-L1 status includes careful surgical or biopsied tumor specimens. These specimens are gathered through invasive procedures, representing a risk of difficulties and potential challenges in getting reliable and representative tissue samples. Using a single center cohort of 189 patients, our objective was to evaluate various fusion methods that used non-invasive computed tomography (CT) and $$^{18}$$ 18 F-FDG positron emission tomography (PET) images as inputs to various deep learning models to automatically predict PD-L1 in non-small cell lung cancer (NSCLC). We compared three different architectures (ResNet, DenseNet, and EfficientNet) and considered different input data (CT only, PET only, PET/CT early fusion, PET/CT late fusion without as well as with partially and fully shared weights to determine the best model performance. Models were assessed utilizing areas under the receiver operating characteristic curves (AUCs) considering their 95% confidence intervals (CI). The fusion of PET and CT images as input yielded better performance for PD-L1 classification. The different data fusion schemes systematically outperformed their individual counterparts when used as input of the various deep models. Furthermore, early fusion consistently outperformed late fusion, probably as a result of its capacity to capture more complicated patterns by merging PET and CT derived content at a lower level. When we looked more closely at the effects of weight sharing in late fusion architectures, we discovered that while it might boost model stability, it did not always result in better results. This suggests that although weight sharing could be beneficial when modality parameters are similar, the anatomical and metabolic information provided by CT and PET scans are too dissimilar to consistently lead to improved PD-L1 status predictions.

Physiological signal recognition is crucial in emotion recognition, and recent advancements in multi-modal fusion have enabled the integration of various physiological signals for improved recognition tasks. However, current models for emotion recognition with hyper complexmulti-modal signals face limitations due to fusionmethods and insufficient attentionmechanisms, preventing further enhancement in classification performance. To address these challenges, we propose a new model framework named Signal Channel Attention Network (SCA-Net), which comprises three main components: an encoder, an attention fusion module, and a decoder. In the attention fusion module, we developed five types of attention mechanisms inspired by existing research and performed comparative experiments using the public dataset MAHNOB-HCI. All of these experiments demonstrate the effectiveness of the attention module we addressed for our baseline model in improving both accuracy and F1 score metrics. We also conducted ablation experiments within the most effective attention fusionmodule to verify the benefits of multi-modal fusion. Additionally, we adjusted the training process for different attention fusion modules by employing varying early stopping parameters to prevent model overfitting. [ABSTRACT FROM AUTHOR]

Multi-modal knowledge graph (MKG) refers to a structured semantic network that accurately represents the real-world information by incorporating multiple modalities. Existing researches primarily focus on leveraging multi-modal fusion to enhance the representation capability of entity nodes and link prediction to deal with the incompleteness of the MKG. However, the inherent heterogeneity between structural modality and semantic modality poses challenges to the multi-modal fusion, as noise interference could compromise the effectiveness of the fusion representation. In this study, we propose a novel hierarchical Transformer architecture, named MNFormer, which captures the structural and semantic information while avoiding heterogeneity issues by fully integrating both multi-hop neighbor paths and image-text embeddings. During the encoding stage of MNFormer, we design multiple layers of Multi-hop Neighbor Fusion (MNF) module that employ attentions to merge the image and text features. These MNF modules progressively fuse the information of neighboring entities hop by hop along the neighbor paths of the source entity. The Transformer during decoding stage is then utilized to integrate the outputs of all MNF modules, whose output is subsequently employed to match target entities and accomplish MKG completion. Moreover, we develop a semantic direction loss to enhance the fitting performance of MNFormer. Experimental results on four datasets demonstrate that MNFormer exhibits notable competitiveness when compared to the state-of-the-art models. Additionally, ablation studies showcase the significant ability of MNFormer to effectively combine structural and semantic information, leading to enhanced performance through complementary enhancements. [ABSTRACT FROM AUTHOR]

Alzheimer's disease (AD) is an insidious and progressive neurodegenerative disease, and the annual relevant social cost for AD patients can reach about $1 trillion worldwide. Therefore, early diagnosis and treatment of AD play a vital role in slowing disease progression. However, existing detection methods for cognitive impairment cannot consistently screen the stage of AD. To tackle this challenge, we propose an AD detection system, Ubi-AD, which combines the features of multiple biomarkers to realize passive and accurate AD detection. Unlike existing work, Ubi-AD can passively recognize the AD digital biomarkers during daily smartwatch usage without interfering with the user. At the user end, Ubi-AD first extracts the non-speech sounds (pause words, such as em, ah), which contain no privacy-sensitive content. Then, Ubi-AD recognizes the user's walking activity, dining activity, and sleep activity from daily activities. Ubi-AD analyzes these data from smartwatch and predicts the AD stages using a multi-modal fusion neural network at the cloud end. We evaluate our model on a collected dataset from 45 volunteers. As a result, Ubi-AD can reach a detection accuracy of 93.4%, which means that Ubi-AD can provide multiple effective biomarkers for ubiquitous and passive detection in daily life. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Skin cancer is one of the most prevalent malignancies in the world. Deep learning-based methods have been successfully used for skin disease diagnosis and achieved great recognition performance, most of which relied on dermoscopic images alone. Existing multi-model methods of skin lesion diagnosis have the following two shortcomings: 1) They mainly focus on learning complementary information while ignoring the correlation between clinical and dermoscopic images. 2) The feature extractor is not optimized by imposing constraints, which may result in limited expression of the extracted features. To address these issues, this study proposes a new method, named multi-modal bilinear fusion with hybrid attention mechanism (MBF-HA) for multi-modal skin lesion classification. Specifically, MBF-HA introduced a common representation learning framework to learn the correlated features by exploring the shared characteristics between two modalities. Moreover, MBF-HA uses the hybrid attention-based reconstruction module which encourages the feature extractor to detect and localize lesion regions on each modality, thus, enhancing the discriminative power of the output feature representation. We perform comprehensive experiments on a well-established multi-modal and multi-label skin disease dataset: a 7-point Checklist database, MBF-HA achieves an average accuracy of 76.3% in the multi-classification tasks and 76.0% in the diagnostic task. The experimental results show that MBF-HA outperforms known related works and is expected to help physicians make more precise clinical diagnoses. [ABSTRACT FROM AUTHOR]

Programmed death-ligand 1 (PD-L1) expressions play a crucial role in guiding therapeutic interventions such as the use of tyrosine kinase inhibitors (TKIs) and immune checkpoint inhibitors (ICIs) in lung cancer. Conventional determination of PD-L1 status includes careful surgical or biopsied tumor specimens. These specimens are gathered through invasive procedures, representing a risk of difficulties and potential challenges in getting reliable and representative tissue samples. Using a single center cohort of 189 patients, our objective was to evaluate various fusion methods that used non-invasive computed tomography (CT) and 18 F-FDG positron emission tomography (PET) images as inputs to various deep learning models to automatically predict PD-L1 in non-small cell lung cancer (NSCLC). We compared three different architectures (ResNet, DenseNet, and EfficientNet) and considered different input data (CT only, PET only, PET/CT early fusion, PET/CT late fusion without as well as with partially and fully shared weights to determine the best model performance. Models were assessed utilizing areas under the receiver operating characteristic curves (AUCs) considering their 95% confidence intervals (CI). The fusion of PET and CT images as input yielded better performance for PD-L1 classification. The different data fusion schemes systematically outperformed their individual counterparts when used as input of the various deep models. Furthermore, early fusion consistently outperformed late fusion, probably as a result of its capacity to capture more complicated patterns by merging PET and CT derived content at a lower level. When we looked more closely at the effects of weight sharing in late fusion architectures, we discovered that while it might boost model stability, it did not always result in better results. This suggests that although weight sharing could be beneficial when modality parameters are similar, the anatomical and metabolic information provided by CT and PET scans are too dissimilar to consistently lead to improved PD-L1 status predictions. [ABSTRACT FROM AUTHOR]

In this study, we address the critical need for enhanced situational awareness and victim detection capabilities in Search and Rescue (SAR) operations amidst disasters. Traditional unmanned ground vehicles (UGVs) often struggle in such chaotic environments due to their limited manoeuvrability and the challenge of distinguishing victims from debris. Recognising these gaps, our research introduces a novel technological framework that integrates advanced gesture-recognition with cutting-edge deep learning for camerabased victim identification, specifically designed to empower UGVs in disaster scenarios. At the core of our methodology is the development and implementation of the Meerkat Optimization Algorithm--Stacked Convolutional Neural Network--Bi--Long Short Term Memory--Gated Recurrent Unit (MOASConv-Bi-LSTM-GRU) model, which sets a new benchmark for hand gesture detection with its remarkable performance metrics: accuracy, precision, recall, and F1-score all approximately 0.9866. This model enables intuitive, real-time control of UGVs through hand gestures, allowing for precise navigation in confined and obstacle-ridden spaces, which is vital for effective SAR operations. Furthermore, we leverage the capabilities of the latest YOLOv8 deep learning model, trained on specialised datasets to accurately detect human victims under a wide range of challenging conditions, such as varying occlusions, lighting, and perspectives. Our comprehensive testing in simulated emergency scenarios validates the effectiveness of our integrated approach. The system demonstrated exceptional proficiency in navigating through obstructions and rapidly locating victims, even in environments with visual impairments like smoke, clutter, and poor lighting. Our study not only highlights the critical gaps in current SAR response capabilities but also offers a pioneering solution through a synergistic blend of gesture-based control, deep learning, and purposebuilt robotics. The key findings underscore the potential of our integrated technological framework to significantly enhance UGV performance in disaster scenarios, thereby optimising life-saving outcomes when time is of the essence. This research paves the way for future advancements in SAR technology, with the promise of more efficient and reliable rescue operations in the face of disaster. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Data Acquisition & Processing / Shu Ju Cai Ji Yu Chu Li is the property of Editorial Department of Journal of Nanjing University of Aeronautics & Astronautics and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Multi-modal emotion recognition has attracted increasing attention in human-computer interaction, as it extracts complementary information from physiological and behavioral features. Compared to single modal approaches, multi-modal fusion methods are more susceptible to uncertainty in emotion recognition, such as heterogeneity and inconsistent predictions across different modalities. Previous multi-modal approaches ignore systematic modeling of uncertainty in fusion and revelation of dynamic variations in emotion process. In this article, we propose a dynamic confidence-aware fusion network for robust recognition of heterogeneous emotion features, including electroencephalogram (EEG) and facial expression. First, we develop a self-attention based multi-channel LSTM network to preliminarily align the heterogeneous emotion features. Second, we propose a confidence regression network to estimate true class probability (TCP) on each modality, which helps explore the uncertainty at modality level. Then, different modalities are weighted fused according to above two types of uncertainty. Finally, we adopt self-paced learning (SPL) mechanism to further improve the model robustness by alleviating negative effect from the hard learning samples. The experimental results on several multi-modal emotion datasets demonstrate the proposed method outperforms the state-of-the-art methods in emotion recognition performance and explicitly reveals the dynamic variation of emotion with uncertainty estimation. [ABSTRACT FROM AUTHOR]

In recent years, to address the issue of networked data sparsity in node classification tasks, cross-network node classification (CNNC) leverages the richer information from a source network to enhance the performance of node classification in the target network, which typically has sparser information. However, in real-world applications, labeled nodes may be collected from multiple sources with multiple modalities (e.g., text, vision, and video). Naive application of single-source and single-modal CNNC methods may result in sub-optimal solutions. To this end, in this article, we propose a model called Multi-source and Multi-modal Cross-network Deep Network Embedding (M2CDNE) for cross-network node classification. In M2CDNE, we propose a deep multi-modal network embedding approach that combines the extracted deep multi-modal features to make the node vector representations network invariant. In addition, we apply dynamic adversarial adaptation to assess the significance of marginal and conditional probability distributions between each source and target network to make node vector representations label discriminative. Furthermore, we devise to classify nodes in the target network through the related source classifier and aggregate different predictions utilizing respective network weights, corresponding to the discrepancy between each source and target network. Extensive experiments performed on real-world datasets demonstrate that the proposed M2CDNE significantly outperforms the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Multi-modal fusion technology gradually become a fundamental task in many fields, such as autonomous driving, smart healthcare, sentiment analysis, and human-computer interaction. It is rapidly becoming the dominant research due to its powerful perception and judgment capabilities. Under complex scenes, multi-modal fusion technology utilizes the complementary characteristics of multiple data streams to fuse different data types and achieve more accurate predictions. However, achieving outstanding performance is challenging because of equipment performance limitations, missing information, and data noise. This paper comprehensively reviews existing methods based on multi-modal fusion techniques and completes a detailed and in-depth analysis. According to the data fusion stage, multi-modal fusion has four primary methods: early fusion, deep fusion, late fusion, and hybrid fusion. The paper surveys the three major multi-modal fusion technologies that can significantly enhance the effect of data fusion and further explore the applications of multi-modal fusion technology in various fields. Finally, it discusses the challenges and explores potential research opportunities. Multi-modal tasks still need intensive study because of data heterogeneity and quality. Preserving complementary information and eliminating redundant information between modalities is critical in multi-modal technology. Invalid data fusion methods may introduce extra noise and lead to worse results. This paper provides a comprehensive and detailed summary in response to these challenges. [ABSTRACT FROM AUTHOR]

PURPOSES: To provide abdominal contrast-enhanced MR image synthesis, we developed an gradient regularized multi-modal multi-discrimination sparse attention fusion generative adversarial network (GRMM-GAN) to avoid repeated contrast injections to patients and facilitate adaptive monitoring. METHODS: With IRB approval, 165 abdominal MR studies from 61 liver cancer patients were retrospectively solicited from our institutional database. Each study included T2, T1 pre-contrast (T1pre), and T1 contrast-enhanced (T1ce) images. The GRMM-GAN synthesis pipeline consists of a sparse attention fusion network, an image gradient regularizer (GR), and a generative adversarial network with multi-discrimination. The studies were randomly divided into 115 for training, 20 for validation, and 30 for testing. The two pre-contrast MR modalities, T2 and T1pre images, were adopted as inputs in the training phase. The T1ce image at the portal venous phase was used as an output. The synthesized T1ce images were compared with the ground truth T1ce images. The evaluation metrics include peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and mean squared error (MSE). A Turing test and experts contours evaluated the image synthesis quality. RESULTS: The proposed GRMM-GAN model achieved a PSNR of 28.56, an SSIM of 0.869, and an MSE of 83.27. The proposed model showed statistically significant improvements in all metrics tested with p-values < 0.05 over the state-of-the-art model comparisons. The average Turing test score was 52.33%, which is close to random guessing, supporting the models effectiveness for clinical application. In the tumor-specific region analysis, the average tumor contrast-to-noise ratio (CNR) of the synthesized MR images was not statistically significant from the real MR images. The average DICE from real vs. synthetic images was 0.90 compared to the inter-operator DICE of 0.91. CONCLUSION: We demonstrated the function of a novel multi-modal MR image synthesis neural network GRMM-GAN for T1ce MR synthesis based on pre-contrast T1 and T2 MR images. GRMM-GAN shows promise for avoiding repeated contrast injections during radiation therapy treatment.

Parkinson's disease (PD) is a prevalent neurological disorder characterized by progressive dopaminergic neuron loss, leading to both motor and non-motor symptoms. Early and accurate diagnosis is challenging due to the subtle and variable nature of early symptoms. This study aims to address these diagnostic challenges by proposing a novel method, Localized Region Extraction and Multi-Modal Fusion (LRE-MMF), designed to enhance diagnostic accuracy through the integration of structural MRI (sMRI) and resting-state functional MRI (rs-fMRI) data. The LRE-MMF method utilizes the complementary strengths of sMRI and rs-fMRI: sMRI provides detailed anatomical information, while rs-fMRI captures functional connectivity patterns. We applied this approach to a dataset consisting of 20 PD patients and 20 healthy controls (HC), all scanned with a 3 T MRI. The primary objective was to determine whether the integration of sMRI and rs-fMRI through the LRE-MMF method improves the classification accuracy between PD and HC subjects. LRE-MMF involves the division of imaging data into localized regions, followed by feature extraction and dimensionality reduction using Principal Component Analysis (PCA). The resulting features were fused and processed through a neural network to learn high-level representations. The model achieved an accuracy of 75 %, with a precision of 0.8125, recall of 0.65, and an AUC of 0.8875. The validation accuracy curves indicated good generalization, with significant brain regions identified, including the caudate, putamen, thalamus, supplementary motor area, and precuneus, as per the AAL atlas. These results demonstrate the potential of the LRE-MMF method for improving early diagnosis and understanding of PD by effectively utilizing both sMRI and rs-fMRI data. This approach could contribute to the development of more accurate diagnostic tools.