Search

AbstractPurposeMethodsResultsConclusionWe designed a dual-modal fusion network to detect glaucomatous optic neuropathy, which utilized both retinal nerve fiber layer thickness from optical coherence tomography reports and fundus images.A total of 327 healthy subjects (410 eyes) and 87 glaucomatous optic neuropathy patients (113 eyes) were included. The retinal nerve fiber layer thickness from optical coherence tomography reports and fundus images were used as predictors in the dual-modal fusion network to diagnose glaucoma. The area under the receiver operation characteristic curve, accuracy, sensitivity, and specificity were measured to compare our method and other approaches.The accuracy of our dual-modal fusion network using both retinal nerve fiber layer thickness from optical coherence tomography reports and fundus images was 0.935 and we achieved a significant larger area under the receiver operation characteristic curve of our method with 0.968 (95% confidence interval, 0.937–0.999). For only using retinal nerve fiber layer thickness, we compared the area under the receiver operation characteristic curves between our network and other three approaches: 0.916 (95% confidence interval, 0.855, 0.977) with our optical coherence tomography Net; 0.841 (95% confidence interval, 0.749, 0.933) with Clock sectors division; 0.862 (95% confidence interval, 0.757, 0.968) with inferior, superior, nasal temporal sectors division and 0.886 (95% confidence interval, 0.815, 0.957) with optic disc sectors division. For only using fundus images, we compared the area under the receiver operation characteristic curves between our network and other two approaches: 0.867 (95% confidence interval: 0.781–0.952) with our Image Net; 0.774 (95% confidence interval: 0.670, 0.878) with ResNet50; 0.747 (95% confidence interval: 0.628, 0.866) with VGG16.Our dual-modal fusion network utilizing both retinal nerve fiber layer thickness from optical coherence tomography reports and fundus images can diagnose glaucoma with a much better performance than the current approaches based on optical coherence tomography only or fundus images only. [ABSTRACT FROM AUTHOR]

The interpretation of cardiac function using echocardiography requires a high level of diagnostic proficiency and years of experience. This study proposes a multi‐fusion residual attention U‐Net, MURAU‐Net, to construct automatic segmentation for evaluating cardiac function from echocardiographic video. MURAU‐Net has two benefits: (1) Multi‐fusion network to strengthen the links between spatial features. (2) Inter‐frame links can be established to augment the temporal coherence of sequential image data, thereby enhancing its continuity. To evaluate the effectiveness of the proposed method, we performed nine‐fold cross‐validation using CAMUS dataset. Among state‐of‐the‐art methods, MURAU‐Net achieves highly competitive score, for example, Dice similarity of 0.952 (ED phase) and 0.931 (ES phase) in LVEndo$$ {\mathrm{LV}}_{\mathrm{Endo}} $$, 0.966 (ED phase) and 0.957 (ES phase) in LVEpi$$ {\mathrm{LV}}_{\mathrm{Epi}} $$, and 0.901 (ED phase) and 0.917 (ES phase) in LA$$ \mathrm{LA} $$, respectively. It also achieved the Dice similarity of 0.9313 in the EchoNet‐Dynamic dataset for the overall left ventricle segmentation. In addition, we show MURAU‐Net can accurately segment multiclass cardiac ultrasound videos and output the animation of segmentation results using the original two‐chamber cardiac ultrasound dataset MUCO. [ABSTRACT FROM AUTHOR]

Currently, there is a great deal of interest in multimodal aspect-level sentiment classification using both textual and visual information, which changes the traditional use of only single-modal to identify sentiment polarity. Considering that existing methods could be strengthened in terms of classification accuracy, we conducted a study on aspect-level multimodal sentiment classification with the aim of exploring the interaction between textual and visual features. Specifically, we construct a multimodal aspect-level sentiment classification framework with multi-image gate and fusion networks called MFSC. MFSC consists of four parts, i.e., text feature extraction, visual feature extraction, text feature enhancement, and multi-feature fusion. Firstly, a bidirectional long short-term memory network is adopted to extract the initial text feature. Based on this, a text feature enhancement strategy is designed, which uses text memory network and adaptive weights to extract the final text features. Meanwhile, a multi-image gate method is proposed for fusing features from multiple images and filtering out irrelevant noise. Finally, a text-visual feature fusion method based on an attention mechanism is proposed to better improve the classification performance by capturing the association between text and images. Experimental results show that MFSC has advantages in classification accuracy and macro-F1. [ABSTRACT FROM AUTHOR]

In the past decade, supervised denoising models trained on large datasets have demonstrated impressive performance in image denoising due to their superior denoising effect. However, these models lack flexibility and exhibit varying degrees of degradation in denoising performance in practical applications, particularly when the noise distribution of the given noisy images does not match the training images. Our preliminary experiments suggest that even under ideal conditions, the denoised images obtained using these supervised denoising models (also known as preprocessed images) are already very similar to the ground truth images in terms of pixel intensities. Adding low-level noise to a preprocessed image can approximate the intensities of some pixels to their original values, but not all pixels. Based on this observation, we propose a novel two-stage approach to enhance the denoising effect of existing supervised denoisers using a low-noise injection strategy. In the first stage, we use a state-of-the-art supervised denoiser to denoise the given noisy image and obtain a preprocessed image. Then, we repeatedly inject different random low-level Gaussian noises to further improve certain pixels of the preprocessed image. The generated images are used as target images, and we obtain corresponding fine-tuned images within the framework of the unsupervised deep image prior (DIP) method by fully utilizing its flexibility. As a result, we obtain several denoised fine-tuned images that, respectively, approximate the ground truth image at specific pixels and complement each other. In the second stage, these fine-tuned images are fed to an unsupervised fusion network, which fully leverages the complementarity among the sample images to generate a fused image as the final denoised result. Experimental results demonstrate that the proposed method significantly improves the denoising effectiveness of synthetic noisy images, especially far surpassed the state-of-the-art methods in dealing with real noisy images. [ABSTRACT FROM AUTHOR]

Automatic pest detection from agricultural crops is a challenging agricultural application. It has been observed that many farmers lose crop yields mainly due to pests that spread rapidly. An intelligent pest recognition system (IPRS) is an innovative application that assists farmers in increasing agricultural productivity by accurately detecting pests that may cause crop damage. It allows one to identify the type of pest and apply the appropriate pesticides in a timely manner. There is a large deviation in deep neural network (DNN) performance on clean images and naturally degraded images. This article proposes a novel data augmentation techniques and feature fusion techniques to improve the accuracy as well as the desired robustness. The proposed visual regenerative fusion network (VRFNet) fuses the multiscale features from global feature extraction (GFE) network and the visual regeneration (VR) network to capture the semantic details for identifying pests from the distorted images which improves the accuracy. The proposed patch-based augmentation approach can effectively simulate the environment in which the leaves can partially conceal the insects and helps in improving the robustness. The proposed model achieves an accuracy of 99.12 and 68.34 on the publicly available D0 and IP102 datasets, respectively. These results clearly demonstrate that the proposed fusion method achieved a high success rate for distorted and occluded images captured from the agriculture farms. [ABSTRACT FROM AUTHOR]

Manually annotating liver tumor contours is a time-consuming and labor-intensive task for clinicians. Therefore, automated segmentation is urgently needed in clinical diagnosis. However, automatic segmentation methods face certain challenges due to heterogeneity, fuzzy boundaries, and irregularity of tumor tissue. In this paper, a novel deep learning-based approach with multi-scale-aware (MSA) module and twin-split attention (TSA) module is proposed for tumor segmentation. The MSA module can bridge the semantic gap and reduce the loss of detailed information. The TSA module can recalibrate the channel response of the feature map. Eventually, we can count tumors based on the segmentation results from a 3D perspective for cancer grading. Extensive experiments conducted on the LiTS2017 dataset show the effectiveness of the proposed method by achieving a Dice index of 85.97% and a Jaccard index of 81.56% over the state of the art. In addition, the proposed method also achieved a Dice index of 83.67% and a Jaccard index of 80.11% in 3Dircadb dataset verification, which further reflects its robustness and generalization ability. [ABSTRACT FROM AUTHOR]

The topic of innovation appears again and again in the government work report. The state and enterprises also pay more and more attention to product optimization and the output of innovative products, and more innovative and technological content has emerged from the products. At the same time, China’s foreign trade is transitioning from the traditional big country of foreign trade to the modern foreign trade power. In this paper, we combine natural language understanding and interactive image command mapping networks to improve the command optimization of digital products. The improved Graph-LSTM model is utilized to process the natural language instruction vectors, and the ICF network is utilized to take the environment image and user instructions as inputs to construct a fusion network in which the environment image and the natural language instructions can complement each other to complete the instruction mapping and to generate the location heat map for the execution of the instructions. Using principal component analysis, panel regression modeling, and other methods, we extracted and screened indicators of the influencing factors of export trade performance evaluation and constructed the performance evaluation index system. The synergistic enhancement of product instruction optimization and export trade performance is analyzed through empirical analysis and enhancement effect evaluation. The regression results of the selection equation are analyzed, and in the second column of the model, the coefficients of marketing cost and financial cost are 35.254 and -2.568, respectively, and the P-value is greater than 0.1, which can be analyzed in the next stage. In terms of the level of economic development, the difference between the comprehensive performance of export trade between Guangdong and Heilongjiang is 55.11, which is a large gap, and the optimization of product instructions has a greater impact on the performance of export trade in the eastern region.

Addressing the challenges of integrating industry and education, this research unveils a knowledge monitoring model aimed at refining the talent cultivation process, particularly in higher trade colleges and universities in China. The “2+1” model is developed in response to issues such as ineffective teaching practices, an abundance of practical courses without depth, and weak university-industry collaboration. Based on a comprehensive analysis of the talent cultivation demand and integration policies, this model employs a multi-scale integration network to foster a meaningful fusion of industry and education. Through case study evaluation, a strong positive correlation (0.727, significant at the 0.01 level) between overall satisfaction and the accomplishment of skills cultivation highlights the model’s effectiveness. This contribution not only advances our understanding of talent development in vocational education but also has significant implications for the broader application of industry-education collaborative models.

A brain–computer interface (BCI) is a computer-based system that allows for communication between the brain and the outer world, enabling users to interact with computers using neural activity. This brain signal is obtained from electroencephalogram (EEG) signals. A significant obstacle to the development of BCIs based on EEG is the classification of subject-independent motor imagery data since EEG data are very individualized. Deep learning techniques such as the convolutional neural network (CNN) have illustrated their influence on feature extraction to increase classification accuracy. In this paper, we present a multi-branch (five branches) 2D convolutional neural network that employs several hyperparameters for every branch. The proposed model achieved promising results for cross-subject classification and outperformed EEGNet, ShallowConvNet, DeepConvNet, MMCNN, and EEGNet_Fusion on three public datasets. Our proposed model, EEGNet Fusion V2, achieves 89.6% and 87.8% accuracy for the actual and imagined motor activity of the eegmmidb dataset and scores of 74.3% and 84.1% for the BCI IV-2a and IV-2b datasets, respectively. However, the proposed model has a bit higher computational cost, i.e., it takes around 3.5 times more computational time per sample than EEGNet_Fusion. [ABSTRACT FROM AUTHOR]

Clustering methods have been widely used in single-cell RNA-seq data for investigating tumor heterogeneity. Since traditional clustering methods fail to capture the high-dimension methods, deep clustering methods have drawn increasing attention these years due to their promising strengths on the task. However, existing methods consider either the attribute information of each cell or the structure information between different cells. In other words, they cannot sufficiently make use of all of this information simultaneously. To this end, we propose a novel single-cell deep fusion clustering model, which contains two modules, i.e. an attributed feature clustering module and a structure-attention feature clustering module. More concretely, two elegantly designed autoencoders are built to handle both features regardless of their data types. Experiments have demonstrated the validity of the proposed approach, showing that it is efficient to fuse attributes, structure, and attention information on single-cell RNA-seq data. This work will be further beneficial for investigating cell subpopulations and tumor microenvironment. The Python implementation of our work is now freely available at https://github.com/DayuHuu/scDFC. [ABSTRACT FROM AUTHOR]

Diabetic retinopathy (DR) is a lasting diabetic condition that causes vision damage, and if left untreated, the progression of DR may result in permanent blindness. Therefore, it is obligatory to identify the pathological changes in the retina to detect the severity level of DR. Manual observation of retinal disorders makes the detection process more intricate, tedious, and error prone due to the prevalence of subtle pathologies in the retina. Deep learning-enabled optimization techniques have recently made promising progress in DR detection and classification tasks. Therefore, the current work suggests an enhanced classification approach based on metaheuristic optimization for severity grading in fundus images. Initially, a pre-processing step eradicates noise and enhances the image contrast. Following the optic disc segmentation using dilated convolution-based spatial attention U-Net, blood vessels are segmented using an entropy-based hybrid technique. Subsequently, the lesion area is segmented using the attention-based fusion U-Net model with a weighted focal loss. Then, features are extracted and fused using a two-layer fusion network. Based on the fused features, the DR stages are classified with refined deep residual network (RDRN) integrating a squeeze–excitation module and tunicate swarm spider monkey optimization (TSSMO) algorithm. The fundus images from the DIARETDB1 and DIARETDB0 datasets were used to evaluate the performance of the suggested method using different measures. The comparative performance reveals the efficacy of the developed method in classifying the DR severity stages. [ABSTRACT FROM AUTHOR]

Sarcasm is a sophisticated figurative language that is prevalent on social media platforms. Automatic sarcasm detection is significant for understanding the real sentiment tendencies of users. Traditional approaches mostly focus on content features by using lexicon, n-gram, and pragmatic feature-based models. However, these methods ignore the diverse contextual clues that could provide more evidence of the sarcastic nature of sentences. In this work, we propose a Contextual Sarcasm Detection Model (CSDM) by modeling enhanced semantic representations with user profiling and forum topic information, where context-aware attention and a user-forum fusion network are used to obtain diverse representations from distinct aspects. In particular, we employ a Bi-LSTM encoder with context-aware attention to obtain a refined comment representation by capturing sentence composition information and the corresponding context situations. Then, we employ a user-forum fusion network to obtain the comprehensive context representation by capturing the corresponding sarcastic tendencies of the user and the background knowledge about the comments. Our proposed method achieves values of 0.69, 0.70, and 0.83 in terms of accuracy on the Main balanced, Pol balanced and Pol imbalanced datasets, respectively. The experimental results on a large Reddit corpus, SARC, demonstrate that our proposed method achieves a significant performance improvement over state-of-art textual sarcasm detection methods. [ABSTRACT FROM AUTHOR]

Code summarization is the process of automatically generating brief and informative summaries of source code to aid in software comprehension and maintenance. In this paper, we propose a novel model called READSUM, REtrieval-augmented ADaptive transformer for source code SUMmarization, that combines both abstractive and extractive approaches. Our proposed model generates code summaries in an abstractive manner, taking into account both the structural and sequential information of the input code, while also utilizing an extractive approach that leverages a retrieved summary of similar code to increase the frequency of important keywords. To effectively blend the original code and the retrieved similar code at the embedding layer stage, we obtain the augmented representation of the original code and the retrieved code through multi-head self-attention. In addition, we develop a self-attention network that adaptively learns the structural and sequential information for the representations in the encoder stage. Furthermore, we design a fusion network to capture the relation between the original code and the retrieved summary at the decoder stage. The fusion network effectively guides summary generation based on the retrieved summary. Finally, READSUM extracts important keywords using an extractive approach and generates high-quality summaries using an abstractive approach that considers both the structural and sequential information of the source code. We demonstrate the superiority of READSUM through various experiments and an ablation study. Additionally, we perform a human evaluation to assess the quality of the generated summary.

Copyright of Journal of South-Central Minzu University (Natural Science Edition) is the property of Journal of South-Central Minzu University (Natural Science Edition) Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The application of dynamic gestures is extensive in the field of automated intelligent manufacturing. Due to the temporal and spatial complexity of dynamic gesture data, traditional machine learning algorithms struggle to extract accurate gesture features. Existing dynamic gesture recognition algorithms have complex network designs, high parameter counts, and inadequate gesture feature extraction. In order to solve the problems of low accuracy and high computational complexity in current dynamic gesture recognition, a network model based on the MetaFormer architecture and an attention mechanism was designed. The proposed network fuses a CNN (convolutional neural network) and Transformer model by embedding spatial attention convolution and temporal attention convolution into the Transformer model. Specifically, the token mixer in the MetaFormer block is replaced by the Spatial Attention Convolution Block and Temporal Attention Convolution Block to obtain the Spatial Attention Former Block and Temporal Attention Former Block. Firstly, each frame of the input image is quickly down-sampled by the PoolFormer block and then input to the Spatial Attention Former Block to learn spatial feature information. Then, the spatial feature maps learned from each frame are concatenated along the channel dimension and input to the Temporal Attention Former Block to learn the temporal feature information of the gesture action. Finally, the learned overall feature information is classified to obtain the category of dynamic gestures. The model achieves an average recognition accuracy of 96.72% and 92.16% on two publicly available datasets, Jester and NVGesture, respectively. [ABSTRACT FROM AUTHOR]

The classification of White blood cells (WBCs) plays an important role. However, the traditional method of blood smear analysis is laborious. This paper presented a classification method of WBCs based on hyperspectral images and Deep learning (DL). The U-net network was proposed to extract spectral features of WBCs region of interest (ROI) under the pseudo-color images with specific hyperspectral images (420.8, 557.2 and 667.4 nm). For spectral features and the pseudo-colour images of hyperspectral data, attention-aided one and two-dimensional convolutional neural networks were applied to establish WBCs classification models, respectively. The overall average accuracy can reach 94.20% and 92.60%, respectively. A fusion network was constructed to make full use of the spectral and image spatial features, and its classification accuracy reached 96.20%. In terms of overall average accuracy, the fusion network model was the optimal, but for individual types of WBCs, each network had its own unique advantages. [ABSTRACT FROM AUTHOR]

Copyright of Chinese Medical Equipment Journal is the property of Chinese Medical Equipment Journal Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Multimodal sentiment analysis (MSA) has become a popular field of research in recent years. The aim is to combine the three modalities of text, video, and audio to obtain comprehensive emotional information. However, current research often treats these three modalities equally, downplaying the crucial role of text modality in MSA and ignoring the redundant information generated during multimodal fusion. To address these problems, we propose the Text-Centric Hierarchical Fusion Network (TCHFN), employing a hierarchical fusion strategy. In this framework, low-level fusion involves cross-modal interactions between pairs of modalities, while high-level fusion extends these interactions to involve all three modalities. Through the design of the Cross-modal Reinforced Transformer (CRT), we achieve cross-modal enhancement of the target modality, facilitating a nuanced fusion process with text serving as its core. Additionally, we design Text-Centric Contrastive Learning (TCCL) to align non-text modalities with the text modality, emphasising the central role of text in the fusion process. After fusion, a multimodal fusion output gate is employed to mitigate redundant information within the multimodal fusion representation, which is subsequently processed by a linear layer for prediction. Simultaneously, to fully leverage limited labelled datasets, we introduced knowledge distillation. This approach involves preserving the model parameters that yield the best performance during training as a teacher model. The teacher model aids in capturing rich emotional information, enabling the model to transcend local optima and discover more optimal parameters, thereby enhancing overall model performance. Extensive experiments on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate the superiority of our model over state-of-the-art methods in MSA tasks. [ABSTRACT FROM AUTHOR]

In the research field of few-shot learning, the main difference between image-based and video-based is the additional temporal dimension. In recent years, some works have used the Transformer to deal with frames, then get the attention feature and the enhanced prototype, and the results are competitive. However, some video frames may relate little to the action, and only using single frame-level or segment-level features may not mine enough information. We address these problems sequentially through an end-to-end method named "Task-Specific Alignment and Multiple-level Transformer Network (TSA-MLT)". The first module (TSA) aims to filter the action-irrelevant frames for action duration alignment. Affine Transformation for frame sequence in the time dimension is used for linear sampling. The second module (MLT) focuses on the Multiple-level feature of the support prototype and query sample to mine more information for the alignment, which operates on different level features. We adopt a fusion loss according to a fusion distance that fuses the L2 sequence distance, which focuses on temporal order alignment, and the Optimal Transport distance, which focuses on measuring the gap between the appearance and semantics of the videos. Extensive experiments show our method achieves state-of-the-art results on the HMDB51 and UCF101 datasets and a competitive result on the benchmark of Kinetics and something-2-something V2 datasets. Our code will be available at the URL: https://github.com/cofly2014/tsa-mlt.git. [ABSTRACT FROM AUTHOR]

A brain–computer interface (BCI) is a computer-based system that allows for communication between the brain and the outer world, enabling users to interact with computers using neural activity. This brain signal is obtained from electroencephalogram (EEG) signals. A significant obstacle to the development of BCIs based on EEG is the classification of subject-independent motor imagery data since EEG data are very individualized. Deep learning techniques such as the convolutional neural network (CNN) have illustrated their influence on feature extraction to increase classification accuracy. In this paper, we present a multi-branch (five branches) 2D convolutional neural network that employs several hyperparameters for every branch. The proposed model achieved promising results for cross-subject classification and outperformed EEGNet, ShallowConvNet, DeepConvNet, MMCNN, and EEGNet_Fusion on three public datasets. Our proposed model, EEGNet Fusion V2, achieves 89.6% and 87.8% accuracy for the actual and imagined motor activity of the eegmmidb dataset and scores of 74.3% and 84.1% for the BCI IV-2a and IV-2b datasets, respectively. However, the proposed model has a bit higher computational cost, i.e., it takes around 3.5 times more computational time per sample than EEGNet_Fusion.

The purpose is to minimize subjective errors in the manual analysis of volleyball game videos and improve the traditional Human Behavior Recognition (HBR) algorithms’ excessive calculation, high hardware requirements, and poor long-stream video modeling ability. Firstly, this paper expounds on the relevant theories. Secondly, a fusion Convolutional Neural Network (CNN)-based HBR model is implemented. It combines the two-stream CNN (TSCNN), Three-Dimensional (3D) CNN, and Long Short-Term Memory (LSTM) and gives full play to the LSTM’s long-term Dynamic Information Extraction (DIE) ability. Finally, the public dataset is selected to verify the model’s volleyball-game-video-oriented HBR performance. Here are the experimental results. (1) The optimal key parameters of the proposed fusion-CNN-based HBR model are determined as follows: the number of video segments is three, the average method is used for feature fusion, and then the HBR accuracy is the highest when the fusion ratio of spatial feature map and temporal feature map is 4:6, and the learning rate is 0.0014. (2) The average recognition accuracy of the proposed fusion-CNN-based HBR model on three different datasets is 4%, 2.7%, and 3% higher than other popular networks, respectively. The improvement effect of the model is remarkable, and it is suitable for studying Human Behavior Analysis (HBA) in volleyball match videos. Finally, the proposed HBR model can provide more accurate results for volleyball videos’ HBR, which is significant in promoting the rapid development of volleyball sports. The proposed model can classify and label videos and understand and describe video behaviors to simplify video processing procedures and save social resources.

Predicting the wafer material removal rate (MRR) is an important step in semiconductor manufacturing for total quality control. This work proposes a deep learning model called a fusion network to predict the MRR, in which we consider separating features into shallow and deep features and use the characteristics of deep learning to perform a fusion of these two kinds of features. In the proposed model, the deep features go through a sequence of nonlinear transformations and the goal is to learn the complex interactions among the features to obtain the deep feature embeddings. Additionally, the proposed method is flexible and can incorporate domain knowledge into the model by encoding the knowledge as shallow features. Once the learning of deep features is completed, the proposed model uses the shallow features and the learned deep feature embeddings to obtain new features for the subsequent layers. This work performs experiments on a dataset from the 2016 Prognostics and Health Management Data Challenge. The experimental results show that the proposed model outperforms the competition winner and three ensemble learning methods. The proposed method is a single model, whereas the comparison methods are ensemble models. Besides the experimental results, we conduct extensive experiments to analyze the proposed method. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved great performance on blind Image Quality Assessment (IQA), but it is still challenging for using one network to accurately predict the quality of images with different distortions. In this paper, a Distortion-Aware Convolutional Neural Network (DACNN) is proposed for blind IQA, which works effectively for not only synthetically distorted images but also authentically distorted images. The proposed DACNN consists of a distortion aware module, a distortion fusion module, and a quality prediction module. In the distortion aware module, a Siamese network-based pretraining strategy is proposed to design a synthetic distortion-aware network for full learning the synthetic distortions, and an authentic distortion-aware network is used for extracting the authentic distortions. To efficiently fuse the learned distortion features, and make the network pay more attention to the essential features, a weight-adaptive fusion network is proposed to adaptively adjust the weight of each distortion. Finally, the quality prediction module is adopted to map the fused features to a quality score. Extensive experiments on four authentic IQA databases and four synthetic IQA databases have proved the effectiveness of the proposed DACNN. [ABSTRACT FROM AUTHOR]

Diabetic Retinopathy (DR), one of the most progressive sight-threatening diseases caused by the long-term diabetic condition, can lead to vision impairment and blindness later. Early diagnosis and timely treatment help control and avert DR from its progression. However, manual grading is exceptionally challenging and arduous due to the complex anatomical features in the retina. Therefore, developing an automated diagnostic method for screening DR is obligatory. This paper proposes a deep learning-enabled optimized feature selection approach to classify the stage of DR severity in the fundus image. At first, a pre-processing phase eradicates the noise and improves the contrast in the retinal fundus image. Then, blood vessel segmentation is performed using the Coherence Enhancing Energy Based Regularized Level Set Evolution method in the green channel fundus image. Subsequently, the optic disk is segmented with Canny Anisotropic Diffusion filter and morphological transformations. Next, the candidate lesion region is detected using an Attention-based Fusion Network (AFU-Net). Then, shape and texture features are extracted, and then, the optimal subset of features is selected using the Improved Harris Hawk Optimization algorithm. Finally, a deep Convolutional Neural Network classifies the DR stages, and the model weight is updated using the same algorithm. The proposed method achieved superior performance in two benchmark public datasets compared with the existing state-of-the-art methods using F1-score, accuracy, sensitivity, and specificity measures. [ABSTRACT FROM AUTHOR]

Nextpoint‐of‐interest (POI) recommendation has received widespread attention in recent years due to its superiority of recommending where users will go to next. However, there exist two limitations in many recommendation methods: (1) the hardness of modeling users' short‐term preferences adaptively based on input sequence; (2) the efficient learning of the joint information between users' long‐ and short‐term preferences. To this end, we propose an attention mechanism and adaptive convolution actuated fusion network (AMACF) innovatively, which optimizes forecast effectiveness of user preference. To better model some contextual information such as category, temporal, we utilize long‐ and short‐term memory network to learn contextual features of POIs in historical check‐ins and embed self‐attention mechanism to capture users' preference in the long‐term module. In the short‐term module, for capturing the complicated interest, the adaptive convolution network (Ada‐CN) is novelly proposed, which applies the attention mechanism to aggregate multiple parallel convolution kernels selectively and adjust to short‐term preference according to users' continuously updated check‐ins. Furthermore, an attention‐based fusion mechanism is designed to combine long‐ and short‐term preferences, where contributions to the next POI of different users are evaluated sufficiently. Experimental results on two real‐world data sets indicates that AMACF outperforms baseline methods for next POI recommendation. [ABSTRACT FROM AUTHOR]

Pigmented skin disease is caused by abnormal melanocyte and melanin production, which can be induced by genetic and environmental factors. It is also common among the various types of skin diseases. The timely and accurate diagnosis of pigmented skin disease is important for reducing mortality. Patients with pigmented dermatosis are generally diagnosed by a dermatologist through dermatoscopy. However, due to the current shortage of experts, this approach cannot meet the needs of the population, so a computer-aided system would help to diagnose skin lesions in remote areas containing insufficient experts. This paper proposes an algorithm based on a fusion network for the detection of pigmented skin disease. First, we preprocess the images in the acquired dataset, and then we perform image flipping and image style transfer to augment the images to alleviate the imbalance between the various categories in the dataset. Finally, two feature-level fusion optimization schemes based on deep features are compared with a classifier-level fusion scheme based on a classification layer to effectively determine the best fusion strategy for satisfying the pigmented skin disease detection requirements. Gradient-weighted Class Activation Mapping (Grad_CAM) and Grad_CAM++ are used for visualization purposes to verify the effectiveness of the proposed fusion network. The results show that compared with those of the traditional detection algorithm for pigmented skin disease, the accuracy and Area Under Curve (AUC) of the method in this paper reach 92.1 and 95.3%, respectively. The evaluation indices are greatly improved, proving the adaptability and accuracy of the proposed method. The proposed method can assist clinicians in screening and diagnosing pigmented skin disease and is suitable for real-world applications.

This work focuses on the task of elderly activity recognition, which is a challenging task due to the existence of individual actions and human-object interactions in elderly activities. Thus, we attempt to effectively aggregate the discriminative information of actions and interactions from both RGB videos and skeleton sequences by attentively fusing multi-modal features. Recently, some nonlinear multi-modal fusion approaches are proposed by utilizing nonlinear attention mechanism that is extended from Squeeze-and-Excitation Networks (SENet). Inspired by this, we propose a novel Expansion-Squeeze-Excitation Fusion Network (ESE-FN) to effectively address the problem of elderly activity recognition, which learns modal and channel-wise Expansion-Squeeze-Excitation (ESE) attentions for attentively fusing the multi-modal features in the modal and channel-wise ways. Specifically, ESE-FN firstly implements the modal-wise fusion with the Modal-wise ESE Attention (M-ESEA) to aggregate discriminative information in modal-wise way, and then implements the channel-wise fusion with the Channel-wise ESE Attention (C-ESEA) to aggregate the multi-channel discriminative information in channel-wise way (referring to). Furthermore, we design a new Multi-modal Loss (ML) to keep the consistency between the single-modal features and the fused multi-modal features by adding the penalty of difference between the minimum prediction losses on single modalities and the prediction loss on the fused modality. Finally, we conduct experiments on a largest-scale elderly activity dataset, i.e., ETRI-Activity3D (including 110,000+ videos, and 50+ categories), to demonstrate that the proposed ESE-FN achieves the best accuracy compared with the state-of-the-art methods. In addition, more extensive experimental results show that the proposed ESE-FN is also comparable to the other methods in terms of normal action recognition task. [ABSTRACT FROM AUTHOR]

Optical coherence tomography (OCT) is a new type of tomography that has experienced rapid development and potential in recent years. It is playing an increasingly important role in retinopathy diagnoses. At present, due to the uneven distributions of medical resources in various regions, the uneven proficiency levels of doctors in grassroots and remote areas, and the development needs of rare disease diagnosis and precision medicine, artificial intelligence technology based on deep learning can provide fast, accurate, and effective solutions for the recognition and diagnosis of retinal OCT images. To prevent vision damage and blindness caused by the delayed discovery of retinopathy, a fusion network (FN)-based retinal OCT classification algorithm (FN-OCT) is proposed in this paper to improve upon the adaptability and accuracy of traditional classification algorithms. The InceptionV3, Inception-ResNet, and Xception deep learning algorithms are used as base classifiers, a convolutional block attention mechanism (CBAM) is added after each base classifier, and three different fusion strategies are used to merge the prediction results of the base classifiers to output the final prediction results (choroidal neovascularization (CNV), diabetic macular oedema (DME), drusen, normal). The results show that in a classification problem involving the UCSD common retinal OCT dataset (108,312 OCT images from 4,686 patients), compared with that of the InceptionV3 network model, the prediction accuracy of FN-OCT is improved by 5.3% (accuracy = 98.7%, area under the curve (AUC) = 99.1%). The predictive accuracy and AUC achieved on an external dataset for the classification of retinal OCT diseases are 92 and 94.5%, respectively, and gradient-weighted class activation mapping (Grad-CAM) is used as a visualization tool to verify the effectiveness of the proposed FNs. This finding indicates that the developed fusion algorithm can significantly improve the performance of classifiers while providing a powerful tool and theoretical support for assisting with the diagnosis of retinal OCT. [ABSTRACT FROM AUTHOR]

RGB-D salient object detection task still encounters three challenges: (1) how to effectively integrate superior information from different modalities, (2) how to effectively mine common information of features at different levels, and (3) how to detect salient objects in complex scenes, such as complex backgrounds, low-quality depth maps, small targets, and high foreground-background similarity. To address the above challenges, we propose a novel Perceptual Localization and Focus Refinement Network, termed PLFRNet, based on the mechanism of human visual capture of salient objects in images. The network includes three key components: an encoder, a Perceptual Localization Module (PLM), and a Focus-Refinement Decoder (FRD). Specifically, we first adopt a two-stream asymmetric Pyramid Visual Transformer as the encoder to extract RGB and depth features. Then, we develop the PLM under the guidance of a Perceptual Localization Unit (PLU) delicately designed. This module can mine the common information of features at different levels and integrate the advantageous information from multiple modalities to localize salient objects. Finally, we propose the FRD focusing on detailed information guided by the attention mechanism. Furthermore, it further refines the located objects by gradually interacting with low-level features to achieve salient object detection. Extensive experimental results show that this method achieves state-of-the-art performance compared with 13 RGB-D models on 6 public datasets. The codes are released at https://github.com/hjy0518/PLFRNet/. [ABSTRACT FROM AUTHOR]

Optical coherence tomography (OCT) is a new type of tomography that has experienced rapid development and potential in recent years. It is playing an increasingly important role in retinopathy diagnoses. At present, due to the uneven distributions of medical resources in various regions, the uneven proficiency levels of doctors in grassroots and remote areas, and the development needs of rare disease diagnosis and precision medicine, artificial intelligence technology based on deep learning can provide fast, accurate, and effective solutions for the recognition and diagnosis of retinal OCT images. To prevent vision damage and blindness caused by the delayed discovery of retinopathy, a fusion network (FN)-based retinal OCT classification algorithm (FN-OCT) is proposed in this paper to improve upon the adaptability and accuracy of traditional classification algorithms. The InceptionV3, Inception-ResNet, and Xception deep learning algorithms are used as base classifiers, a convolutional block attention mechanism (CBAM) is added after each base classifier, and three different fusion strategies are used to merge the prediction results of the base classifiers to output the final prediction results (choroidal neovascularization (CNV), diabetic macular oedema (DME), drusen, normal). The results show that in a classification problem involving the UCSD common retinal OCT dataset (108,312 OCT images from 4,686 patients), compared with that of the InceptionV3 network model, the prediction accuracy of FN-OCT is improved by 5.3% (accuracy = 98.7%, area under the curve (AUC) = 99.1%). The predictive accuracy and AUC achieved on an external dataset for the classification of retinal OCT diseases are 92 and 94.5%, respectively, and gradient-weighted class activation mapping (Grad-CAM) is used as a visualization tool to verify the effectiveness of the proposed FNs. This finding indicates that the developed fusion algorithm can significantly improve the performance of classifiers while providing a powerful tool and theoretical support for assisting with the diagnosis of retinal OCT.

Sarcasm is a sophisticated figurative language that is prevalent on social media platforms. Automatic sarcasm detection is significant for understanding the real sentiment tendencies of users. Traditional approaches mostly focus on content features by using lexicon, n-gram, and pragmatic feature-based models. However, these methods ignore the diverse contextual clues that could provide more evidence of the sarcastic nature of sentences. In this work, we propose a Contextual Sarcasm Detection Model (CSDM) by modeling enhanced semantic representations with user profiling and forum topic information, where context-aware attention and a user-forum fusion network are used to obtain diverse representations from distinct aspects. In particular, we employ a Bi-LSTM encoder with context-aware attention to obtain a refined comment representation by capturing sentence composition information and the corresponding context situations. Then, we employ a user-forum fusion network to obtain the comprehensive context representation by capturing the corresponding sarcastic tendencies of the user and the background knowledge about the comments. Our proposed method achieves values of 0.69, 0.70, and 0.83 in terms of accuracy on the Main balanced, Pol balanced and Pol imbalanced datasets, respectively. The experimental results on a large Reddit corpus, SARC, demonstrate that our proposed method achieves a significant performance improvement over state-of-art textual sarcasm detection methods.

The application of dynamic gestures is extensive in the field of automated intelligent manufacturing. Due to the temporal and spatial complexity of dynamic gesture data, traditional machine learning algorithms struggle to extract accurate gesture features. Existing dynamic gesture recognition algorithms have complex network designs, high parameter counts, and inadequate gesture feature extraction. In order to solve the problems of low accuracy and high computational complexity in current dynamic gesture recognition, a network model based on the MetaFormer architecture and an attention mechanism was designed. The proposed network fuses a CNN (convolutional neural network) and Transformer model by embedding spatial attention convolution and temporal attention convolution into the Transformer model. Specifically, the token mixer in the MetaFormer block is replaced by the Spatial Attention Convolution Block and Temporal Attention Convolution Block to obtain the Spatial Attention Former Block and Temporal Attention Former Block. Firstly, each frame of the input image is quickly down-sampled by the PoolFormer block and then input to the Spatial Attention Former Block to learn spatial feature information. Then, the spatial feature maps learned from each frame are concatenated along the channel dimension and input to the Temporal Attention Former Block to learn the temporal feature information of the gesture action. Finally, the learned overall feature information is classified to obtain the category of dynamic gestures. The model achieves an average recognition accuracy of 96.72% and 92.16% on two publicly available datasets, Jester and NVGesture, respectively.

Deep learning (DL)-based modulation recognition methods of underwater acoustic communication signals are mostly applied to a single hydrophone reception scenario. In this paper, we propose a novel end-to-end multihydrophone fusion network (MHFNet) for multisensory reception scenarios. MHFNet consists of a feature extraction module and a fusion module. The feature extraction module extracts the features of the signals received by the multiple hydrophones. Then, through the neural network, the fusion module fuses and classifies the features of the multiple signals. MHFNet takes full advantage of neural networks and multihydrophone reception to effectively fuse signal features for realizing improved modulation recognition performance. Experimental results on simulation and practical data show that MHFNet is superior to other fusion methods. The classification accuracy is improved by about 16%. [ABSTRACT FROM AUTHOR]

Scene recognition is a kind of image recognition problems which is aimed at predicting the category of the place at which the image is taken. In this paper, a new scene recognition method using the convolutional neural network (CNN) is proposed. The proposed method is based on the fusion of the object and the scene information in the given image and the CNN framework is named as FOS (fusion of object and scene) Net. To combine the object and the scene information effectively, a new fusion framework named CCG (correlative context gating) is proposed. In addition, a new loss named scene coherence loss (SCL) is developed to train the FOSNet and to improve the scene recognition performance. The proposed SCL is based on the idea that the scene class does not change all over the image. The proposed FOSNet was experimented with three most popular scene recognition datasets, and their state-of-the-art performance is obtained in two sets: 60.14% on Places 2 and 90.30% on MIT indoor 67. The second highest performance of 77.28% is obtained on SUN 397.

The Convolution Neural Network (CNN) is widely used in the super-resolution task of depth map. However, the ones with simple architecture and high efficiency generally lack accuracy, while the ones with high accuracy demonstrate low efficiency and training difficulties due to their over-deep level and complex architecture. We propose a depth map super-resolution fusion framework. This framework fuses multiple Progressive Convolution Neural Networks (PCNNs) with different architectures by a pixel-wise Partial Differential Equation (PDE). Each individual PCNN uses progressive learning and deep supervising to construct a mapping from low resolution space to high resolution space. The PDE model automatically classifies and processes the high-resolution depth maps with different feature output by fusing multiple PCNNs. The fusion term in PDE is used to preserve or integrate the complementary features of the depth maps, and the divergence term in PDE is used to remove noise to improve the spatial accuracy and visual effect of the final output depth map. This method enables simple structured Neural Networks with high accuracy, high efficiency and relatively simple network training for depth map super-resolution.

We propose a two‐stage convolutional neural network (CNN) architecture for robust recognition of hand gestures, called HGR‐Net, where the first stage performs accurate semantic segmentation to determine hand regions, and the second stage identifies the gesture. The segmentation stage architecture is based on the combination of fully convolutional residual network and atrous spatial pyramid pooling. Although the segmentation sub‐network is trained without depth information, it is particularly robust against challenges such as illumination variations and complex backgrounds. The recognition stage deploys a two‐stream CNN, which fuses the information from the red–green–blue and segmented images by combining their deep representations in a fully connected layer before classification. Extensive experiments on public datasets show that our architecture achieves almost as good as state‐of‐the‐art performance in segmentation and recognition of static hand gestures, at a fraction of training time, run time, and model size. Our method can operate at an average of 23 ms per frame.

The biomimetic electronic nose (e-nose) technology is a novel technology used for the identification and monitoring of complex gas molecules, and it is gaining significance in this field. However, due to the complexity and multiplicity of gas mixtures, the accuracy of electronic noses in predicting gas concentrations using traditional regression algorithms is not ideal. This paper presents a solution to the difficulty by introducing a fusion network model that utilizes a transformer-based multikernel feature fusion (TMKFF) module combined with a 1DCNN_LSTM network to enhance the accuracy of regression prediction for gas mixture concentrations using a portable electronic nose. The experimental findings demonstrate that the regression prediction performance of the fusion network is significantly superior to that of single models such as convolutional neural network (CNN) and long short-term memory (LSTM). The present study demonstrates the efficacy of our fusion network model in accurately predicting the concentrations of multiple target gases, such as SO 2 , NO 2 , and CO, in a gas mixture. Specifically, our algorithm exhibits substantial benefits in enhancing the prediction performance of low-concentration SO 2 gas, which is a noteworthy achievement. The determination coefficient ( R 2 ) values of 93, 98, and 99% correspondingly demonstrate that the model is very capable of explaining the variation in the concentration of the target gases. The root-mean-square errors (RMSE) are 0.0760, 0.0711, and 3.3825, respectively, while the mean absolute errors (MAE) are 0.0507, 0.0549, and 2.5874, respectively. These results indicate that the model has relatively small prediction errors. The method we have developed holds significant potential for practical applications in detecting atmospheric pollution detection and other molecular detection areas in complex environments.