Search

Video captioning is designed to generate natural language descriptions based on video content. Traditional methods extract visual features and interactive relationship features between objects, but the problem of video feature isolation and semantic hierarchy is ignored. This paper proposes a Multi-Level Video Captioning Method based on semantic space (S-MLM) to solve the above problems. S-MLM extracts different levels of visual elements and visual relationships, and the visual information of different levels is aggregated layer by layer to complete the generation of low-level to high-level visual features. The multi-level structure semantic graph is constructed from the semantic point of view. It does not rely on external knowledge bases, and uses its own information as guidance to enhance feature representation and improve semantic understanding. We conduct experiments on MSVD and MSR-VTT datasets, and the experimental results show that the performance of video captioning is further improved. [ABSTRACT FROM AUTHOR]

With the rapid development of software-defined networking (SDN), SDN-Based multi-level flow table architectures are always employed to address issues such as QoS, security policies, and matching efficiency. The rise of semantic communication has also sparked researchers’ interest in semantic information and its utilization. Many studies on semantic representation and semantic summarization have emerged. This study takes a new perspective to reduce the number of table entries by utilizing the semantic relationships implied in the topic tree in the topic-based pub/sub systems and introduces the concept of semantic aggregation. Semantic aggregation of table entries can work with multi-level flow table architecture to reduce the number of table entries while ensuring the correct delivery of streams. We propose a semantic-based table entry encoding algorithm to implement our idea and conduct several experiments to examine its performance. The experiment results demonstrate that our algorithm can achieve high space and efficiency optimization rate in a short encoding time.

Convolutional neural networks (CNN) have been widely used in image scene classification and have achieved remarkable progress. However, because the extracted deep features can neither focus on the local semantics of the image, nor capture the spatial morphological variation of the image, it is not appropriate to directly use CNN to generate the distinguishable feature representations. To relieve this limitation, a global-local feature adaptive fusion (GLFAF) network is proposed. The GLFAF framework extracts multi-scale and multi-level features by using a designed CNN. Then, to leverage the complementary advantages of the multi-scale and multi-level features, we design a global feature aggregate module to discover global attention features and further learn the multiple deep dependencies of spatial scale variations among these global features. Meanwhile, a local feature aggregate module is designed to aggregate the multi-scale and multi-level features. Specially, multi-level features at the same scale are fused based on channel attention, and then spatial fused features at different scales are aggregated based on channel dependence. Moreover, spatial contextual attention is designed to refine spatial features across scales and different fisher vector layers are designed to learn semantic aggregation among spatial features. Subsequently, two different feature adaptive fusion modules are introduced to explore the complementary associations of global and local aggregate features, which can obtain comprehensive and differentiated image scene presentation. Finally, a large number of experiments on real scene datasets coming from three different fields show that the proposed GLFAF approach can more accurately realize scene classification than other state-of-the-art models. [ABSTRACT FROM AUTHOR]

Image-text retrieval is a basic task in visual-language domain,which aims at mining the relationships between different modalities.However,most existing approaches rely heavily on associating specific regions of an image with each word in a sentence with similar semantics and underappreciate the significance of multi-granular information in images,resulting in irrelevant matches between the two modalities and semantically ambiguous embedding.Generally,an image contains object-level,action-le-vel,relationship-level or even scene-level information that is not explicitly labeled.Therefore,it is challenging to align complex visual information with ambiguous descriptions.To tackle this issue,this paper proposes a granularity aware and semantic aggregating(GASA) network to obtain multi-visual representations and narrow the cross-modal gap.Specifically,the granularity-aware feature selection module selects copious multi-granularity information of images and conducts a multi-scale fusion,guided by an adaptive gated fusion mechanism and a pyramid structure.The semantic aggregation module clusters the multi-granularity information from visual and textual clues in a shared space to obtain the residual representations.Experiments are conducted on two benchmark datasets,and the results show our model outperforms the state-of-the-arts by over 2% on R@1 of MSCOCO 1k.Besides,our model outperforms the state-of-the-art by 4.1% in terms of Flickr30k on R@Sum.

In this paper, a lightweight, high-definition mapping method is proposed for autonomous driving to address the drawbacks of traditional mapping methods, such as high cost, low efficiency, and slow update frequency. The proposed method is based on multi-source data fusion perception and involves generating local semantic maps (LSMs) using multi-sensor fusion on a vehicle and uploading multiple LSMs of the same road section, obtained through crowdsourcing, to a cloud server. An improved, two-stage semantic alignment algorithm, based on the semantic generalized iterative closest point (GICP), was then used to optimize the multi-trajectories pose on the cloud. Finally, an improved density clustering algorithm was proposed to instantiate the aligned semantic elements and generate vector semantic maps to improve mapping efficiency. Experimental results demonstrated the accuracy of the proposed method, with a horizontal error within 20 cm, a vertical error within 50 cm, and an average map size of 40 Kb/Km. The proposed method meets the requirements of being high definition, low cost, lightweight, robust, and up-to-date for autonomous driving. [ABSTRACT FROM AUTHOR]

In this paper, a lightweight, high-definition mapping method is proposed for autonomous driving to address the drawbacks of traditional mapping methods, such as high cost, low efficiency, and slow update frequency. The proposed method is based on multi-source data fusion perception and involves generating local semantic maps (LSMs) using multi-sensor fusion on a vehicle and uploading multiple LSMs of the same road section, obtained through crowdsourcing, to a cloud server. An improved, two-stage semantic alignment algorithm, based on the semantic generalized iterative closest point (GICP), was then used to optimize the multi-trajectories pose on the cloud. Finally, an improved density clustering algorithm was proposed to instantiate the aligned semantic elements and generate vector semantic maps to improve mapping efficiency. Experimental results demonstrated the accuracy of the proposed method, with a horizontal error within 20 cm, a vertical error within 50 cm, and an average map size of 40 Kb/Km. The proposed method meets the requirements of being high definition, low cost, lightweight, robust, and up-to-date for autonomous driving.

Numerous knowledge bases have been published on the web, and there are serious heterogeneous problems among them. Unifying these knowledge bases at the semantic level can better promote the development of the Linked Data Project. Various effective methods, the mainstream one of which is the iterative entity alignment algorithm based on TransE (Translation-based Embedding, an efficient knowledge graph embedding representation algorithm), have been put forward to solve the heterogeneous problems among knowledge bases. Although the TransE-based iterative entity alignment algorithm can shorten the distance between two entities with the same semantics, it has low accuracy because it ignores the importance of semantic aggregation generated by many attributes of entities in the entity alignment process. To solve this problem, a novel entity alignment algorithm based on semantic aggregation and attribute attention, named EASA, is proposed in this paper. On the one hand, semantic aggregation of entities can be generated by different attributes and attribute values. On the other hand, the addition of attribute attention can be used to distinguish the different roles of different attributes in the entity alignment process. The experimental results show that our method achieves significant improvements compared to baselines for entity alignment on Chinese and English datasets. The data and source code for this paper can be obtained from https://www.github.com/xinan711456/EASA.

Accurate segmentation of pulmonary nodule is essential for subsequent pathological analysis and diagnosis. However, current U-Net architectures often rely on a simple skip connection scheme, leading to the fusion of feature maps with different semantic information, which can have a negative impact on the segmentation model. In response to this challenge, this study introduces a novel U-shaped model specifically designed for pulmonary nodule segmentation. The proposed model incorporates features such as the U-Net backbone, semantic aggregation feature pyramid module, and reverse attention module. The semantic aggregation module combines semantic information with multi-scale features, addressing the semantic gap between the encoder and decoder. The reverse attention module explores missing object parts and captures intricate details by erasing the currently predicted salient regions from side-output features. The proposed model is evaluated using the LIDC-IDRI dataset. Experimental results reveal that the proposed method achieves a dice similarity coefficient of 89.11%and a sensitivity of 90.73 %, outperforming state-of-the-art approaches comprehensively., Competing Interests: Declaration of competing interest No conflict of interest. We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and company that could be construed as influencing the position presented in the manuscript., (Copyright © 2024. Published by Elsevier Ltd.)

A major challenge in place recognition is to be robust against viewpoint changes and appearance changes caused by self and environmental variations. Humans achieve this by recognizing objects and their relationships in the scene under different conditions. Inspired by this, we propose a hierarchical visual place recognition pipeline based on semantic-aggregation and scene understanding for the images. The pipeline contains coarse matching and fine matching. Semantic-aggregation happens in residual aggregation of visual information and semantic information in coarse matching, and semantic association of semantic edges in fine matching. Through the above two processes, we realized a robust coarse-to-fine pipeline of visual place recognition across viewpoint and condition variations. Experimental results on the benchmark datasets show that our method performs better than several state-of-the-art methods, improving the robustness against severe viewpoint changes and appearance changes while maintaining good matching-time performance. Moreover, we prove that it is possible for a computer to realize place recognition based on scene understanding. [ABSTRACT FROM AUTHOR]

A major challenge in place recognition is to be robust against viewpoint changes and appearance changes caused by self and environmental variations. Humans achieve this by recognizing objects and their relationships in the scene under different conditions. Inspired by this, we propose a hierarchical visual place recognition pipeline based on semantic-aggregation and scene understanding for the images. The pipeline contains coarse matching and fine matching. Semantic-aggregation happens in residual aggregation of visual information and semantic information in coarse matching, and semantic association of semantic edges in fine matching. Through the above two processes, we realized a robust coarse-to-fine pipeline of visual place recognition across viewpoint and condition variations. Experimental results on the benchmark datasets show that our method performs better than several state-of-the-art methods, improving the robustness against severe viewpoint changes and appearance changes while maintaining good matching-time performance. Moreover, we prove that it is possible for a computer to realize place recognition based on scene understanding.