Search

Building footprint extraction from remotely sensed imagery is a critical task in the field of illegal building discovery, urban dynamic monitoring, and disaster emergency response. Recent research has made significant progress in this area by utilizing deep learning techniques. However, it remains difficult to efficiently balance the spatial detail and rich semantic features. In particular, the extracted building edge is often inaccurate, especially in areas where the buildings are densely distributed, and the boundary of adjacent building instances is difficult to distinguish accurately. Additionally, identifying buildings with varying scales remains a challenging problem. To address the above problems, we designed a novel framework that aggregated multilevel contextual information extracted from multiple encoders. Furthermore, we introduced an instance constraint into contrastive learning to enhance the robustness of the feature representation. Experimental results demonstrated that our proposed method achieved 91.07% and 74.58% on the intersection over union metric on the WHU and Massachusetts datasets, respectively, outperforming the most recent related methods. Notably, our method significantly improved the accuracy of building boundaries, especially at the building instance level, and the integrity of multi-scale buildings.

The issue of drug safety has received increasing attention in modern society. Estimating the frequency of drug side effects proves to be an effective approach to improving drug development safety. Clinical trials are the most widely used manner in the medical field, but their long duration and high labor costs are always challenging for researchers. Many drug side effect frequency prediction methods based on graph neural networks have also been proposed and achieved good results. However, most current mainstream methods extract feature information separately for drugs and side effects but fail to capture their interaction information, which seriously degenerates the final prediction performance. To solve this problem, we have proposed a network to explore the underlying relationship between drugs and the frequency of side effects by combining graph attention learning, cross attention interaction, and feature aggregation into a unified framework. In this network, we first use the graph attention mechanism to accomplish effective feature extraction for drugs and side effects, respectively. Then, by designing a cross-attention interaction module, the chemical characteristics of each atom of the drug and the correlation between the side effects are captured to gather information on the interaction between the drug and the side effects. Subsequently, we fully fuse graph attention features and interaction attention features by embedding a feature fusion module to obtain enhanced features that fuse drugs and side effects. Finally, the predicted frequency is obtained using matrix inner product operation. Experimental results on the SIDER dataset show that our proposed method achieves the best performance when compared to previous state-of-the-art methods in both warm start and cold start scenarios. We also conducted ablation experiments to demonstrate the effectiveness of different modules embedded in the network. The code, datasets, and materials are available at https://github.com/zixiaojin66/A-3Net-master. In addition, we construct a user-friendly web server for testing at: https://a3net666.streamlit.app/. • We propose a deep learning model for predicting the frequency of drug side effects. • The CAM and GAM effectively perform feature extraction for drugs and side effects. • Experimental results demonstrate its superiority over state-of-the-art models. [ABSTRACT FROM AUTHOR]

The existing deraining methods are based on convolutional neural networks (CNN) learning the mapping relationship between rainy and clean images. However, the real-valued CNN processes the color images as three independent channels separately, which fails to fully leverage color information. Additionally, sliding-window-based neural networks cannot effectively model the non-local characteristics of an image. In this work, we proposed a non-local feature aggregation quaternion network (NLAQNet), which is composed of two concurrent sub-networks: the Quaternion Local Detail Repair Network (QLDRNet) and the Multi-Level Feature Aggregation Network (MLFANet). Furthermore, in the subnetwork of QLDRNet, the Local Detail Repair Block (LDRB) is proposed to repair the backdrop of an image that has not been damaged by rain streaks. Finally, within the MLFANet subnetwork, we have introduced two specialized blocks, namely the Non-Local Feature Aggregation Block (NLAB) and the Feature Aggregation Block (Mix), specifically designed to address the restoration of rain-streak-damaged image backgrounds. Extensive experiments demonstrate that the proposed network delivers strong performance in both qualitative and quantitative evaluations on existing datasets. The code is available at https://github.com/xionggonghe/NLAQNet. [ABSTRACT FROM AUTHOR]

Histopathological analysis is the gold standard for cancer diagnosis, but traditional manual identification methods are time-consuming, labor-intensive, and highly relied on the experience of pathologists. In recent years, deep learning methods can assist pathologists to achieve accurate and automated detection of lesion areas. However, lesion detection is a difficult task due to its various size, shape, and surrounding complex scenes. Existing methods ignore the global interaction between lesion area and normal area, so it is insufficient to extract important global features. In addition, most methods fail to effectively fuse deep and shallow features, resulting in the extracted features lacking discrimination, and these deficiencies lead to incomplete extraction and irregular boundaries of lesion. To address the above-mentioned issues, an end-to-end histopathological image segmentation network based on global context-aware module and deep feature aggregation (GCMDFA) is proposed for segmenting lesions in histopathological images (HIs). In the GCMDFA, a global context-aware module is applied to better explore the interactive relationship between any two pixels to directly capture global contextual information related to lesion tissue of HIs for achieving global features. To better decode the features of encoder, a deep feature aggregation structure is designed to effectively fuse deep and shallow layer features of HIs. Experimental results on three HI datasets (including public and external data) show that the proposed GCMDFA method is more competitive compared with some state-of-the-art (SOTA) methods. This research can assist pathologists to analyze HIs in clinical computer-aided diagnosis. • A GCMDFA method is developed for lesion segmentation of histopathological images. • GCM is applied to capture global contextual information related to lesions. • DFA is designed to fuse deep and shallow layer features of HIs. • Experiments on three datasets show that our method is more competitive. [ABSTRACT FROM AUTHOR]

Pavement crack detection is an essential defect inspection task to ensure traffic and driving safety. Despite the recent studies that have achieved promising performance for inspecting pavement cracks, they still suffer from the challenges of background complexity, crack diversity, and generalization ability. To address these problems, we propose a contrastive decoupling network (CDNet) for detecting pavement cracks. Specifically, this contrastive decoupling framework separately extracts the global and local features with contrastive learning. It can effectively boost the discriminative power and generalization ability of the feature representations. Moreover, we propose a global semantic enhancement (GSE) module to enhance the semantic information of the global features. This module can reinforce the distinguishability for accurately identifying cracks and backgrounds. Furthermore, we propose a local detail refinement (LDR) module to refine the detailed information of the local features. This module can strengthen the localizability for detecting cracks with precise shapes. In addition, we propose a dynamic dependency-aware feature aggregation (DDFA) scheme to adaptively integrate the global and local features based on contextual dependencies. This scheme can enrich the output features for effectively detecting cracks in each image. Experimental results on four pavement crack detection datasets, namely Crack500, CrackTree200, CFD, and AEL, demonstrate that our proposed method outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Aggregating features at various levels or scales has been empirically demonstrated to enhance feature representations in object detection. However, existing approaches tend to aggregate features or embed contextual information indiscriminately through simple concatenation or addition, which disregards the misalignment resulting from repeated sampling operations. This paper proposes a feature-aligned network based on YOLOv5 to address the misalignment issues, namely AlignYOLO. The network consists of three primary modules: the self-attention convolution (SAC) module, the feature aggregation and alignment (FAA) module, and the multiscale aligned channel attention (MSACA) module. Firstly, the SAC module comprehensively extracts information by simultaneously employing both convolution and self-attention. Secondly, the FAA module aggregates features across layers and aligns them through the adoption of a learnable interpolation strategy. Lastly, the MSACA module employs multiscale convolution to capture contextual information. The in-layer features are aligned with the learnable interpolation strategy. Additionally, channel attention is leveraged to enhance feature representations. Extensive experiments are conducted on benchmark datasets to evaluate the effectiveness of the proposed method, where AlignYOLO outperforms state-of-the-art detectors. [ABSTRACT FROM AUTHOR]

Remote sensing image change detection is an important technology in remote sensing data analysis. Existing mainstream solutions are divided into supervised and unsupervised solutions. Among supervised methods, most remote sensing image change methods based on deep learning are related to semantic segmentation. However, these methods only use deep learning model to extract and process single-level information of the image, and do not separately train the changed and unchanged areas according to the task characteristics of change detection. Also this model do not aggregate the deep and shallow-level feature information, so many local change details cannot be detected. This paper proposes a multilateral feature-guided aggregation network. The network firstly extracts the whole information, the changed area information and the unchanged area information of remote sensing image through the main network and two auxiliary networks: difference network and assimilation network. Then, the whole information of the image is aggregated with the changed and unchanged region information by feature aggregation network (all subnetworks are composed of convolutional neural networks (CNNs)). The proposed method is end-to-end trainable, and each component in the network does not need to be trained separately. [ABSTRACT FROM AUTHOR]

Multi-frame human pose estimation is at the core of many computer vision tasks. Although state-of-the-art approaches have demonstrated remarkable results for human pose estimation on static images, their performances inevitably come short when being applied to videos. A central issue lies in the visual degeneration of video frames induced by rapid motion and pose occlusion in dynamic environments. This problem, by nature, is insurmountable for a single frame. Therefore, incorporating complementary visual cues from other video frames becomes an intuitive paradigm. Current state-of-the-art methods usually leverage information from adjacent frames, which unfortunately place excessive focus on only the temporally nearby frames. In this paper, we argue that combining global semantically similar information and local temporal visual context will deliver more comprehensive and more robust representations for human pose estimation. Towards this end, we present an effective framework, namely global-local enhanced pose estimation (GLPose) network. Our framework consists of a feature processing module that conditionally incorporates global semantic information and local visual context to generate a robust human representation and a feature enhancement module that excavates complementary information from this aggregated representation to enhance keyframe features for precise estimation. We empirically find that the proposed GLpose outperforms existing methods by a large margin and achieves new state-of-the-art results on large benchmark datasets. [ABSTRACT FROM AUTHOR]

Visual sound separation (VSS) aims to obtain each sound component from the mixed audio signals with the guidance of visual information. Existing works mainly capture the global-level audio-visual correspondence and exploit various visual features to enhance the appearance and motion features of visual modality. However, they commonly neglect the intrinsic properties of the audio modality, resulting in less effective audio feature extraction and unbalanced audio-visual correspondence. To tackle this problem, we propose a novel end-to-end framework termed Modeling Two-Stream Correspondence (MTSC) for VSS by explicitly extracting the timbre and content features in audio modality. The proposed MTSC method employs a two-stream architecture to enhance audio-visual correspondence for both the appearance-timbre and motion-content features. Moreover, with the advanced two-stream pipeline, more lightweight appearance and motion features for visual modality are exploited. Extensive experiments conducted on two benchmark musical instrument datasets demonstrate that with the above properties, our MTSC method remarkably outperforms seven state-of-the-art VSS approaches. The implementation code and extensive experimental results of the proposed MTSC method are provided at https://github.com/CFM-MSG/MTSC-VSS. [ABSTRACT FROM AUTHOR]

Few-shot learning is a challenging problem in computer vision that aims to learn a new visual concept from very limited data. A core issue is that there is a large amount of uncertainty introduced by the small training set. For example, the few images may include cluttered backgrounds or different scales of objects. Existing approaches mostly address this problem from either the original image space or the embedding space by using meta-learning. To the best of our knowledge, none of them tackle this problem from both spaces jointly. To this end, we propose a fusion spatial attention approach that performs spatial attention in both image and embedding spaces. In the image space, we employ a Saliency Object Detection (SOD) module to extract the saliency map of an image and provide it to the network as an additional channel. In the embedding space, we propose an Adaptive Pooling (Ada-P) module tailored to few-shot learning that introduces a meta-learner to adaptively fuse local features of the feature maps for each individual embedding. The fusion process assigns different pooling weights to the features at different spatial locations. Then, weighted pooling can be conducted over an embedding to fuse local information, which can avoid losing useful information by considering the spatial importance of the features. The SOD and Ada-P modules can be used within a plug-and-play module and incorporated into various existing few-shot learning approaches. We empirically demonstrate that designing spatial attention methods for few-shot learning is a nontrivial task and our method has proven effective for it. We evaluate our method using both shallow and deeper networks on three widely used few-shot learning benchmarks, miniImageNet, tieredImageNet and CUB, and demonstrate very competitive performance. • Our approach performs a fusion spatial attention in both image and embedding space. • SOD extracts saliency maps to tell a feature extractor where to focus. • Ada-P learns a proper pooling operation for few-shot learning problems. • Ada-P adaptively generates pooling weights by considering spatial importance. • Ada-P and SOD can be used as a plug-and-play module for unseen novel classes. [ABSTRACT FROM AUTHOR]

Human Action Recognition (HAR) is a highly notable area of study in contemporary computer vision. Many investigations focused on recognizing a person's actions from video streams based on extracting features regarding orientation and motion. This article presents a Joints and Trajectory-pooled 3D-Deep Positional Attention (PA)-based Hierarchical Bidirectional Recurrent Convolutional Descriptors (JTDPAHBRD) approach which uses a PA-based Hierarchical Bidirectional Recurrent Neural Network (PAHBRNN) for enhancing the feature aggregation process. First, the entire video is segregated into multiple blocks and they are provided to the 2-stream bilinear Convolutional 3D (C3D) model which applies the PAHBRNN as feature aggregation. In PAHBRNN, the feature vectors related to the different parts of a human skeleton in a certain clip are hierarchically aggregated using the position-aware guidance vector. Then, 2 different streams in the C3D network are fused and trained end-to-end using the softmax loss to get the final video descriptor for a particular video sequence. Further, the Support Vector Machine (SVM) classifier is applied to classify the resultant video descriptor to recognize the person's actions. At last, the investigational outcomes demonstrate the JTDPAHBRD achieves 99.6% better recognition accuracy than the classical state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Guangxi Normal University - Natural Science Edition is the property of Gai Kan Bian Wei Hui and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Face detection generally requires prior boxes and an extra non-maximum suppression(NMS) post-processing in modern deep learning methods. However, anchor design and anchor matching strategy significantly affect the performance of face detectors, so we have to spend a lot of time on anchor designing for different business scenarios. The other issue is that NMS cannot be easily parallelized and it may become a bottleneck of detection speed. In this paper, we propose a simple yet efficient pure convolutional neural network face detection method, named dual-branch center face detector(DBCFace for short), which solve face detection via a dual branch fully convolutional framework without extra anchor design and NMS. Extensive experiments are conducted on four popular face detection benchmarks, including AFW, PASCAL face, FDDB, and WIDER FACE, demonstrating that our method is comparable with state-of-the-art methods while the speed is faster. [ABSTRACT FROM AUTHOR]

Models based on capsule neural network (CapsNet), a novel deep learning method, have recently made great achievements in hyperspectral remote sensing image (HSI) classification due to their excellent ability to implicitly model the spatial relationship knowledge embedded in HSIs. However, the number of labeled samples is a common bottleneck in HSI classification, limiting the performance of these deep learning models. To alleviate the problem of limited labeled samples and further explore the potential of CapsNet in the HSI classification field, this study proposes a multiscale feature aggregation capsule neural network (MS-CapsNet) based on CapsNet via the implementation of two branches that simultaneously extract spectral, local spatial, and global spatial features to integrate multiscale features and improve model robustness. Furthermore, because deep features are generally more discriminative than shallow features, two kinds of capsule residual (CapsRES) blocks based on 3D convolutional capsule (3D-ConvCaps) layers and residual connections are proposed to increase the depth of the network and solve the limited labeled sample problem in HSI classification. Moreover, a squeeze-and-excitation (SE) block is introduced in the shallow layers of MS-CapsNet to enhance its feature extraction ability. In addition, a reasonable initialization strategy that transfers parameters from two well-designed, pretrained deep convolutional capsule networks is introduced to help the model find a good set of initializing weight parameters and further improve the HSI classification accuracy of MS-CapsNet. Experimental results on four widely used HSI datasets demonstrate that the proposed method can provide results comparable to those of state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Copyright of Journal of South China University of Technology (Natural Science Edition) is the property of South China University of Technology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

For multi-view convolutional neural network based 3D object recognition, how to fuse the information of multiple views is a key factor affecting the recognition performance. Most traditional methods use max-pooling algorithm to obtain the final 3D object feature, which does not take into account the correlative information between different views. To make full use of the effective information of multiple views, this paper introduces the hierarchical graph attention based multi-view convolutional neural network for 3D object recognition. At first, the view selection module is proposed to reduce redundant view information in multiple views, which can select the projective views with more effective information. Then, the correlation weighted feature aggregation module is proposed to better fuse multiple view features. Finally, the hierarchical feature aggregation network structure is designed to further to make full use of the correlation information of multiple views. Extensive experimental results have validated the effectiveness of the proposed method.

Recent efforts in semantic segmentation using deep learning framework have made notable advances. While achieving high performance, however, they often require heavy computation, making them impractical to be used in real world applications. There are two reasons that produce prohibitive computational cost: 1) heavy backbone CNN to create high resolution of contextual information and 2) complex modules to aggregate multi-level features. To address these issues, we propose the computationally efficient architecture called “Sketch-and-Fill Network (SFNet)” with a three-stage Coarse-to-Fine Aggregation (CFA) module for semantic segmentation. In the proposed network, lower-resolution contextual information is first produced so that the overall computation in the backbone CNN is largely reduced. Then, to alleviate the detail loss of the lower-resolution contextual information, the CFA module forms global structures and fills fine details in a coarse-to-fine manner. To preserve global structures, the contextual information is passed without any reduction to the CFA module. Experimental results show that the proposed SFNet achieves significantly lower computational loads while delivering comparable or improved segmentation performance with state-of-the-art methods. Qualitative results show that our method is superior to state-of-the-art methods in capturing fine detail while keeping global structures on Cityscapes, ADE20K and RUGD benchmarks.

In recent decades, surveillance and home security systems based on video analysis have been proposed for the automatic detection of abnormal situations. Nevertheless, in several real applications, it may be easier to detect a given event from audio information, and the use of audio surveillance systems can greatly improve the robustness and reliability of event detection. In this paper, a novel system for the detection of polyphonic urban noise is proposed for on-campus audio surveillance. The system aggregates different acoustic features to improve the classification accuracy of urban noise. A combination model composed of a capsule neural network (CapsNet) and recurrent neural network (RNN) is employed as the classifier. CapsNet overcomes some limitations of convolutional neural networks (CNNs), such as the loss of position information after max-pooling, and the RNN mainly models the temporal dependency of context information. The combination of these networks further improves the accuracy and robustness of polyphonic sound events detection. Moreover, a monitoring platform is designed to visualize noise maps and acoustic event information. The deployment architecture of the system is used in real environments, and experiments were also conducted on two public datasets. The results demonstrate that the proposed method is superior to existing state-of-art methods for the polyphonic sound event detection task.

Traffic sign detection is a critical task in the visual system of the Advanced Driver Assistance System (ADAS) and the Automated Driving System (ADS). Although the general object detection has achieved promising results by using Feature Pyramid Network (FPN) in recent years, we still observed that FPN cannot obtain satisfactory results in traffic sign detection because the size and class distribution of traffic signs are extremely unbalanced. To overcome this problem, a novel Plug-and-Play neck network Integrated Feature Pyramid Network with Feature Aggregation (IFA-FPN) is proposed in this paper based on the statistical characteristics of traffic signs. First, a lightweight operation is introduced to fully utilize the model and improve the inference speed of the model. Second, an Integrated Operation (IO) is introduced to solve the imbalance problem of Region-of-Interests (RoIs) in pyramid levels. Third, we introduce a Feature Aggregation (FA) structure to strengthen the feature representation capacity of feature maps, thereby enhancing the network robustness against the size discrepancy of traffic signs. The experiments are performed on three mainstream datasets, i.e., the German Traffic Sign Detection Benchmark (GTSDB), Swedish Traffic Sign Dataset (STSD), and Tsinghua-Tencent 100k dataset (TT100k). The experimental results demonstrate the superiority of the proposed IFA-FPN in the traffic sign detection tasks. Specifically, when the proposed IFA-FPN is applied to the Cascade RCNN, it achieves 80.3% mAP in GTSDB which surpasses FPN by 9.9%, 65.2% in mAP in STSD which surpasses FPN by 3.5%, and 93.6% in mAP in TT100k which surpasses FPN by 1.6%.

The recording process of observation is influenced by multiple factors, such as viewpoint, illumination, and state of the object-of-interest etc.Thus, the image observation of the same object may vary a lot under different conditions. This leads to severe intra-class variance which greatly challenges the discrimination ability of the vision model. However, the current prevailing softmax loss for visual recognition only pursues perfect inter-class separation in the feature space. Without considering the intra-class compactness, the learned model easily collapses when it encounters the instances that deviate a lot from their class centroid. To resist the intra-class variance, we start by organizing the input instances as a graph. From this viewpoint, we find that the normalized cut on the graph is a favorable surrogate metric of the intra-class variance within the training batch. Inspired by the equivalence between the normalized cut and random walk, we propose a feature aggregation scheme using transition probabilities as guidance. By imposing supervision on the aggregated features, we can constrain the transition probabilities to form a graph partition consistent with the given labels. Thus, the normalized cut as well as intra-class variance can be well suppressed. To validate the effectiveness of this idea, we instantiate it in spatial, temporal, and spatial-temporal scenarios. Experimental results on corresponding benchmarks demonstrate that the proposed feature aggregation leads to significant improvement in performance. Our method is on par with, or even better than current state-of-the-arts in both tasks. [ABSTRACT FROM AUTHOR]

In recent years, RGBT tracking has become a hot topic in the field of visual tracking, and made great progress. In this paper, we propose a novel Trident Fusion Network (TFNet) to achieve effective fusion of different modalities for robust RGBT tracking. In specific, to deploy the complementarity of features of all convolutional layers, we propose a recursive strategy to densely aggregate these features that yield robust representations of target objects in two modalities. Moreover, we design a trident architecture to integrate the fused features and both modality-specific features for robust target representations. There are three main advantages. First, retaining the classification layer of each modality is beneficial to enhance feature learning of single modality, and compared with aggregate branches, single-modality branches pay more attention to the mining of modal specific information. Second, when some modality is noisy or invalid, the modality-specific branches would capture more discriminative features for RGBT tracking. Finally, the integration of aggregation branches and single-modality branches is beneficial to the complementary learning of different modalities. In addition, we also introduce a feature pruning module in each branch to prune the redundant features and avoid network overfitting. Experimental results on four RGBT tracking benchmark datasets suggest that our tracker achieves superior performance against the state-of-the-art RGBT tracking methods. [ABSTRACT FROM AUTHOR]

We present a unified multi-task learning architecture for fast and accurate pedestrian detection. Different from existing methods which often focus on either a new loss function or architecture, we propose an improved multi-task convolutional neural network learning architecture to effectively and efficiently interfuse the task of pedestrian detection and semantic segmentation. To achieve this, we integrate a lightweight semantic segmentation branch to Faster R-CNN detection framework that enables end-to-end hard parameter sharing in order to boost the detection performance and maintain computational efficiency as follows. Firstly, a Semantic Segmentation to Feature Module (SS2FM) refines the convolutional features in RPN stage by integrating the features generated from the semantic segmentation branch. Secondly, a Semantic Segmentation to Confidence Module (SS2CM) refines the classification confidence in RPN stage by fusing it with the semantic segmentation confidence. We also introduce an effective anchor matching point transform to alleviate the problem of feature misalignment for heavily occluded pedestrians. The proposed unified multi-task learning architecture lends itself well to more robust pedestrian detection in diverse scenarios with negligible computation overhead. In addition, the proposed architecture can achieve high detection performance with low resolution input images, which significantly reduces the computational complexity. Experiment results on CityPersons and Caltech datasets show that our method is the fastest among all state-of-the-art pedestrian detection methods while exhibiting competitive detection performance. [ABSTRACT FROM AUTHOR]

Privacy image classification can help people detect privacy images when people share images. In this paper, we propose a novel method using multi-level and multi-scale features for privacy image classification. We first use CNN (Convolutional Neural Network) to extract multi-levels features. Then, max-pooling layers are employed to obtain multi-scale features at each level. Finally, we propose two feature aggregation models, called Privacy-MSML and Privacy-MLMS to fuse those features for image privacy classification. In Privacy-MSML, multi-scale features of the same level are first integrated and then the integrated features are fused. In Privacy-MLMS, multi-level features of the same scale are first integrated and then the integrated features are fused. Our experiments on a real-world dataset demonstrate the proposed method can achieve better performance compared with the state-of-the-art solutions. [ABSTRACT FROM AUTHOR]

Multi-person pose estimation is a fundamental yet challenging task in computer vision. Although great success has been made in this field due to the rapid development of deep learning, complex situations (e.g., extreme poses, occlusions, overlapped persons, and crowded scenes) are still not well solved. To further mitigate these issues, we propose a novel Content-aware Feature Enhancement Network (CFENet), which consists of three effective modules: Feature Aggregation and Selection Module (FASM), Feature Fusion Module (FFM) and Dense Upsampling Convolution (DUC) module. The FASM includes Feature Aggregation Module (FAM) and Information Selection Module (ISM). The FAM constructs the hierarchical multi-scale feature aggregations in a granular level to capture more accurate fine-grained representations. The ISM makes the aggregated representations more distinguished, which adaptively highlights the discriminative human part representations both in the spatial location and channel context. Then, we perform FFM which effectively fuses high-resolution spatial features and low-resolution semantic features to obtain more reliable context information for well-estimated joints. Finally, we adopt DUC module to generate more precise prediction, which can recover missing joint details that are usually unavailable in common upsampling process. Comprehensive experiments demonstrate that the proposed approach outperforms most of the popular methods and achieves a competitive performance with the state-of-the-art methods over three benchmark datasets: the recent big dataset CrowdPose, the COCO keypoint detection dataset and the MPII Human Pose dataset. Our code will be released upon acceptance. [ABSTRACT FROM AUTHOR]

Existing methods for single images raindrop removal either have poor robustness or suffer from parameter burdens. In this paper, we propose a new Adjacent Aggregation Network (A2Net) with lightweight architectures to remove raindrops from single images. Instead of directly cascading convolutional layers, we design an adjacent aggregation architecture to better fuse features for rich representations generation, which can lead to high quality images reconstruction. To further simplify the learning process, we utilize a problem-specific knowledge to force the network focus on the luminance channel in the YUV color space instead of all RGB channels. By combining adjacent aggregating operation with color space transformation, the proposed A2Net can achieve state-of-the-art performances on raindrop removal with significant parameters reduction.

Fully convolutional neural networks (FCNs) have shown their advantages in the salient object detection task. However, the prediction results do not perform well in most existing FCN-based methods, such as coarse object boundaries or even getting wrong predictions, which resulted from ignoring the difference between multi-level features during feature aggregation or underutilizing the spatial details suitable for locating boundaries. In this paper, we propose a novel end-to-end multi-level context aggregation network (MLCANet) to solve the problem mentioned-above, in which both bottom-up and top-down message passing can cooperate in a joint manner.The bottom-up process that aggregates low-level fine details features into high-level semantically-richer features would enhance high-level features, and in turn the top-down process that passes refined features from deeper layers to the shallower ones could benefit from the enhanced high-level features. Also by considering that the features from different layers may not be equally important, a multi-level feature aggregation mechanism with channel-wise attention is proposed to aggregate multi-level features by flexibly adjusting their contributions and absorbing useful information to refine themselves. The features after message passing which simultaneously encode semantic information and spatial details are used to predict saliency maps in our network. Extensive experiments demonstrate that our method can obtain high quality saliency maps with clear boundaries, and perform favorably against the state-of-the-art methods without any pre-processing and post-processing.

Several or even dozens of times spatial scale variation is one of the major bottleneck for pedestrian detection. Although the Region-based Convolutional Neural Network (R-CNN) families have shown promising results for object detection, they are still limited to detect pedestrians with large scale variations due to the fixed receptive field sizes on a single convolutional output layer. In contrast to previous methods that simply combined pedestrian predictions on feature maps with different resolution, we propose a scale-aware hierarchical detection network for pedestrian detection under large scale variations. First, we introduce a cross-scale features aggregation module to accomplish feature augmentation for pedestrian representation through merging the lateral connection, the top-down path and bottom-up path. Specifically, the cross-scale features aggregation module can adaptively fuse hierarchical features to enhance feature pyramid representation for robust semantic and accurate localization. Further, we design a scale-aware hierarchical detection network to effectively integrate multiscale pedestrian detection into a unified framework by adaptively perceiving the augmented feature level for special-scale pedestrian detection. Experimentally, the proposed scale-aware hierarchical detection network forms a more robust and discriminative model for pedestrian instances with different scales on widely-used ETH and Caltech benchmarks. In particular, compared with the state-of-the-art method FasterRCNN+ATT, the log-average miss rate of pedestrian detection is reduced by 11.98% for medium scale pedestrians (between 30-80 pixels in height), and 14.12% for whole scale pedestrians (above 20 pixels in height) on Caltech benchmark.

Objective . Convolutional neural network (CNN) is developing rapidly in the field of medical image registration, and the proposed U-Net further improves the precision of registration. However, this method may discard certain important information in the process of encoding and decoding steps, consequently leading to a decline in accuracy. To solve this problem, a multi-channel semantic-aware and residual attention mechanism network (MSRA-Net) is proposed in this paper. Approach . Our proposed network achieves efficient information aggregation by cleverly extracting the features of different channels. Firstly, a context-aware module (CAM) is designed to extract valuable contextual information. And the depth-wise separable convolution is employed in the CAM to alleviate the computational burden. Then, a new multi-channel semantic-aware module (MCSAM) is designed for more comprehensive fusion of up-sampling features. Additionally, the residual attention module is introduced in the up-sampling process to extract more semantic information and minimize information loss. Main results . This study utilizes Dice score, average symmetric surface distance and negative Jacobian determinant evaluation metrics to evaluate the influence of registration. The experimental results demonstrate that our proposed MSRA-Net has the highest accuracy compared to several state-of-the-art methods. Moreover, our network has demonstrated the highest Dice score across multiple datasets, thereby indicating that the superior generalization capabilities of our model. Significance . The proposed MSRA-Net offers a novel approach to improve medical image registration accuracy, with implications for various clinical applications. Our implementation is available athttps://github.com/shy922/MSRA-Net., (© 2024 Institute of Physics and Engineering in Medicine.)

Remote sensing (RS) images have been widely used in disaster monitoring due to their wide observation and timeliness. Wildfire is a type of destructive disaster, and smoke is an important signal of the occurrence of wildfires; therefore, it is necessary to perform smoke detection in RS images. Smoke-like scenes such as clouds captured by satellites make RS smoke detection a tough task. Differences in resolution and the complexity of the geographical environment further increases the difficulty of detection: smoke in some images only occupies a small area, while in other images it may fill the whole image. Thus, global information and salient features are both essential for RS smoke detection. From this point of view, we design a self-adaptive feature aggregation (SAFA) network to distinguish smoke from other scenes in RS images. SAFA has two pathways: the global information extraction pathway (GIEP) for capturing global information and the salient feature extraction pathway (SFEP) for extracting salient features. The self-adaptive aggregation means GIEP and SFEP are summed up by trainable weight coefficients to make the final prediction. We conducted experiments on USTC_SmokeRS data, which are specially set up for RS smoke detection. This dataset contains smoke and other smoke-like scenes such as clouds, dust and haze. The experimental results show that SAFA achieves the new state-of-the-art classification accuracy of 96.22% on this dataset. [ABSTRACT FROM AUTHOR]

• We propose a Local Graph Aggregation (LGA) module for CNN information diffusion. • We present the LABNet architecture with a cascade of LGA modules for vehicle re-id. • Our method outperforms state-of-the-art vehicle re-id methods on several datasets. Vehicle re-identification is an important computer vision task where the objective is to identify a specific vehicle among a set of vehicles seen at various viewpoints. Recent methods based on deep learning utilize a global average pooling layer after the backbone feature extractor, however, this ignores any spatial reasoning on the feature map. In this paper, we propose local graph aggregation on the backbone feature map, to learn associations of local information and hence improve feature learning as well as reduce the effects of partial occlusion and background clutter. Our local graph aggregation network considers spatial regions of the feature map as nodes and builds a local neighborhood graph that performs local feature aggregation before the global average pooling layer. We further utilize a batch normalization layer to improve the system effectiveness. Additionally, we introduce a class balanced loss to compensate for the imbalance in the sample distributions found in the most widely used vehicle re-identification datasets. Finally, we evaluate our method in three popular benchmarks and show that our approach outperforms many state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In recent years, there has been an increasing interest in Environmental Sound Classification (ESC), and it is a challenging non-speech audio event classification problem because of the complexity of the environment. However, the classification accuracy of the conventional methods is significantly dependent on the robustness of representative features and the effectiveness of the constructed model, which causes the poor adaptability of current models. Considering this, a novel ESC scheme based on stacked Deep Neural Networks with multi-dimensional aggregated features is proposed. Firstly, we use the aggregated features composed of time-domain features and time–frequency (TF) domain features to capture a more comprehensive representation of sounds. Afterward, the feature reduction based on Principal Component Analysis (PCA) is employed to select the most discriminative representations. Finally, a novel Stacked Deep Neural Networks based on ensemble learning and data augmentation is presented to improve the ESC scheme's generalizing capability. The experimental results demonstrate that the proposed method is appropriate for ESC problems, which achieves 96.1% and 98.1% accuracy scores for ESC-10 and UrbanSound8K datasets, respectively, and outperforms most state-of-art methods in ESC tasks at the aspect of both accuracy and computational burden. [ABSTRACT FROM AUTHOR]

• Strengthen the descriptor by effectively combing features at different levels of CNN. • Capture the spatial information of a local patch with spatial context pyramid. • Comparable performance with state-of-the-art and comprehensive ablation experiments. Despite that efforts have shifted to learning local descriptors with convolutional neural network (CNN) from hand-crafted realm, the inherent feature hierarchy within CNN has been rarely explored. To increase both the invariant and discriminative abilities of the CNN-based local descriptors by making use of the complementary representation powers of the feature maps at different levels of CNN, in this paper, we design a multi-level feature aggregation (MLFA) module to communicate information across pyramid levels effectively. Then, each level extracts a feature vector after feature fusion and the final descriptor concatenates these outputs. Moreover, to leverage the spatial structure within a local patch, we propose a novel spatial context pyramid (SCP) module to capture the spatial information. SCP is devised in a residual manner and only several additional parameters are introduced to the model. We implement our algorithm based on the HardNet framework and carry out comprehensive evaluation on the UBC Phototour, HPatches and ETH datasets. The experimental results demonstrate that the proposed method performs favorably against the state-of-the-art ones. Ablation study is also provided to show the effectiveness of each component. [ABSTRACT FROM AUTHOR]

Retinal vessel segmentation plays an important role in the diagnosis of eye-related diseases and biomarkers discovery. Existing works perform multi-scale feature aggregation in an inter-layer manner, namely inter-layer feature aggregation. However, such an approach only fuses features at either a lower scale or a higher scale, which may result in a limited segmentation performance, especially on thin vessels. This discovery motivates us to fuse multi-scale features in each layer, intra-layer feature aggregation, to mitigate the problem. Therefore, in this paper, we propose Pyramid-Net for accurate retinal vessel segmentation, which features intra-layer pyramid-scale aggregation blocks (IPABs). At each layer, IPABs generate two associated branches at a higher scale and a lower scale, respectively, and the two with the main branch at the current scale operate in a pyramid-scale manner. Three further enhancements including pyramid inputs enhancement, deep pyramid supervision, and pyramid skip connections are proposed to boost the performance. We have evaluated Pyramid-Net on three public retinal fundus photography datasets (DRIVE, STARE, and CHASE-DB1). The experimental results show that Pyramid-Net can effectively improve the segmentation performance especially on thin vessels, and outperforms the current state-of-the-art methods on all the adopted three datasets. In addition, our method is more efficient than existing methods with a large reduction in computational cost. We have released the source code at https://github.com/JerRuy/Pyramid-Net.

Recently, point-based networks have begun to prevail because they retain more original geometric information from point clouds than other deep learning-based methods. However, we observe that: (1) the set abstraction design for local aggregation in point-based networks neglects that the points in a local region may belong to different semantic categories, and (2) most works focus on single-scale local features while ignoring the importance of multi-scale global features. To tackle the above issues, we propose two novel strategies named semantic-based local aggregation (SLA) and multi-scale global pyramid (MGP). The key idea of SLA is to augment local features based on the semantic similarity of neighboring points in the local region. Additionally, we propose a hierarchical global aggregation (HGA) module to extend local feature aggregation to global feature aggregation. Based on HGA, we introduce MGP to obtain discriminative multi-scale global features from multi-resolution point cloud scenes. Extensive experiments on two prevailing benchmarks, S3DIS and Semantic3D, demonstrate the effectiveness of our method.

Video super-resolution reconstruction is the process of reconstructing low-resolution video frames into high-resolution video frames. Most of the current methods use motion estimation and motion compensation to extract temporal series information, but the inaccuracy of motion estimation will lead to the degradation of the quality of video super-resolution results. Additionally, when using convolution network to extract feature information, the number of feature information is limited by the number of feature channels, resulting in poor reconstruction results. In this paper, we propose a recurrent structure of regional focus network for video super-resolution, which can avoid the influence of inaccurate motion compensation on super-resolution results. Meanwhile, regional focus blocks in the network can focus on different areas of video frames, extract different features from shallow to deep layers, and skip-connect to the last layer of the network through feature aggregation to improve the richness of features participating in the reconstruction. The experimental results show that our method has higher computational efficiency and better video super-resolution results than other temporal modeling methods.

Copyright of Journal of Computer-Aided Design & Computer Graphics / Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao is the property of Gai Kan Bian Wei Hui and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Plant disease diagnosis is very critical for agriculture due to its importance for increasing crop production. Recent advances in image processing offer us a new way to solve this issue via visual plant disease analysis. However, there are few works in this area, not to mention systematic researches. In this paper, we systematically investigate the problem of visual plant disease recognition for plant disease diagnosis. Compared with other types of images, plant disease images generally exhibit randomly distributed lesions, diverse symptoms and complex backgrounds, and thus are hard to capture discriminative information. To facilitate the plant disease recognition research, we construct a new large-scale plant disease dataset with 271 plant disease categories and 220,592 images. Based on this dataset, we tackle plant disease recognition via reweighting both visual regions and loss to emphasize diseased parts. We first compute the weights of all the divided patches from each image based on the cluster distribution of these patches to indicate the discriminative level of each patch. Then we allocate the weight to each loss for each patch-label pair during weakly-supervised training to enable discriminative disease part learning. We finally extract patch features from the network trained with loss reweighting, and utilize the LSTM network to encode the weighed patch feature sequence into a comprehensive feature representation. Extensive evaluations on this dataset and another public dataset demonstrate the advantage of the proposed method. We expect this research will further the agenda of plant disease recognition in the community of image processing. [ABSTRACT FROM AUTHOR]

Existing methods for dynamic scene recognition mostly use global features extracted from the entire video frame or a video segment. In this paper, a part-based method is proposed to aggregate local features from video frames. A pre-trained Fast R-CNN model is used to extract local convolutional features from the regions of interest of training images. These features are clustered to locate representative parts. A set cover problem is then formulated to select the discriminative parts, which are further refined by fine-tuning the Fast R-CNN model. Local features from a video segment are extracted at different layers of the fine-tuned Fast R-CNN model and aggregated both spatially and temporally. Extensive experimental results show that the proposed method is very competitive with state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

This letter proposes an attention-based convolutional neural network architecture for multitasking learning to accurately classify not only the presence of an earthquake but also the event type of the earthquake. In particular, to improve the performance in earthquake-type classification, we develop an attention-based feature aggregation framework embedded in multitask learning architecture. Representative experimental results show that the proposed method provides an effective structure for an earthquake detection and event classification with an earthquake database of the Korean peninsula and the Circum-Pacific belt. [ABSTRACT FROM AUTHOR]

Detecting small-scale pedestrians in aerial images is a challenging task that can be difficult even for humans. Observing that the single image based method cannot achieve robust performance because of the poor visual cues of small instances. Considering that multiple frames may provide more information to detect such difficult case instead of only single frame, we design a novel video based pedestrian detection method with a two-stream network pipeline to fully utilize the temporal and contextual information of a video. An aggregated feature map is proposed to absorb the spatial and temporal information with the help of spatial and temporal sub-networks. To better capture motion information, a more refined flow net (SPyNet) is adopted instead of a simple flownet. In the spatial stream subnetwork, we modified the backbone network structure by increasing the feature map resolution with relatively larger receptive field to make it suitable for small-scale detection. Experimental results based on drone video datasets demonstrate that our approach improves detection accuracy in the case of small-scale instances and reduces false positive detections. By exploiting the temporal information and aggregating the feature maps, our two-stream method improves the detection performance by 8.48% in mean Average Precision (mAP) from that of the basic single stream R-FCN method, and it outperforms the state-of-the-art method by 3.09% on the Okutama Human-action dataset. [ABSTRACT FROM AUTHOR]

Recently, an emerging trend in medical image classification is to combine radiomics framework with deep learning classification network in an integrated system. Although this combination is efficient in some tasks, the deep learning-based classification network is often difficult to capture an effective representation of lesion regions, and prone to face the challenge of overfitting, leading to unreliable features and inaccurate results, especially when the sizes of the lesions are small or the training dataset is small. In addition, these combinations mostly lack an effective feature selection mechanism, which makes it difficult to obtain the optimal feature selection. In this paper, we introduce a novel and effective deep semantic segmentation feature-based radiomics (DSFR) framework to overcome the above-mentioned challenges, which consists of two modules: the deep semantic feature extraction module and the feature selection module. Specifically, the extraction module is utilized to extract hierarchical semantic features of the lesions from a trained segmentation network. The feature selection module aims to select the most representative features by using a novel feature similarity adaptation algorithm. Experiments are extensively conducted to evaluate our method in two clinical tasks: the pathological grading prediction in pancreatic neuroendocrine neoplasms (pNENs), and the prediction of thrombolytic therapy efficacy in deep venous thrombosis (DVT). Experimental results on both tasks demonstrate that the proposed method consistently outperforms the state-of-the-art approaches by a large margin. [ABSTRACT FROM AUTHOR]

Multi-head self-attentions (MSAs) in Transformer are low-pass filters, which will tend to reduce high-frequency signals. Convolutional layers (Convs) in Convolutional Neural Network (CNN) are high-pass filters, which will tend to capture high-frequency components of the images. Therefore, CNN and Transformer contain complementary information, and the combination of the two is necessary for satisfactory detection results. In this work, we propose a novel framework PileNet that efficiently combine CNN and Transformer for accurate salient object detection (SOD). Specifically in PileNet, we introduce complementary encoder that extracts multi-level complementary saliency features. Next, we simplify the complementary features by adjusting the number of channels for all features to a fixed value. By introducing the multi-level feature aggregation (MLFA) and multi-level feature refinement (MLFR) units, the low- and high-level features can easily be transmitted to feature blocks at various pyramid levels. Finally, we fuse all the refined saliency features in a Unet-like structure from top to bottom and use multi-point supervision mechanism to produce the final saliency maps. Extensive experimental results over five widely used saliency benchmark datasets clearly demonstrate that our proposed model can accurately locate the entire salient objects with clear object boundaries and outperform sixteen previous state-of-the-art saliency methods in terms of a wide range of metrics. • A high-and-low pass complementary filter is used to generate encoders. • We design an effective multi-level feature refinement unit. • We design a multi-level feature aggregation unit with shared parameters. • A multi-point supervision mechanism is proposed to generate saliency maps. [ABSTRACT FROM AUTHOR]

Camouflaged object detection (COD) aims to identify objects that are visually blended into their highly similar surroundings, which is an extremely complex and challenging visual task in real-world scenarios, and has recently attracted increasing research interest in the field of computer vision due to its valuable applications. The existing deep learning based methods of COD have the following problems: 1) the ambiguous boundary of the camouflaged objects in prediction map, 2) an inaccurate detection of the camouflaged object with accurate and complete structure details. To this end, an attention guided multi-level feature aggregation network is proposed for this task, which is based on three key designs. First, by embedding spatial pyramid attention (SPA) in ResNet-like backbone network, better multi-level features are extracted to identify the camouflaged objects with complete internal details. Second, an edge context module is designed to make full use of edge information, which highlights camouflaged object structure and generates accurate edge localization of COD. Third, the feature aggregation module based on local attention is used to fuse these enhanced multi-level features and edge context cues, which consists of two major components: the iterative Attentional Feature Fusion (iAFF) module and the Dual-branch Global Context Module (DGCM). Compared with the existing 16 state-of-the-art methods, extensive experiments on four widely-used benchmark datasets under four authoritative evaluation metrics illustrate that the proposed method is very beneficial to the COD task. • A ResNet-like backbone equiped with SPA-block is designed to get better initial features. • Several simple modules are used to boost features for more discriminative representation. • FAM module is designed to fuse features and edge cue in multiple supervision to get more accurate results. • Experiments and evaluations validate the effectiveness of the modules, and show superior performance of the proposed method. [ABSTRACT FROM AUTHOR]