Search

Accompanied with the rapid increase of the demand for routine examination of leucorrhea, efficiency and accuracy become the primary task. However, in super depth of field (SDoF) system, the problem of automatic detection and localization of cells in leucorrhea micro-images is still a big challenge. The changing of the relative position between the cell center and focus plane of microscope lead to variable cell morphological structure in the two-dimensional image, which is an important reason for the low accuracy of current deep learning target detection algorithms. In this paper, an object detection method based on Retinanet in state of super depth of field is proposed, which can achieve high precision detecting of leucorrhea components by the SDoF feature aggregation module. Compared with the current mainstream algorithms, the mean average accuracy (mAP) index has been improved significantly, the mAP index is 82.7% for SDoF module and 83.0% for SDoF+ module, with an average increase of more than 10%. These improved features can significantly improve the efficiency and accuracy of the algorithm. The algorithm proposed in this paper can be integrated into the leucorrhea automatic detection system.

Height estimation from single images, strictly speaking, is an ill-posed problem. However, recently, it is shown that it is both possible and feasible to learn a mapping from image statistics to height information. In spite of recent efforts in this field, how to learn fine-shape preserving features, such as object boundaries and contours, is still an open issue. In this work, we propose a progressive learning network to estimate height information from single aerial images in a coarse-to-fine manner. In particular, a gated feature aggregation module is introduced to effectively combine low-level and high-level features. The proposed method is validated on three public datasets, including the Vaihingen dataset, the Potsdam dataset, and the DFC2019 dataset. Both quantitative and qualitative experimental results demonstrate that the proposed method can achieve more accurate height estimation from single aerial images, especially with better object boundary and contour preserving capability, than four related height estimation methods.

Facade image parsing is essential to the semantic understanding and 3-D reconstruction of urban scenes. Considering the occlusion and appearance ambiguity in single-view images and the easy acquisition of multiple views, in this letter, we propose a multiview enhanced deep architecture for facade parsing. The highlight of this architecture is a cross-view feature aggregation module that can learn to choose and fuse useful convolutional neural network (CNN) features from nearby views to enhance the representation of a target view. Benefitting from the multiview enhanced representation, the proposed architecture can better deal with the ambiguity and occlusion issues. Moreover, our cross-view feature aggregation module can be straightforwardly integrated into existing single-image parsing frameworks. Extensive comparison experiments and ablation studies are conducted to demonstrate the good performance of the proposed method and the validity and transportability of the cross-view feature aggregation module.

At night or in other low‐illumination environments, optical imaging devices cannot capture details and color information in images accurately because of the reduced number of photons captured and the low signal‐to‐noise ratio. Consequently, the image is very noisy with low contrast and inaccurate color information, which affects human visual perception and creates significant challenges in computer vision tasks. Low‐light image enhancement has great research value because it aims to reduce image noise and improve image quality. In this study, we propose an LBP‐based progressive feature aggregation network (P‐FANet) for low‐light image enhancement. The LBP feature has insensitivity to illumination, and it contains rich texture information. In the network, we input the LBP feature into each iteration of the network in an accompanying manner, which helps to restore some detailed information of the low‐light image. First, we input the low‐light image into the dual attention mechanism model to extract global features. Second, the extracted different features enter the feature aggregation module (FAM) for feature fusion. Third, we use the recurrent layer to share the features extracted at different stages, and use the residual layer to further extract deeper features. Finally, the enhanced image is output. The rationality of the method in this study has been verified through ablation experiments. Many experimental results show that the method in this study has greater advantages in subjective and objective evaluations compared with many other advanced methods.

Privacy image classification can help people detect privacy images when people share images. In this paper, we propose a novel method using multi-level and multi-scale features for privacy image classification. We first use CNN (Convolutional Neural Network) to extract multi-levels features. Then, max-pooling layers are employed to obtain multi-scale features at each level. Finally, we propose two feature aggregation models, called Privacy-MSML and Privacy-MLMS to fuse those features for image privacy classification. In Privacy-MSML, multi-scale features of the same level are first integrated and then the integrated features are fused. In Privacy-MLMS, multi-level features of the same scale are first integrated and then the integrated features are fused. Our experiments on a real-world dataset demonstrate the proposed method can achieve better performance compared with the state-of-the-art solutions.

In text classification models based on deep learning, feature extraction and feature aggregation are two key steps. As one of the basic feature extraction methods, CNN has certain limitations due to its inability to effectively extract temporal features from text data. Using max-pooling can significantly reduce the amount of calculation while performing feature aggregation, but it will have an adverse effect on the classification results due to the loss of some text features. In this paper, in response to the above two issues, a Bidirectional Gated Temporal Convolutional Attention(BG-TCA) model is proposed. In the feature extraction stage, the BG-TCA model uses the bidirectional TCN to extract the bidirectional temporal features in text data, and a gating mechanism similar to the LSTM is added between the convolution layers. In the feature aggregation stage, the BG-TCA model uses the attention mechanism to replace the max-pooling method, which makes it possible to distinguish the importance of different features while retaining the text features to the maximum. Finally, experimental results on five benchmark datasets show that the classification accuracy of the BG-TCA model has been greatly improved compared to basic models, and is better than several other state-of-the-art models.

In this paper, we focus on the text classification task, which is a most import task in the area of Natural Language Processing (NLP). We propose an innovative convolutional neural network (CNN) model to perform temporal feature aggregation (TFA) effectively, which has a highly representative capacity to extract sequential features from vectorized numerical embeddings. First, we feed embedded vectors into a bi-directional LSTM (Bi-LSTM) model to capture the contextual information of each word. Afterwards, we propose to use the state-of-the-art deep-learning models as key components of the architecture, i.e., the Xception model and the WaveNet model, to extract temporal features from deep convolutional layers concurrently. To facilitate an effective feature fusion, we concatenate the outputs of two component models before forwarding to a drop-out layer to alleviate over-fitting and subsequently a fully-connected dense layer to perform the final classification of input texts. Experiments demonstrate that the proposed method achieves performance comparable to the state-of-the-art models while at a significantly lower computational complexity. Our approach obtains the cross-validation score of 95.83% for the Quora Insincere Question Classification (QIQC) dataset, and the cross-validation score of 83.10% for the Spooky Author Identification (SAI) dataset, respectively, which are among the best published results. The proposed method can be readily generalized to signal processing tasks, e.g., environmental sound classification (ESC) and machine fault analysis (MFA).

Recently, crowd counting has drawn widespread attention in computer vision, but it is extremely challenging because of the varying scales and densities. Many existing methods focus on improving the multi-scale representation by utilizing multi-column or multi-branch architectures with different kernel sizes. However, such networks cannot extract the feature maps with large receptive fields due to limitation of depth. In addition, the importance of utilizing the multi-level feature information in a deep network is ignored. In this paper, we propose a multi-scale and multi-level features aggregation network (MFANet) for accurate and efficient crowd counting, and it can be trained by end-to-end. A vital component of the network is the scale and level aggregation module (SLAM), which can extract multi-scale features and make full use of multi-level feature information for more accurate estimation. When six SLAMs are stacked together and applied to our network, our method can achieve the best performance. Furthermore, we introduce a new loss function called normalized Euclidean loss (NEL) to balance the contribution of all samples to network training. To demonstrate the performance of the proposed method, extensive experiments are conducted on four benchmark crowd counting datasets, including ShanghaiTec Part A/B, UCF-CC-50, Mall, and UCF-QNRF. Experimental results show that our MFANet achieves state-of-the-art performance in crowd counting and crowd localization.

Traffic sign detection is a critical task in the visual system of the Advanced Driver Assistance System (ADAS) and the Automated Driving System (ADS). Although the general object detection has achieved promising results by using Feature Pyramid Network (FPN) in recent years, we still observed that FPN cannot obtain satisfactory results in traffic sign detection because the size and class distribution of traffic signs are extremely unbalanced. To overcome this problem, a novel Plug-and-Play neck network Integrated Feature Pyramid Network with Feature Aggregation (IFA-FPN) is proposed in this paper based on the statistical characteristics of traffic signs. First, a lightweight operation is introduced to fully utilize the model and improve the inference speed of the model. Second, an Integrated Operation (IO) is introduced to solve the imbalance problem of Region-of-Interests (RoIs) in pyramid levels. Third, we introduce a Feature Aggregation (FA) structure to strengthen the feature representation capacity of feature maps, thereby enhancing the network robustness against the size discrepancy of traffic signs. The experiments are performed on three mainstream datasets, i.e., the German Traffic Sign Detection Benchmark (GTSDB), Swedish Traffic Sign Dataset (STSD), and Tsinghua-Tencent 100k dataset (TT100k). The experimental results demonstrate the superiority of the proposed IFA-FPN in the traffic sign detection tasks. Specifically, when the proposed IFA-FPN is applied to the Cascade RCNN, it achieves 80.3% mAP in GTSDB which surpasses FPN by 9.9%, 65.2% in mAP in STSD which surpasses FPN by 3.5%, and 93.6% in mAP in TT100k which surpasses FPN by 1.6%.

Recent efforts in semantic segmentation using deep learning framework have made notable advances. While achieving high performance, however, they often require heavy computation, making them impractical to be used in real world applications. There are two reasons that produce prohibitive computational cost: 1) heavy backbone CNN to create high resolution of contextual information and 2) complex modules to aggregate multi-level features. To address these issues, we propose the computationally efficient architecture called “Sketch-and-Fill Network (SFNet)” with a three-stage Coarse-to-Fine Aggregation (CFA) module for semantic segmentation. In the proposed network, lower-resolution contextual information is first produced so that the overall computation in the backbone CNN is largely reduced. Then, to alleviate the detail loss of the lower-resolution contextual information, the CFA module forms global structures and fills fine details in a coarse-to-fine manner. To preserve global structures, the contextual information is passed without any reduction to the CFA module. Experimental results show that the proposed SFNet achieves significantly lower computational loads while delivering comparable or improved segmentation performance with state-of-the-art methods. Qualitative results show that our method is superior to state-of-the-art methods in capturing fine detail while keeping global structures on Cityscapes, ADE20K and RUGD benchmarks.

In recent decades, surveillance and home security systems based on video analysis have been proposed for the automatic detection of abnormal situations. Nevertheless, in several real applications, it may be easier to detect a given event from audio information, and the use of audio surveillance systems can greatly improve the robustness and reliability of event detection. In this paper, a novel system for the detection of polyphonic urban noise is proposed for on-campus audio surveillance. The system aggregates different acoustic features to improve the classification accuracy of urban noise. A combination model composed of a capsule neural network (CapsNet) and recurrent neural network (RNN) is employed as the classifier. CapsNet overcomes some limitations of convolutional neural networks (CNNs), such as the loss of position information after max-pooling, and the RNN mainly models the temporal dependency of context information. The combination of these networks further improves the accuracy and robustness of polyphonic sound events detection. Moreover, a monitoring platform is designed to visualize noise maps and acoustic event information. The deployment architecture of the system is used in real environments, and experiments were also conducted on two public datasets. The results demonstrate that the proposed method is superior to existing state-of-art methods for the polyphonic sound event detection task.

Existing methods for dynamic scene recognition mostly use global features extracted from the entire video frame or a video segment. In this paper, a part-based method is proposed to aggregate local features from video frames. A pre-trained Fast R-CNN model is used to extract local convolutional features from the regions of interest of training images. These features are clustered to locate representative parts. A set cover problem is then formulated to select the discriminative parts, which are further refined by fine-tuning the Fast R-CNN model. Local features from a video segment are extracted at different layers of the fine-tuned Fast R-CNN model and aggregated both spatially and temporally. Extensive experimental results show that the proposed method is very competitive with state-of-the-art approaches.

In comparison with static image object detection, focusing on video objects has greater research significance in realizing intelligent monitoring and automatic anomaly detection. However, it may be challenging to apply the most advanced image recognition networks to video data, as the number of static frame files represented in a video is often huge, thereby causing the problem of the slow evaluation speed, in addition to other issues, such as motion blur, low resolution, occlusion, and object deformation. In the present study, to mitigate these deficiencies, we applied sparse feature propagation to improve the detection speed and dense feature aggregation to refine the detection accuracy. Moreover, we utilized the key frame scheduling strategy relying on the consistency of feature information. Implementing these technologies allowed steadily improving the detection speed and accuracy to achieve high performance. To verify the applicability of the optimized video detection strategy proposed in this paper, we selected the part of the video data in the ImageNet VID training dataset. Then, the other part of this dataset was used to conduct the experiments, including the calculation and comparison of mean average precision (MAP) and frames per second (FPS).

Existing methods for single images raindrop removal either have poor robustness or suffer from parameter burdens. In this paper, we propose a new Adjacent Aggregation Network (A2Net) with lightweight architectures to remove raindrops from single images. Instead of directly cascading convolutional layers, we design an adjacent aggregation architecture to better fuse features for rich representations generation, which can lead to high quality images reconstruction. To further simplify the learning process, we utilize a problem-specific knowledge to force the network focus on the luminance channel in the YUV color space instead of all RGB channels. By combining adjacent aggregating operation with color space transformation, the proposed A2Net can achieve state-of-the-art performances on raindrop removal with significant parameters reduction.

Several or even dozens of times spatial scale variation is one of the major bottleneck for pedestrian detection. Although the Region-based Convolutional Neural Network (R-CNN) families have shown promising results for object detection, they are still limited to detect pedestrians with large scale variations due to the fixed receptive field sizes on a single convolutional output layer. In contrast to previous methods that simply combined pedestrian predictions on feature maps with different resolution, we propose a scale-aware hierarchical detection network for pedestrian detection under large scale variations. First, we introduce a cross-scale features aggregation module to accomplish feature augmentation for pedestrian representation through merging the lateral connection, the top-down path and bottom-up path. Specifically, the cross-scale features aggregation module can adaptively fuse hierarchical features to enhance feature pyramid representation for robust semantic and accurate localization. Further, we design a scale-aware hierarchical detection network to effectively integrate multiscale pedestrian detection into a unified framework by adaptively perceiving the augmented feature level for special-scale pedestrian detection. Experimentally, the proposed scale-aware hierarchical detection network forms a more robust and discriminative model for pedestrian instances with different scales on widely-used ETH and Caltech benchmarks. In particular, compared with the state-of-the-art method FasterRCNN+ATT, the log-average miss rate of pedestrian detection is reduced by 11.98% for medium scale pedestrians (between 30-80 pixels in height), and 14.12% for whole scale pedestrians (above 20 pixels in height) on Caltech benchmark.

Fully convolutional neural networks (FCNs) have shown their advantages in the salient object detection task. However, the prediction results do not perform well in most existing FCN-based methods, such as coarse object boundaries or even getting wrong predictions, which resulted from ignoring the difference between multi-level features during feature aggregation or underutilizing the spatial details suitable for locating boundaries. In this paper, we propose a novel end-to-end multi-level context aggregation network (MLCANet) to solve the problem mentioned-above, in which both bottom-up and top-down message passing can cooperate in a joint manner.The bottom-up process that aggregates low-level fine details features into high-level semantically-richer features would enhance high-level features, and in turn the top-down process that passes refined features from deeper layers to the shallower ones could benefit from the enhanced high-level features. Also by considering that the features from different layers may not be equally important, a multi-level feature aggregation mechanism with channel-wise attention is proposed to aggregate multi-level features by flexibly adjusting their contributions and absorbing useful information to refine themselves. The features after message passing which simultaneously encode semantic information and spatial details are used to predict saliency maps in our network. Extensive experiments demonstrate that our method can obtain high quality saliency maps with clear boundaries, and perform favorably against the state-of-the-art methods without any pre-processing and post-processing.

Along with current multi-scale based detectors, Feature Aggregation and Enhancement (FAE) modules have shown superior performance gains for cutting-edge object detection. However, these hand-crafted FAE modules show inconsistent improvements on face detection, which is mainly due to the significant distribution difference between its training and applying corpus, COCO vs. WIDER Face. To tackle this problem, we essentially analyse the effect of data distribution, and consequently propose to search an effective FAE architecture, termed AutoFAE by a differentiable architecture search, which outperforms all existing FAE modules in face detection with a considerable margin. Upon the found AutoFAE and existing backbones, a supernet is further built and trained, which automatically obtains a family of detectors under the different complexity constraints. Extensive experiments conducted on popular benchmarks, WIDER Face and FDDB, demonstrate the state-of-the-art performance-efficiency trade-off for the proposed automatic and scalable face detector (ASFD) family. In particular, our strong ASFD-D6 outperforms the best competitor with AP 96.7/96.2/92.1 on WIDER Face test, and the lightweight ASFD-D0 costs about 3.1 ms, more than 320 FPS, on the V100 GPU with VGA-resolution images., ACM MM2021

Temporal action detection aims to locate specific segments of action instances in an untrimmed video. Most existing approaches commonly extract the features of all candidate video segments and then classify them separately. However, they may neglect the underlying relationship among candidates unconsciously. In this paper, we propose a novel model termed Candidate-Aware Aggregation (CAA) to tackle this problem. In CAA, we design the Global Awareness (GA) module to exploit long-range relations among all candidates from a global perspective, which enhances the features of action instances. The GA module is then embedded into a multi-level hierarchical network named FENet, to aggregate local features in adjacent candidates to suppress background noise. As a result, the relationship among candidates is explicitly captured from both local and global perspectives, which ensures more accurate prediction results for the candidates. Extensive experiments conducted on two popular benchmarks ActivityNet-1.3 and THUMOS-14 demonstrate the superiority of CAA comparing to the state-of-the-art methods.

For license plate detection (LPD), most of the existing work is based on images as input. If these algorithms can be applied to multiple frames or videos, they can be adapted to more complex unconstrained scenes. In this paper, we propose a LPD framework for detecting license plates in multiple frames or videos, called AWFA-LPD, which effectively integrates the features of nearby frames. Compared with image based detection models, our network integrates optical flow extraction module, which can propagate the features of local frames and fuse with the reference frame. Moreover, we concatenate a non-link suppression module after the detection results to post-process the bounding boxes. Extensive experiments demonstrate the effectiveness and efficiency of our framework.

With the increasing availability of Electronic Health Records (EHRs) and advances in deep learning techniques, developing deep predictive models that use EHR data to solve healthcare problems has gained momentum in recent years. The majority of clinical predictive models benefit from structured data in EHR (e.g., lab measurements and medications). Still, learning clinical outcomes from all possible information sources is one of the main challenges when building predictive models. This work focuses on two sources of information that have been underused by researchers; unstructured data (e.g., clinical notes) and a patient network. We propose a novel hybrid deep learning model, DeepNote-GNN, that integrates clinical notes information and patient network topological structure to improve 30-day hospital readmission prediction. DeepNote-GNN is a robust deep learning framework consisting of two modules: DeepNote and patient network. DeepNote extracts deep representations of clinical notes using a feature aggregation unit on top of a state-of-the-art Natural Language Processing (NLP) technique - BERT. By exploiting these deep representations, a patient network is built, and Graph Neural Network (GNN) is used to train the network for hospital readmission predictions. Performance evaluation on the MIMIC-III dataset demonstrates that DeepNote-GNN achieves superior results compared to the state-of-the-art baselines on the 30-day hospital readmission task. We extensively analyze the DeepNote-GNN model to illustrate the effectiveness and contribution of each component of it. The model analysis shows that patient network has a significant contribution to the overall performance, and DeepNote-GNN is robust and can consistently perform well on the 30-day readmission prediction task.

Recently, cargo ship detection in remote sensing images based on deep learning is of great significance for cargo ship monitoring. However, the existing detection network is not only unable to realize autonomous operation on spaceborne platforms due to the limitation of computing and storage, but the detection result also lacks the directional information of the cargo ship. In order to address the above problems, we propose a novel cargo ship detection and directional discrimination method for remote sensing images based on a lightweight network. Specifically, we design an efficient and lightweight feature extraction network called the one-shot aggregation and depthwise separable network (OSADSNet), which is inspired by one-shot feature aggregation modules and depthwise separable convolutions. Additionally, we combine the RPN with the K-Mean++ algorithm to obtain the K-RPN, which can produce a more suitable region proposal for cargo ship detection. Furthermore, without introducing extra parameters, the directional discrimination of the cargo ship is transformed into a classification task, and the directional discrimination is completed when the detection task is completed. Experiments on a self-built remote sensing image cargo ship dataset indicate that our model can provide relatively accurate and fast detection for cargo ships (mAP of 91.96% and prediction time of 46 ms per image) and discriminate the directions (north, east, south, and west) of cargo ships, with fewer parameters (model size of 110 MB), which is more suitable for autonomous operation on spaceborne platforms. Therefore, the proposed method can meet the needs of cargo ship detection and directional discrimination in remote sensing images on spaceborne platforms.

Deep convolutional neural networks (CNNs) are successful in self-extracting features for video object detection. The deep features and shallow features extracted from CNN are different. The shallow features have low-level semantic information, while the deep features contain high-level semantic information. In this paper, we propose an effective feature fusion method: Multi-level feature aggregation (MFA), which connects the output layer of each stage to the input layer of other stages and combines the output of each stage at the last layer of the network. This architecture can effectively combine shallow features and deep features to enhance the ability of expressing features and recognition accuracy. MFA is a flexible and end-to-end network. In addition, our experiments prove that MFA achieves significant accuracy on DET and VID datasets on object detection, and our method achieves mAP on DET and VID.

Image retrieval based on deep convolutional features has demonstrated state-of-the-art performance in popular benchmarks. In this paper, we present a unified solution to address deep convolutional feature aggregation and image re-ranking by simulating the dynamics of heat diffusion. A distinctive problem in image retrieval is that repetitive or \emph{bursty} features tend to dominate final image representations, resulting in representations less distinguishable. We show that by considering each deep feature as a heat source, our unsupervised aggregation method is able to avoid over-representation of \emph{bursty} features. We additionally provide a practical solution for the proposed aggregation method and further show the efficiency of our method in experimental evaluation. Inspired by the aforementioned deep feature aggregation method, we also propose a method to re-rank a number of top ranked images for a given query image by considering the query as the heat source. Finally, we extensively evaluate the proposed approach with pre-trained and fine-tuned deep networks on common public benchmarks and show superior performance compared to previous work., Comment: The paper has been accepted to IEEE Transactions on Multimedia

In this paper we propose a novel framework to process Doppler-radar signals for hand gesture recognition. Doppler-radar sensors provide many advantages over other emerging sensing modalities, including low development costs and high sensitivity to capture subtle gestures with precision. Furthermore, they have attractive properties for ubiquitous deployment and can be conveniently embedded into different devices. In this scope, current recognition methods still rely in deep CNN-LSTM and $3D$ CNN-LSTM structures that require sufficient labelled data to optimize millions of parameters and significant amount of computational resources for inference; which limits their deployment. Indeed, subtle gestures recognition is a challenging task due to the high variability of gestures among different subjects. To overcome the challenges in the recognition task and the limitations of the current methods, we propose a shallow learning approach for gesture recognition, that is based on unsupervised range-Doppler features representation, along with a learnable pooling aggregation via NetVLAD. The proposed framework can encode extremely valuable information across time, and results in features that are highly discriminative for hand gesture recognition. Experimentation on publicly available Doppler-radar data shows that the proposed framework outperforms state-of-the-art approaches in terms of recognition accuracy and speed for sequence-level hand gesture classification.

Traffic sign detection has achieved promising results in recent years. Nevertheless, there are still two problems remain to be overcome. One problem is the detection of small traffic signs, which usually occupy less than 2% of the image area. The other problem is fine-grained classification, with difficulties arising from similar appearances between traffic signs. For example, different speed-limit traffic signs have differences solely from the speed numbers. In this paper, we propose a Feature Aggregation MultiPath Network (FAMN) to tackle the problems simultaneously. First, we propose a Feature Aggregation (FA) structure to aggregate regional features from different feature maps by using element-wise Max, then convolution layers are used to extract rich semantic features. Accordingly, objects of different scales can choose the best features to improve performance of small object detection. Second, we propose a Multipath Network (MN) structure to obtain fine-grained features. The MN structure consists of three paths, extracting instance-level, part-level, and context-level features, respectively. The three types of features are then concatenated to form fine-grained features of the proposals. Experimental results demonstrate the effectiveness of our proposed FAMN. Specifically, FAMN is able to obtain an average F1-measure of 93.1% in TT100K dataset, 2.9% higher than the state-of-the-art.

Object counting has become an important task in computer vision for its practical applications in surveillance system. Previous methods for object counting have achieved promising results in accuracy, but few researchers focus on the real-time performance of counting methods. In this paper, we propose an efficient and accurate light-weight network for object counting, called Partial Feature Aggregation Network (PFANet). In this novel method, a Partial Feature Aggregation (PFA) structure is designed to accelerate networks and improve the utilization of multi-scale features. Moreover, PFANet uses the dilated convolution to enlarge the receptive-filed of network. Experiments on two datasets indicate our network exceeds the existing real-time counting networks in both accuracy and efficiency.

The main challenges of semantic segmentation in vehicle-mounted scenes are object scale variation and trading off model accuracy and efficiency. Lightweight backbone networks for semantic segmentation usually extract single-scale features layer-by-layer only by using a fixed receptive field. Most modern real-time semantic segmentation networks heavily compromise spatial details when encoding semantics, and sacrifice accuracy for speed. Many improving strategies adopt dilated convolution and add a sub-network, in which either intensive computation or redundant parameters are brought. We propose a multi-level and multi-scale feature aggregation network (MMFANet). A spatial pyramid module is designed by cascading dilated convolutions with different receptive fields to extract multi-scale features layer-by-layer. Subseqently, a lightweight backbone network is built by reducing the feature channel capacity of the module. To improve the accuracy of our network, we design two additional modules to separately capture spatial details and high-level semantics from the backbone network without significantly increasing the computation cost. Comprehensive experimental results show that our model achieves 79.3% MIoU on the Cityscapes test dataset at a speed of 58.5 FPS, and it is more accurate than SwiftNet (75.5% MIoU). Furthermore, the number of parameters of our model is at least 53.38% less than that of other state-of-the-art models.

Attention-based techniques have been successfully used for rating image quality, and have been widely employed for set-based face recognition. Nevertheless, for video face recognition, where the base convolutional neural network (CNN) trained on large-scale data already provides discriminative features, fusing features with only predicted quality scores to generate representation are likely to cause duplicate sample dominant problem, and degrade performance correspondingly. To resolve the problem mentioned above, we propose a redundancy removing aggregation network (RRAN) for video face recognition. Compared with other quality-aware aggregation schemes, RRAN can take advantage of similarity information to tackle the noise introduced by redundant video frames. By leveraging metric learning, RRAN introduces a distance calibration scheme to align distance distributions of negative pairs of different video representations, which improves the accuracy under a uniform threshold. A series of experiments is conductedon multiple realistic data sets to evaluate the performance of RRAN, including YouTube Faces, IJB-A, and IJB-C. In comprehensive experiments, we demonstrate that our method can diminish the overall influence of poor quality components with large proportion in the video and further improve the overall recognition performance with individual difference. Specifically, RRAN achieves a 96.84% accuracy on YouTube Face, outperforming all existing aggregation schemes.

Despite the significant progress made by deep learning in natural image matting, there has been so far no representative work on deep learning for video matting due to the inherent technical challenges in reasoning temporal domain and lack of large-scale video matting datasets. In this paper, we propose a deep learning-based video matting framework which employs a novel and effective spatio-temporal feature aggregation module (ST-FAM). As optical flow estimation can be very unreliable within matting regions, ST-FAM is designed to effectively align and aggregate information across different spatial scales and temporal frames within the network decoder. To eliminate frame-by-frame trimap annotations, a lightweight interactive trimap propagation network is also introduced. The other contribution consists of a large-scale video matting dataset with groundtruth alpha mattes for quantitative evaluation and real-world high-resolution videos with trimaps for qualitative evaluation. Quantitative and qualitative experimental results show that our framework significantly outperforms conventional video matting and deep image matting methods applied to video in presence of multi-frame temporal information. Our dataset is available at https://github.com/nowsyn/DVM.

In this paper, we propose a novel bidirectional multiscale feature aggregation (BMFA) network with attentional fusion modules for text-independent speaker verification. The feature maps from different stages of the backbone network are iteratively combined and refined in both a bottom-up and top-down manner. Furthermore, instead of simple concatenation or element-wise addition of feature maps from different stages, an attentional fusion module is designed to compute the fusion weights. Experiments are conducted on the NIST SRE16 and VoxCeleb1 datasets. The experimental results demonstrate the effectiveness of the bidirectional aggregation strategy and show that the proposed attentional fusion module can further improve the performance.

Multi-level feature aggregation and part feature extraction are widely used to boost the performance of person re-identification (Re-ID). Most multi-level feature aggregation methods treat feature maps on different levels equally and use simple local operations for feature fusion, which neglects the long-distance connection among feature maps. On the other hand, the popular horizon pooling part based feature extraction methods may lead to feature misalignment. In this paper, we propose a novel Part-aware Attention Network (PAN) to connect part feature maps and middle-level features. Given a part feature map and a source feature map, PAN uses part features as queries to perform second-order information propagation from the source feature map. The attention is computed based on the compatibility of the source feature map with the part feature map. Specifically, PAN uses high-level part features of different human body parts to aggregate information from mid-level feature maps. As a part-aware feature aggregation method, PAN operates on all spatial positions of feature maps so that it can discover long-distance relations. Extensive experiments show that PAN achieves leading performance on Re-ID benchmarks Market1501, DukeMTMC, and CUHK03.

The 3D point cloud data contains the semantic information of the measured scene object, and has the characteristics of massive, high density and high precision. It has become a main data type used to understand, analyze and semantically explain the 3D natural scene. This paper takes indoor 3D point cloud data as the research object and studies the semantic segmentation of 3D point cloud. Based on the in-depth study of PointNet++ network, this paper proposes a local information aggregation module based on graph network to optimize PointNet++ network. The local information aggregation module based on graph network proposed in this paper uses graph network learning to explore the mutual influence of points in a local area, and then aggregate local information to generate more representative local features. Finally, we evaluated our model on S3DIS and compared it quantitatively with PointNet++ to show the improved performance of the original algorithm.

In this paper, we focus on the task of generating realistic images given an input semantic layout, which is also called semantic image synthesis. Most of previous methods are based on conditional generative adversarial networks mechanism, which is stacks of convolution, normalization, and non-linearity layers. However, these methods easily generate blurred regions and distorted structures. There are two limits existing: their normalization layers are unable to make a good balance between keeping semantic layout information and geometric changes; and cannot effectively aggregated multi-level feature. To address the above problems, we propose a novel method which incorporates multi-level gate feature aggregation mechanism (GFA) and spatially adaptive batch-instance normalization (SPAda-BIN) for semantic image synthesis. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, in terms of both visual fidelity and quantitative metrics.

Automatic detection of mass in mammograms is a big challenge and plays a crucial role in assisting radiologists for accurate diagnosis. Applying excellent deep learning based detection networks to medical image is a direction. In this paper, an anchor-free YOLOv3 network is proposed for mass detection in mammogram. The main improvements are as follows. Anchor-based (AB) detection scheme may involve too many negative boxes, which may aggravate the unbalance between positive boxes and negative boxes. Aiming at this problem, a novel anchor-free (AF) detection scheme is developed for box regression. To alleviate the problem caused by using mean square error (MSE) as box regression loss, generalized intersection over union (GIoU) loss is adopted to train bounding box regression. Focal loss is adopted as the objectness prediction loss, which can prevent the network degradation caused by the vast number of easy negatives. Moreover, a new feature aggregation manner by summing is designed and employed in the up to down pathway. Comparative experiments are performed on two databases (publicly available dataset INbreast and private dataset TXMD). For INbreast dataset, the proposed method achieved 0.95 true positive rate (TPR) with 1.7 false positives per image (FPs/I). For TXMD dataset, the proposed method achieved 0.94 TPR with 5.75 FPs/I. Experimental results demonstrate that the proposed method outperforms YOLOv3 network and gets a promising result compared with existing mass detection methods.

We investigate the use of recent advances in deep learning and propose an end-to-end trainable multi-instance convolutional neural network within a mixture-of-experts formulation that combines information from two types of data—images and clinical attributes—for the diagnosis of lymphocytosis. The convolutional network learns to extract meaningful features from images of blood cells using an embedding level approach and aggregates them. Moreover, the mixture-of-experts model combines information from these images as well as clinical attributes to form an end-to-end trainable pipeline for diagnosis of lymphocytosis. Our results demonstrate that even the convolutional network by itself is able to discover meaningful associations between the images and the diagnosis, indicating the presence of important unexploited information in the images. The mixture-of-experts formulation is shown to be more robust while maintaining performance via. a repeatability study to assess the effect of variability in data acquisition on the predictions. The proposed methods are compared with different methods from literature based both on conventional handcrafted features and machine learning, and on recent deep learning models based on attention mechanisms. Our method reports a balanced accuracy of $\text{85.41}\%$ and outperfroms the handcrafted feature-based and attention-based approaches as well that of biologists which scored $\text{79.44}\%$ , $\text{82.89}\%$ and $\text{77.07}\%$ respectively. These results give insights on the potentials of the applicability of the proposed method in clinical practice. Our code and datasets can be found at https://github.com/msahasrabudhe/lymphoMIL .

In the project of false news detection, model which can judge whether the news is fake was built. By using this model, the information safety degree of the society will be strongly improved. To start with the work, a model was created by using the Tensorflow. Then the loss function and optimizer were used to train the model. Finally, a plot of accuracy and loss time was created in order to evaluate the model effectively. After these steps down, a model which can test whether a piece of news is fake will be created. From the process of exploring machine learning, the ability of using Excel to evaluate data was improved, and the developing intelligent recognition with python was achieved.

In recent years, deep learning has achieved promising success for multimedia quality assessment, especially for image quality assessment (IQA). However, since there exist more complex temporal characteristics in videos, very little work has been done on video quality assessment (VQA) by exploiting powerful deep convolutional neural networks (DCNNs). In this paper, we propose an efficient VQA method named Deep SpatioTemporal video Quality assessor (DeepSTQ) to predict the perceptual quality of various distorted videos in a no-reference manner. In the proposed DeepSTQ, we first extract local and global spatiotemporal features by pre-trained deep learning models without fine-tuning or training from scratch. The composited features consider distorted video frames as well as frame difference maps from both global and local views. Then, the feature aggregation is conducted by the regression model to predict the perceptual video quality. Finally, experimental results demonstrate that our proposed DeepSTQ outperforms state-of-the-art quality assessment algorithms.

Keypoints of objects reflect their concise abstractions, while the corresponding connection links (CL) build the skeleton by detecting the intrinsic relations between keypoints. Existing approaches are typically computationally-intensive, inapplicable for instances belonging to multiple classes, and/or infeasible to simultaneously encode connection information. To address the aforementioned issues, we propose an end-to-end category-implicit Keypoint and Link Prediction Network (KLPNet), which is the first approach for simultaneous semantic keypoint detection (for multi-class instances) and CL rejuvenation. In our KLPNet, a novel Conditional Link Prediction Graph is proposed for link prediction among keypoints that are contingent on a predefined category. Furthermore, a Cross-stage Keypoint Localization Module (CKLM) is introduced to explore feature aggregation for coarse-to-fine keypoint localization. Comprehensive experiments conducted on three publicly available benchmarks demonstrate that our KLPNet consistently outperforms all other state-of-the-art approaches. Furthermore, the experimental results of CL prediction also show the effectiveness of our KLPNet with respect to occlusion problems., WACV 2021 paper

BackgroundRadiomic feature analysis has been shown to be effective at modeling cancer outcomes. It has not yet been established how to best combine these radiomic features in patients with multifocal disease. As the number of patients with multifocal metastatic cancer continues to rise, there is a need for improving personalized patient-level prognostication to better inform treatment.MethodsWe compared six mathematical methods of combining radiomic features of 3596 tumors in 831 patients with multiple brain metastases and evaluated the performance of these aggregation methods using three survival models: a standard Cox proportional hazards model, a Cox proportional hazards model with LASSO regression, and a random survival forest.ResultsAcross all three survival models, the weighted average of the largest three metastases had the highest concordance index (95% confidence interval) of 0.627 (0.595-0.661) for the Cox proportional hazards model, 0.628 (0.591-0.666) for the Cox proportional hazards model with LASSO regression, and 0.652 (0.565-0.727) for the random survival forest model.ConclusionsRadiomic features can be effectively combined to establish patient-level outcomes in patients with multifocal brain metastases. Future studies are needed to confirm that the volume-weighted average of the largest three tumors is an effective method for combining radiomic features across other imaging modalities and disease sites.

Convolutional Neural Network (CNN) based multi-task learning methods have been widely used in a variety of applications of computer vision. Towards effective multi-task CNN architectures, recent studies automatically learn the optimal combinations of task-specific features at single network layers. However, they generally construct an unchanged operation of feature aggregation after training, regardless of the characteristics of input features. In this paper, we propose a novel Adaptive Feature Aggregation (AFA) layer for multi-task CNNs, in which a dynamic aggregation mechanism is designed to allow each task to adaptively determine the degree to which the feature aggregation of different tasks is needed according to the feature dependencies. On both pixel-level and image-level tasks, we demonstrate that our approach significantly outperforms the previous state-of-the-art methods of multi-task CNNs.

Due to its ill-posed nature, single image dehazing is a challenging problem. In this paper, we propose an end-to-end feature aggregation attention network (FAAN) for single image dehazing. It incorporates the idea of attention mechanism and residual learning and can adaptively aggregate different level features. In particular, in the proposed FANN, we design a novel block structure consisting of feature attention module, smoothed dilated convolution and local residual learning. The local residual learning allows the less useful information to be bypassed through multiple skip connections. The feature attention module is designed to assign more weight to important features. The smoothed dilated convolution is adopted to enlarge the receptive field without the negative influence of gridding artifacts. The experiments on the RESIDE dataset show that the proposed approach acquires state-of-the-art performance in both qualitative and quantitative measures.

Semantic Scene Completion (SSC) aims to simultaneously predict the volumetric occupancy and semantic category of a 3D scene. In this paper, we propose a real-time semantic scene completion method with a feature aggregation strategy and conditioned prediction module. Feature aggregation fuses feature with different receptive fields and gathers context to improve scene completion performance. And the conditioned prediction module adopts a two-step prediction scheme that takes volumetric occupancy as a condition to enhance semantic completion prediction. We conduct experiments on three recognized benchmarks NYU, NYUCAD, and SUNCG. Our method achieves competitive performance at a speed of 110 FPS on one GTX 1080 Ti GPU.

In recent years, with the improvement of synthetic aperture radar (SAR) imaging resolution, it is urgent to develop methods with higher accuracy and faster speed for ship detection in high-resolution SAR images. Among all kinds of methods, deep-learning-based algorithms bring promising performance due to end-to-end detection and automated feature extraction. However, several challenges still exist: (1) standard deep learning detectors based on anchors have certain unsolved problems, such as tuning of anchor-related parameters, scale-variation and high computational costs. (2) SAR data is huge but the labeled data is relatively small, which may lead to overfitting in training. (3) To improve detection speed, deep learning detectors generally detect targets based on low-resolution features, which may cause missed detections for small targets. In order to address the above problems, an anchor-free convolutional network with dense attention feature aggregation is proposed in this paper. Firstly, we use a lightweight feature extractor to extract multiscale ship features. The inverted residual blocks with depth-wise separable convolution reduce the network parameters and improve the detection speed. Secondly, a novel feature aggregation scheme called dense attention feature aggregation (DAFA) is proposed to obtain a high-resolution feature map with multiscale information. By combining the multiscale features through dense connections and iterative fusions, DAFA improves the generalization performance of the network. In addition, an attention block, namely spatial and channel squeeze and excitation (SCSE) block is embedded in the upsampling process of DAFA to enhance the salient features of the target and suppress the background clutters. Third, an anchor-free detector, which is a center-point-based ship predictor (CSP), is adopted in this paper. CSP regresses the ship centers and ship sizes simultaneously on the high-resolution feature map to implement anchor-free and nonmaximum suppression (NMS)-free ship detection. The experiments on the AirSARShip-1.0 dataset demonstrate the effectiveness of our method. The results show that the proposed method outperforms several mainstream detection algorithms in both accuracy and speed.

Deep learning methods and applications have become an integral part of all finance subfields. Fraud detection is one of these subfields where these applications managed to save billions of dollars every year. However, one weakness of these models is that we cannot understand what rules they follow to make their decisions. This paper aims to introduce a novel deep learning approach to detect fraud in e-commerce, taking into consideration the problem mentioned above by extracting if-then rules using CRED algorithm (a Continuous/discrete Rule Extractor via Decision tree induction) that employs decision trees to extract these rules. Besides using feature aggregation and re-sampling method on two highly imbalanced data-sets, we conducted a comparative study with previous work. The results show that our model gives better performance.

Recently, very deep convolutional neural networks (CNNs) have shown great power in single image super-resolution (SISR) and achieved significant improvements against traditional methods. Among these CNN-based methods, the residual connections play a critical role in boosting the network performance. As the network depth grows, the residual features gradually focused on different aspects of the input image, which is very useful for reconstructing the spatial details. However, existing methods neglect to fully utilize the hierarchical features on the residual branches. To address this issue, we propose a novel residual feature aggregation (RFA) framework for more efficient feature extraction. The RFA framework groups several residual modules together and directly forwards the features on each local residual branch by adding skip connections. Therefore, the RFA framework is capable of aggregating these informative residual features to produce more representative features. To maximize the power of the RFA framework, we further propose an enhanced spatial attention (ESA) block to make the residual features to be more focused on critical spatial contents. The ESA block is designed to be lightweight and efficient. Our final RFANet is constructed by applying the proposed RFA framework with the ESA blocks. Comprehensive experiments demonstrate the necessity of our RFA framework and the superiority of our RFANet over state-of-the-art SISR methods.