Search

Semantic scene labeling plays a very important role in intelligent transportation tasks, such as autonomous driving and advanced driver assistance. Recently, thanks to the advances of deep learning, significant improvements have been achieved for this pixel-wise labeling task. Although effective, current methods lack of explicitly modeling the boundary of objects, resulting in inaccurate labeling results. Meanwhile, traditional level set based methods perform better to capture the evolution of boundaries. However, they are sensitive to the model initialization. To address these issues, in this work we propose a novel deep learning framework, named deep nested level set (DNLS) for boundary-aware semantic scene labeling. Different from previous works, our proposed framework explicitly takes deep learned features and object boundary information into account. More specifically, our proposed framework first predicts semantic probability maps and boundary locations of objects using a bifurcated fully convolutional network (BFCN). Then, these probability maps are seamlessly integrated into a nested level set function for accurate scene labeling. As a result, our approach can automatically initialize the nested level set function, and the whole framework can be trained in an end-to-end manner, providing a new solution for accurate semantic scene parsing. Extensive experiments on public CamVid and Cityscapes datasets demonstrate that our proposed framework produces high-quality predictions with clear object boundaries and spatial consistency.

Semantic context plays a significant role in image segmentation. However, few prior works have explored semantic contexts for 3D point cloud segmentation. In this paper, we propose a simple yet effective Point Context Encoding (PointCE) module to capture semantic contexts of a point cloud and adaptively highlight intermediate feature maps. We also introduce a Semantic Context Encoding loss (SCE-loss) to supervise the network to learn rich semantic context features. To avoid hyperparameter tuning and achieve better convergence performance, we further propose a geometric mean loss to integrate both SCE-loss and segmentation loss. Our PointCE module is general and lightweight, and can be integrated into any point cloud segmentation architecture to improve its segmentation performance with only marginal extra overheads. Experimental results on the ScanNet, S3DIS and Semantic3D datasets show that consistent and significant improvement can be achieved for several different networks by integrating our PointCE module.

Semantic segmentation is a crucial image understanding task, where each pixel of image is categorized into a corresponding label. Since the pixel-wise labeling for ground-truth is tedious and labor intensive, in practical applications, many works exploit the synthetic images to train the model for real-word image semantic segmentation, i.e., Synthetic-to-Real Semantic Segmentation (SRSS). However, Deep Convolutional Neural Networks (CNNs) trained on the source synthetic data may not generalize well to the target real-world data. In this work, we propose two simple yet effective texture randomization mechanisms, Global Texture Randomization (GTR) and Local Texture Randomization (LTR), for Domain Generalization based SRSS. GTR is proposed to randomize the texture of source images into diverse unreal texture styles. It aims to alleviate the reliance of the network on texture while promoting the learning of the domain-invariant cues. In addition, we find the texture difference is not always occurred in entire image and may only appear in some local areas. Therefore, we further propose a LTR mechanism to generate diverse local regions for partially stylizing the source images. Finally, we implement a regularization of Consistency between GTR and LTR (CGL) aiming to harmonize the two proposed mechanisms during training. Extensive experiments on five publicly available datasets (i.e., GTA5, SYNTHIA, Cityscapes, BDDS and Mapillary) with various SRSS settings (i.e., GTA5/SYNTHIA to Cityscapes/BDDS/Mapillary) demonstrate that the proposed method is superior to the state-of-the-art methods for domain generalization based SRSS., Comment: 15 pages, 14 figures, accepted by IEEE Transactions on Image Processing (TIP 2021)

This study proposes a ground-based threat probability model based on a Markov chain to evaluate multiple ground threats faced by the platform during flight. On this basis, a new threat evaluation and jamming allocation system is constructed by combining the model with jamming resource allocation methods. In the threat evaluation phase, the interaction relationship between the platform and radar is represented by the transfer intensity among radar states according to the domain expert knowledge. Then, the potential state probability of radars in each time period is deduced and converted into the danger value as the objective to be optimised in the next stage. In the jamming allocation phase, the system considers some effects such as jamming technologies, multi-channel, the uncertainty of the threatening environment and other factors, and simulates the process of ground threats. There is traversing all jamming combinations at each period to select the best jamming strategy. This study also designs a multi-threat environment to verify the effectiveness of the system and test its relative performance improvement. Compared with the original threat evaluation and jamming allocation system, the new system has proved its feasibility and better application value.

Street Scene Parsing (SSP) is a fundamental and important step for autonomous driving and traffic scene understanding. Recently, Fully Convolutional Network (FCN) based methods have delivered expressive performances with the help of large-scale dense-labeling datasets. However, in urban traffic environments, not all the labels contribute equally for making the control decision. Certain labels such as pedestrian, car, bicyclist, road lane or sidewalk would be more important in comparison with labels for vegetation, sky or building. Based on this fact, in this paper we propose a novel deep learning framework, named Residual Atrous Pyramid Network (RAPNet), for importance-aware SSP. More specifically, to incorporate the importance of various object classes, we propose an Importance-Aware Feature Selection (IAFS) mechanism which automatically selects the important features for label predictions. The IAFS can operate in each convolutional block, and the semantic features with different importance are captured in different channels so that they are automatically assigned with corresponding weights. To enhance the labeling coherence, we also propose a Residual Atrous Spatial Pyramid (RASP) module to sequentially aggregate global-to-local context information in a residual refinement manner. Extensive experiments on two public benchmarks have shown that our approach achieves new state-of-the-art performances, and can consistently obtain more accurate results on the semantic classes with high importance levels.

With the superiority of three-dimensional (3D) scanning data, e.g., illumination invariance and pose robustness, 3D face recognition theoretically has the potential to achieve better results than two-dimensional (2D) face recognition. However, traditional 3D face recognition techniques suffer from high computational costs. This paper proposes a fast and robust 3D face recognition approach with three component technologies: a fast 3D scan preprocessing, multiple data augmentation, and a deep learning technique based on facial component patches. First, unlike the majority of the existing approaches, which require accurate facial registration, the proposed approach uses only three facial landmarks. Second, the specifical deep network with an improved supervision is designed to extract complementary features from four overlapping facial component patches. Finally, a data augmentation technique and three self-collected 3D face datasets are used to enlarge the scale of the training data. The proposed approach outperforms the state-of-the-art algorithms on four public 3D face benchmarks, i.e., 100%, 99.75%, 99.88%, and 99.07% rank-1 IRs with the standard test protocol on the FRGC v2.0, Bosphorus, BU-3DFE, and 3D-TEC datasets, respectively. Further, it requires only 0.84 seconds to identify a probe from a gallery with 466 faces.

Salient Object Detection (SOD), which aims to find the most important region of interest and segment the relevant objects/items in that region, is an important yet challenging task in computer vision and image processing. This vision problem is inspired by the fact that human perceives the main scene elements with high priorities. Thus, accurate detection of salient objects in complex scenes is critical for human-computer interaction. In this paper, we present a novel reflective feature learning framework, which results in high detection accuracy while maintaining a compact model design. The proposed framework utilizes a hyper-densely reflective feature fusion network (named HyperFusion-Net) to automatically predict the most important area and segment the associated objects in an end-to-end manner. Specifically, inspired by the human perception system and image reflection separation, we first decompose the input images into reflective image pairs by content-preserving transforms. Then, the complementary information of reflective image pairs is jointly extracted by an Interweaved Convolutional Neural Network (ICNN) and hierarchically combined with a hyper-dense fusion mechanism. Based on the fused multi-scale features, our method finally achieves a promising way of predicting salient objects, in which we cast the SOD as a pixel-wise classification problem. Extensive experiments on seven public datasets demonstrate that the proposed method consistently outperforms other state-of-the-art methods with a large margin.

Street-level scene segmentation aims to label each pixel of street-view images into specific semantic categories. It has been attracting growing interest due to various real-world applications, especially in the area of autonomous driving. However, this pixel-wise labeling task is very challenging under the complex street-level scenes and large-scale object categories. Motivated by the scene layout of street-view images, in this work we propose a novel Spatial Gated Attention (SGA) module, which automatically highlights the attentive regions for pixel-wise labeling, resulting in effective street-level scene segmentation. The proposed module takes as input the multi-scale feature maps based on a Fully Convolutional Network (FCN) backbone, and produces the corresponding attention mask for each feature map. The learned attention masks can neatly highlight the regions of interest while suppress background clutter. Furthermore, we propose an efficient multi-scale feature interaction mechanism which is able to adaptively aggregate the hierarchical features. Based on the proposed mechanism, the features of different levels are adaptively re-weighted according to the local spatial structure and the surrounding contextual information. Consequently, the proposed modules are able to boost standard FCN architectures and result in an enhanced pixel-wise segmentation for street-level scene images. Extensive experiments on three public available street-level benchmarks demonstrate that the proposed Gated Attention Network (GANet) approach achieves consistently superior performance and outperforms the very recent state-of-the-art methods.

NAND flash memory has been widely used in consumer electronics, such as tablet personal computers and smart phones. However, unlike traditional hard disk, a garbage collection is required to reclaim memory space during data updates. A garbage collection includes a series of extra read, write and erase operations. Both write and erase operations are time consuming process, which affect the effectiveness and efficiency of the NAND flash memory system. Moreover, flash memory blocks are challenged by the limitation of the erase count. Thus, considerable efforts have been paid to reduce the garbage collection overhead and improve wear leveling. In this paper, an efficient and non-time-sensitive file-aware garbage collection algorithm, called FaGC+, is proposed. The FaGC+ algorithm involves a novel update frequency calculation method and a novel cold-hot logical page categorization scheme. The experimental results demonstrate that the proposed algorithm outperforms existing garbage collection algorithms in terms of garbage collection overhead and time-sensitivity in wear leveling control.

To address the high computational and memory cost in 3-D volumetric convolutional neural networks (CNNs), we propose an approach to train binary volumetric CNNs for 3-D object recognition. Our method is specifically designed for 3-D data, in which it transforms the inputs and weights in convolutional/fully connected layers to binary values, which can potentially accelerate the networks by efficient bitwise operations. Two loss calculation methods are designed to solve the accuracy decrease problem when the weights in the last layer are binarized. Four binary volumetric CNNs are obtained from their corresponding floating-point networks using our approach. Evaluations on three public datasets from different domains (Computer Aided Design (CAD), light detection and ranging (LiDAR), and RGB-D) show that our binary volumetric CNNs can achieve a comparable recognition performance as their floating-point counterparts but consume less computational and memory resources.

In recent years, Salient Object Detection (SOD) has shown great success with the achievements of large-scale benchmarks and deep learning techniques. However, existing SOD methods mainly focus on natural images with low-resolutions, e.g., $400\times 400$ or less. This drawback hinders them for advanced practical applications, which need high-resolution, detail-aware results. Besides, lacking of the boundary detail and semantic context of salient objects is also a key concern for accurate SOD. To address these issues, in this work we focus on the High-Resolution Salient Object Detection (HRSOD) task. Technically, we propose the first end-to-end learnable framework, named Dual ReFinement Network (DRFNet), for fully automatic HRSOD. More specifically, the proposed DRFNet consists of a shared feature extractor and two effective refinement heads. By decoupling the detail and context information, one refinement head adopts a global-aware feature pyramid. Without increasing too much computational burden, it can boost the spatial detail information, which narrows the gap between high-level semantics and low-level details. In parallel, the other refinement head adopts hybrid dilated convolutional blocks and group-wise upsamplings, which are very efficient in extracting contextual information. Based on the dual refinements, our approach can enlarge receptive fields and obtain more discriminative features from high-resolution images. Experimental results on high-resolution benchmarks (the public DUT-HRSOD and the proposed DAVIS-SOD) demonstrate that our method is not only efficient but also performs more accurate than other state-of-the-arts. Besides, our method generalizes well on typical low-resolution benchmarks.

Recently, Fully Convolutional Network (FCN) seems to be the go-to architecture for image segmentation, including semantic scene parsing. However, it is difficult for a generic FCN to discriminate pixels around the object boundaries, thus FCN based methods may output parsing results with inaccurate boundaries. Meanwhile, level set based active contours are superior to the boundary estimation due to the sub-pixel accuracy that they achieve. However, they are quite sensitive to initial settings. To address these limitations, in this paper we propose a novel Deep Multiphase Level Set (DMLS) method for semantic scene parsing, which efficiently incorporates multiphase level sets into deep neural networks. The proposed method consists of three modules, i.e., recurrent FCNs, adaptive multiphase level set, and deeply supervised learning. More specifically, recurrent FCNs learn multi-level representations of input images with different contexts. Adaptive multiphase level set drives the discriminative contour for each semantic class, which makes use of the advantages of both global and local information. In each time-step of the recurrent FCNs, deeply supervised learning is incorporated for model training. Extensive experiments on three public benchmarks have shown that our proposed method achieves new state-of-the-art performances., Comment: 12 pages, 8 tables, 8 figures

Incorporating knowledge bases (KB) into end-to-end task-oriented dialogue systems is challenging, since it requires to properly represent the entity of KB, which is associated with its KB context and dialogue context. The existing works represent the entity with only perceiving a part of its KB context, which can lead to the less effective representation due to the information loss, and adversely favor KB reasoning and response generation. To tackle this issue, we explore to fully contextualize the entity representation by dynamically perceiving all the relevant entities} and dialogue history. To achieve this, we propose a COntext-aware Memory Enhanced Transformer framework (COMET), which treats the KB as a sequence and leverages a novel Memory Mask to enforce the entity to only focus on its relevant entities and dialogue history, while avoiding the distraction from the irrelevant entities. Through extensive experiments, we show that our COMET framework can achieve superior performance over the state of the arts., Comment: EMNLP-2021

Although the use of the convolutional neural network (CNN) improved the accuracy of object recognition, it still had a long-running time. In order to solve these problems, the training and testing datasets were split at four different proportions to reduce the impact of inherent error. Using model fine-tuning, the model converged in a small number of iterations, and the average recognition accuracy of BWN test can reach 96.8%. In the segmented dataset, the recognition accuracy of the former was 4.7 percentage points higher than the latter by comparing color dataset and grayscale dataset, which proved that a certain amount of color features will have a positive impact on the model. The segmented dataset was 0.9 percentage points higher than the color dataset; it shows that the model focused more on features of contour and texture by eliminating the background of images. The experiments showed that the binarized convolutional neural network can effectively improve recognition efficiency and accuracy compared with traditional methods.

Street Scene Change Detection (SSCD) aims to locate the changed regions between a given street-view image pair captured at different times, which is an important yet challenging task in the computer vision community. The intuitive way to solve the SSCD task is to fuse the extracted image feature pairs, and then directly measure the dissimilarity parts for producing a change map. Therefore, the key for the SSCD task is to design an effective feature fusion method that can improve the accuracy of the corresponding change maps. To this end, we present a novel Hierarchical Paired Channel Fusion Network (HPCFNet), which utilizes the adaptive fusion of paired feature channels. Specifically, the features of a given image pair are jointly extracted by a Siamese Convolutional Neural Network (SCNN) and hierarchically combined by exploring the fusion of channel pairs at multiple feature levels. In addition, based on the observation that the distribution of scene changes is diverse, we further propose a Multi-Part Feature Learning (MPFL) strategy to detect diverse changes. Based on the MPFL strategy, our framework achieves a novel approach to adapt to the scale and location diversities of the scene change regions. Extensive experiments on three public datasets (i.e., PCD, VL-CMU-CD and CDnet2014) demonstrate that the proposed framework achieves superior performance which outperforms other state-of-the-art methods with a considerable margin., Comment: To appear in Transactions on Image Processing, including 13 pages, 13 figures, 9 tables

Nowadays, the deep learning-based methods have been widely used in the scene-level-based image classification. However, the features automatically obtained from the last fully connected (FC) layer of single CNN without any process have little effect because of high dimensionality. In this paper, we propose a simple enhancing scene-level feature description method for remote sensing scene classification. Firstly, the principal component analysis (PCA) transformation is adopted in our research for reducing redundant dimensionality. Secondly, a new method is used to fuse features obtained by PCA transformation. Finally, the random forest classifier applying to classification makes a significant effect on compressing the training procedure. The results of experiments on the public dataset describe that feature fusion with PCA transformation performs great classification effect. Moreover, compared with the classifier softmax, the random forest classifier outperforms the softmax classifier in the training procedure.

Aerial image labeling plays an important part in the mapping of maps with high precision. The knowledge about the range and intensive degree of aerial building segmentation is necessary for urban planning. Fully convolutional networks (FCNs) have recently shown state-of-the-art performance in image segmentation. In order to get better aerial images segmentation performance, we use a method of combing FCNs with multi-scale features and attention model in order to carry out segmentation automatically in aerial images. Attention model gives each scale feature added extra supervision to achieve better segmentation. Here, U-net and FCN-8s are used as original semantic segmentation model to train with multi-scale images and attention models. The datasets use different proportions of Inria Aerial Image Labeling Dataset, including two semantic classes: building and not building. The results show that the semantic segmentation model combined with multi-scale features and attention model has higher segmentation accuracy and better performance.

Most existing crowd counting methods require object location-level annotation, i.e., placing a dot at the center of an object. While being simpler than the bounding-box or pixel-level annotation, obtaining this annotation is still labor-intensive and time-consuming especially for images with highly crowded scenes. On the other hand, weaker annotations that only know the total count of objects can be almost effortless in many practical scenarios. Thus, it is desirable to develop a learning method that can effectively train models from count-level annotations. To this end, this paper studies the problem of weakly-supervised crowd counting which learns a model from only a small amount of location-level annotations (fully-supervised) but a large amount of count-level annotations (weakly-supervised). To perform effective training in this scenario, we observe that the direct solution of regressing the integral of density map to the object count is not sufficient and it is beneficial to introduce stronger regularizations on the predicted density map of weakly-annotated images. We devise a simple-yet-effective training strategy, namely Multiple Auxiliary Tasks Training (MATT), to construct regularizes for restricting the freedom of the generated density maps. Through extensive experiments on existing datasets and a newly proposed dataset, we validate the effectiveness of the proposed weakly-supervised method and demonstrate its superior performance over existing solutions.